Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment

Authors: teng  xiao
yige  yuan
huaisheng  zhu
mingxiao  li
vasant g honavar
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 05:20:01

Abstract: We study the problem of aligning large language models (LLMs) with human preference data. Contrastive preference optimization has shown promising results in aligning LLMs with available preference data by optimizing the implicit reward associated with the policy. However, the contrastive objective focuses mainly on the relative values of implicit rewards associated with two responses while ignoring
their actual values, resulting in suboptimal alignment with human preferences. To address this limitation, we propose calibrated direct preference optimization (Cal-DPO), a simple yet effective algorithm. We show that substantial improvement in alignment with the given preferences can be achieved simply by calibrating the implicit reward to ensure that the learned implicit rewards are comparable in
scale to the ground-truth rewards. We demonstrate the theoretical advantages of Cal-DPO over existing approaches. The results of our experiments on a variety of standard benchmarks show that Cal-DPO remarkably improves off-the-shelf methods.

Main paper: 
 introduction: 
 Aligning the behavior of large language models (LLMs) with human preferences is crucial for ensuring that the responses of a pretrained LLM are aligned with human or societal values and preferences [1, 2, 3] . In recent years, reinforcement learning from human feedback (RLHF) [2, 4] has become a standard approach for fine-tuning language models based on human preferences. RLHF involves first fitting a reward signal from human preference data and then using reinforcement learning (RL) algorithms such as PPO [5] to optimize language models to generate responses with high reward.While RLHF shows impressive capabilities on diverse tasks ranging from programming to creative writing, its training process is unstable and complex [6, 7] . This potentially worsens the sample complexity and compromises efficient convergence. To address these issues, offline contrastive preference learning methods, which include DPO [7] , IPO [8] , and SLiC [9] , have been proposed to replace RLHF with supervised learning on the preference data. These methods eliminate the need for explicit reward modeling by directly using the likelihood of the policy to define an implicit reward fitted to the preference data, and achieve notable efficiency and competitive performance [10] .While various contrastive preference learning methods employ different pairwise ranking losses, they share a common underlying motivation: Maximize the expected relative difference between the implicit rewards associated with the chosen and rejected responses. Because the ranking loss is invariant to various score transformations (e.g., subtracting a constant), these methods tend to ignore the absolute values of the rewards. Hence, while these methods learn to preserve the relative ordering between the likelihoods of the chosen and the rejected responses, they may reduce the likelihood of the chosen response. Figure 1 illustrates this behavior and its implications. In the case of DPO, a representative method of the contrastive methods, the likelihood of the chosen response Figure 1 : The implicit reward dynamics during training of DPO and Cal-DPO on UltraFeedback data with the base model Zephyr-7b-sft reveal that the rewards for rejected data continuously decrease, while the margins between chosen and rejected data keep increasing. However, in DPO, the rewards for chosen data decrease below zero, whereas in our Cal-DPO, they keep increasing and remain positive. Our Cal-DPO significantly outperforms DPO across reasoning benchmarks. More results on other datasets are provided in Section 5. counter-intuitively continues to decrease despite remaining higher than the likelihood of the rejected response. An undesirable consequence of this behavior is that the learned policy increases the likelihood of unknown out-of-distribution responses, resulting in poor performance. Maximizing the likelihood of the chosen response can be important in many practical applications, e.g., reasoning and mathematical problem solving [11, 12] , limiting the applicability of contrastive preference learning.The preceding discussion raises an important question with significant implications for how we align LLMs with human preferences: How can we design a new objective that effectively alleviates this problem while ensuring that the learned policy theoretically converges to an optimal policy? Our answer to this question is Cal-DPO, a simple yet effective framework for preference learning which optimizes the contrastive preference objective to maximize the relative differences between implicit rewards of chosen and rejected responses, while simultaneously ensuring that learned implicit rewards are calibrated to match the actual values of the ground-truth rewards (see Section 4.1 for a formal definition). The key intuition behind Cal-DPO is quite simple: If the implicit reward estimates from preference data are well-calibrated relative to the ground-truth rewards (meaning both lie on the same scale), we can prevent the likelihood (reward) of chosen responses from continually decreasing. Hence, Cal-DPO is designed to learn an implicit reward parameterized by the policy calibrated against the ground-truth reward. This can be achieved through a simple modification to the existing methods. For instance, Cal-DPO can be implemented on top of DPO with just one line of code and without any additional hyperparameters. Although we refer to our method as Cal-DPO, it notably generalizes to other preference optimization methods such as IPO and SLiC (see Section 4.3) . In addition, we theoretically demonstrate that Cal-DPO possesses several properties that are desirable for fine-tuning LLMs based on preferences, such as mode-seeking behavior, negative preference optimization, or "negative gradient" to push down the likelihood of undesirable responses [10] .The main contributions of this paper are: (i) We propose Cal-DPO, a simple, effective, and intuitive framework for preference learning that facilitates alignment of language models. Cal-DPO aims to learn implicit reward functions for learning policy that are calibrated with respect to groundtruth rewards. (ii) We theoretically analyze the learning behaviors of Cal-DPO and prove that Cal-DPO is guaranteed to yield an optimal policy for preference learning. (iii) We present results of extensive experiments on a range of benchmark tasks, including controlled text generation [13] , summarization [14] , dialogue generation [1] , and several reasoning tasks [15] that demonstrate that Cal-DPO consistently outperforms previous alignment methods for preference fine-tuning. e.g., RSO [17] . While each of these methods works with different loss functions, the primary objective is to increase the likelihood gap between preferred and dispreferred responses [10] . Other works [18, 19, 20, 21, 22] apply contrastive preference learning iteratively and on-policy. However, as shown in Figure 1 , the likelihood of the preferred response frequently decreases during contrastive training, adversely impacting performance on tasks such as coding and mathematical question answering [11, 12] . This paper addresses this limitation by calibrating contrastive objectives for preference learning and supports our approach with extensive theoretical guarantees in practice.There is a substantial body of work that analyzes contrastive preference learning methods from different perspectives. For instance, [23, 24] study the performance of DPO under distribution shift and show that DPO is more susceptible to out-of-distribution responses than PPO. [25] theoretically analyze DPO with noisy preferences. [26, 27, 28] revisit the training objective of DPO from the perspective of noise contrastive estimation [29] . [30] theoretically analyze the gradient vector field of DPO and show that DPO decreases the probability of disfavored responses faster than it increases the probability of generating preferred responses. [31, 32, 33, 34] derive DPO within a token-level MDP formulation. Concurrent work [10] shows that DPO employs a negative gradient to push down the likelihood of undesirable, e.g., rejected responses, and implicitly exhibits a "mode-seeking" behavior. In this paper, our results are complementary. Specifically, we introduce Cal-DPO, which explicitly calibrates the rewards learned by DPO to match the scale of ground-truth rewards, while exhibiting "mode-seeking" behavior by minimizing the reverse KL divergence similar to RLHF. Also worth mentioning is a body of work on calibration in supervised classification [35] , learningto-rank [36] , unsupervised learning [35] , and reinforcement learning [37, 38] . Recently, calibration has also been introduced into language models [39, 40] . In these works, calibration refers to the alignment of the model's assessed confidence scores with the likelihood of its responses being correct. In contrast, our focus is on calibrating the learned implicit rewards in contrastive preference learning to match ground-truth rewards in aligning language models with human preferences. 
notations and preliminaries: 
 Problem Setup. We consider the preference learning scenario as follows: let the text sequences x = [x 1 , x 2 , . . .] denote an input prompt, and y w = [y 1 , y 2 , . . .] and y l = [y 1 , y 2 , . . .] denote two responses, typically sampled from the reference policy π ref (y | x). The response pairs are then presented to human labelers (or an oracle) who express preferences for responses given the prompt, denoted as y w ≻ y l | x, where y w and y l denote preferred and dispreferred responses, respectively. The preference distribution is commonly expressed using a latent reward model r(x, y) as:EQUATIONwhere g : R → [0, 1] is a monotone non-decreasing function (with g(z) = 1 -g(-z)) that converts reward differences into winning probabilities). When g is the sigmoid function σ(x) = 1 1+e -x , we get the Bradley-Terry (BT) preference model [41] . Given dataset D, containing feedback (x, y w , y l ), the goal is to learn a LLM policy π θ (y | x) to align human preference by generating high rewards.RLHF. Typically, given the reward function r(x, y), which dictates the human preferences, RLHF optimizes policy π θ for x to maximize reward with the following RL objective: maxEQUATIONwhere β > 0 is an appropriate KL penalty coefficient. Due to the discrete nature of language generation, we typically optimize the RLHF objective in Equation (2) using RL algorithms, such as PPO [2, 5] . Although RLHF with PPO has achieved remarkable success, the training process of PPO is unstable because of the high variance of the policy gradient estimation [6] .Reward Modeling. One standard approach to reward modeling is to fit a reward function r ϕ (x, y) using the BT preference model in Equation (1) . Specifically, the reward function r ϕ (x, y) can be estimated by maximizing the log-likelihood of the preference feedback (x, y w , y l ):EQUATIONContrastive Preference Learning. To simplify RLHF, contrastive preference learning [42, 7, 9, 8] uses the log-likelihood of the learning policy to implicitly represent the reward function:EQUATIONWith this parameterization, DPO [7] aims to optimize π θ based on the BT model in Equation ( 3):EQUATIONTechnically, minimizing the DPO objective or any other pairwise contrastive objective, such as those of IPO [8] and SLiC [9] , essentially amounts to maximizing the relative reward differences between chosen and rejected responses as shown in [10] . However, as noted earlier, these pairwise ranking objectives are not scale-calibrated and ignore the absolute values of the rewards. Thus, the likelihood of the chosen response can continue to decrease during training as long as the relative difference in the likelihoods between the chosen and rejected responses remains large (see Figure 1 ). This property has resulted in suboptimal performance, especially on reasoning and mathematical problem-solving [11, 12] . In this paper, we address this limitation by proposing a simple yet effective calibrated objective to calibrate the behavior of contrastive preference learning methods such as DPO. 
calibrated direct preference optimization: 
 We proceed to introduce Calibrated Direct Preference Optimization (Cal-DPO), a simple and effective modification of DPO. The key intuition behind Cal-DPO is to calibrate the implicit reward function against the ground-truth rewards. Hence, Cal-DPO is designed to learn an implicit reward parameterized by the policy calibrated against the ground-truth reward. This can be achieved through a simple modification of DPO [7] , assuming, as in the case of DPO, that the preferences adhere to the BT preference model. Thus, Cal-DPO can be implemented on top of DPO with just one line of code and without any additional hyperparameters. In principle, our idea of calibration can be applied to any contrastive preference learning algorithm such as IPO [8] and SLiC [9, 17] (see Section 4.3).To fine-tune a policy on preference feedback, we propose learning a reward model implicitly and using the log-likelihood of the policy to represent the estimated reward [7, 9, 8] . Specifically, we start by defining a preference score of y w relative to y l , where the implicit reward is represented by π θ .EQUATIONHere r θ (x, y) = log π θ (y|x) π ref (y|x) is an implicit reward defined by the training and reference policies π θ and π ref , allowing us to express that, for any θ ∈ Θ, the preference probabilities can be denoted as:EQUATIONwhere we assume that preferences adhere to the BT preference model as in DPO [7] . With preference probabilities expressed in terms of the learning policy, we can find the maximum likelihood estimate by minimizing the preference optimization loss based on the preference feedback:EQUATIONWe can note that this pairwise loss is equivalent to the DPO objective (without β) in Equation (5) . As this contrastive pairwise loss is also not scale-calibrated, ignoring the absolute values of the rewards. Thus, we cannot guarantee that the estimated reward of the chosen response will increase and the reward of the rejected response will decrease, which tends to degrade the performance on math and reasoning benchmarks [11, 12, 43, 44] . We propose to address this limitation by explicitly constraining the implicit reward function log(π θ (y | x)/π ref (y | x)) to a scale that matches the ground-truth reward r(x, y)/β. Intuitively, if the learned implicit rewards are constrained to lie in the range of the ground-truth reward, we can prevent the rewards of chosen responses from continually decreasing. Formally, we define "calibration" with respect to ground-truth reward as follows:Definition 1. (Calibration). An estimated implicit reward log π θ (y|x) π ref (y|x) for the LM policy π θ is called scale calibrated with respect to the ground truth reward if log π θ (y|x)π ref (y|x) = r(x,y) β , ∀(x, y) ∼ D.Thus, in addition to the BT preference loss in Equation ( 8), we propose constraining the learned implicit reward to match the ground-truth reward through solving squared loss regression problems:EQUATIONThe calibration loss requires access to oracle rewards; however, in some scenarios, we may only have access to pairwise preference feedback. In such cases, we define the reward for preference feedback as follows: r(x, y w ) = 1/2 and r(x, y l ) = -1/2, indicating y w ≻ y l | x. Empirically, we find that this works quite well in practice. Combining this calibration loss with the BT preference loss in Equation ( 8), we get the following full loss of Cal-DPO on human preference data:EQUATION+ log π θ (y w | x) π ref (y w | x) - 1 2β 2 + log π θ (y l | x) π ref (y l | x) + 1 2β 2 ,where our modifications to the standard BT preference model are straightforward and depicted in blue. Intuitively, Cal-DPO learns an implicit reward parameterized by the LM policy that is "calibrated" against the ground-truth reward. Specifically, Cal-DPO attempts to push the reward of the chosen response toward 1/2β and the reward of the rejected response toward -1/2β, ensuring that π θ (yw | x) > π ref (y w | x) and π θ (y l | x) < π ref (y l | x). This alignment keeps the estimate of the implicit reward consistent with the ground-truth reward. Our implementation of Cal-DPO builds directly on the DPO codebase with just one additional line (see pseudocode in Appendix B.1). Our experiments show that calibration via a simple square loss can consistently improve the performance of off-the-shelf DPO, demonstrating the potential of calibration for preference fine-tuning.Next, we proceed to present a theoretical analysis of Cal-DPO. We show that Cal-DPO with reward calibration enjoys important properties that are desirable for fine-tuning LLMs with preferences, e.g., mode-seeking behavior, negative preference optimization ("negative gradient" property) to push down the likelihood of undesirable responses [10] . We also show that Cal-DPO minimizes an upper bound on the standard KL-regularized RLHF in Equation (2) . All proofs are provided in the Appendix 4.2.We start by presenting our theoretical framework from a distribution matching perspective. We first interpret the RLHF objective as the optimization of a reverse KL-divergence between π θ and π * [7] :EQUATIONwhere π * (y|x) ∝ π ref (y|x) exp(r(x, y)/β). The derivation is given in Appendix A.1. Since reverse KL can be difficult to optimize [7, 10] , one can instead optimize the forward KL divergence:EQUATIONwhich is the weighted maximum likelihood loss. Although this objective provides a straightforward approach for preference fine-tuning [45, 46, 47] , it leads to poor performance compared to RLHF, as shown by [10] . The poor performance of the preceding objective in Equation ( 12) relative to RLHF is due to the fact that it assigns a positive weight to all samples y for a given x. Since the rewards are always positive, the likelihood loss always tries to increase the probability of a response even if it receives a much smaller reward compared to other responses. We note that the recent work of [10] also empirically demonstrates that the maximum likelihood criterion lacks the "negative gradient" property, resulting in poor performance of MLE in Equation ( 12) compared to RLHF and DPO.Because KL-regularized RLHF and the MLE objective in Equation ( 12) are both population losses which involve the 'oracle' reward r(x, y), we also define a population loss for Cal-DPO to facilitate a direct comparison between Cal-DPO with KL-regularized RLHF and the MLE loss of Equation( 12): EQUATION✓ N/A ✗ ✗ DPO [7] & IPO [8] & SLiC [9, 17] ✓ ✗ ✓ ✗ RLHF (PPO) [2, 5] ✗ N/A ✓ ✓ Cal-DPO ✓ ✓ ✓ ✓Our proposed loss in Equation ( 10) is an empirical estimate of this population loss on preference feedback by sampling two responses, i.e., y w ≻ y l | x and setting a small β (see Appendix A.In what follows, we theoretically show that our Cal-DPO enjoys the "negative gradient" property.Theorem 1. Minimizing the first term in our Cal-DPO in Equation ( 13) is equivalent to minimizing the forward KL divergence, or equivalently MLE in Equation ( 12), while maintaining the following contrastive negative gradient with respect to π θ :EQUATIONw(x, y) = exp(r(x, y)/β)y π ref (y|x) exp(r(x, y)/β) and ŵ(x, y) = (π θ (y|x)/π ref (y|x)) y π ref (y|x)(π θ (y|x)/π ref (y|x)),This theorem establishes the relationship between the objectives of MLE and our Cal-DPO. Notably, the theorem shows that the first term in Cal-DPO also minimizes forward KL divergence with negative gradients. Specifically, we examine this update rule to illustrate how our objective produces a negative gradient [10] . If we sample response y, and w(x, y) -ŵ(x, y) is positive (which happens more often when the reward r(x, y) is high), the update rule will increase the log-probability of this response. This leads to an increase in the probability of generating a response with a high reward. Conversely, if w(x, y) -ŵ(x, y) is negative (which occurs more often when r(x, y) is low), we decrease the probability of response y, while increasing the probability of other responses due to normalization. This implies that Cal-DPO also exhibits a form of the "negative gradient" [10] .Theorem 1 shows that the first contrastive term in Cal-DPO minimizes forward KL divergence similar to MLE but with negative gradients. In practice, minimizing either KL divergence results in policies with distinct properties due to limited data coverage [48, 49, 50] . Specifically, forward KL D KL [π * ∥π θ ] promotes mode-covering behavior, whereas reverse KL D KL [π θ ∥π * ] encourages modeseeking behavior [10, 49, 51, 52] . In other words, forward KL encourages all responses in datasets to have equal probability, resulting in an overestimation of the long tail of the target distribution, whereas reverse KL sharpens the probability mass on certain high-reward regions. Thus, alignment commits to generating a certain subset of high-reward responses, which is more effectively realized by minimizing the reverse KL, as the RL objective in Equation ( 11) does. This also explains why MLE, which optimizes forward KL, performs worse than RLHF with reverse KL as shown in [10] .In what follows, we show that Cal-DPO with calibration loss also theoretically encourages modeseeking behavior by minimizing an upper bound of the RL objective in Equation ( 11):Theorem 2. Minimizing the Cal-DPO objective in Equation ( 13) with respect to π θ will encourage mode-seeking behavior by minimizing an upper bound of the reverse KL divergence, as RLHF does.EQUATIONThis theorem shows that, unlike DPO and MLE, Cal-DPO with calibration asymptotically minimizes a reverse KL divergence with respect to the policy, making it mode-seeking like RLHF. The first term, i.e., preference loss in Cal-DPO, corresponds to optimizing the forward KL as shown in Theorem 1. Hence, our proposed calibration loss can be understood as minimizing the gap between forward KL and reverse KL. Theorem 2 also implies that DPO, RLHF, and Cal-DPO asymptotically converge to the same global optimal policy in the limit given a sufficiently large dataset and model capacity. Thus, our calibration objective in Equation ( 9) is similar in some respects to RLHF. Table 1 presents a comparison of the different methods in terms of strengths and weaknesses compared to Cal-DPO.We observe that our approach to calibrating the learned implicit reward from preference feedback is not limited to the DPO algorithm or the BT preference model. As we shall see below, the general idea behind Cal-DPO extends to other methods such as IPO and SLiC and other preference models. Instead of the sigmoid loss in Equation ( 7), SLiC [9, 17] minimizes a pairwise hinge loss:EQUATIONIPO [8] is a contrastive algorithm similar to DPO and minimizes the following pairwise square loss:EQUATIONA potential advantage of SLiC and IPO over DPO is that they do not require that the preference model be BT and can work with general preference probabilities. By combining our calibration loss in Equation ( 9), it is straightforward to define the calibrated counterparts of both L SLiC and L IPO . Thus, our calibration approach can be generalized to work with SLiC and IPO as well (see Section 5.5). 
experimental setup: 
 Datasets. We evaluate Cal-DPO on four widely used datasets for preference fine-tuning: the UltraFeedback Binarized dataset [53, 54] , Reddit TL;DR summarization dataset [14] , Anthropic-HH dataset [1] , and the IMDb sentiment dataset [13] . Details of the datasets are provided in Appendix C.1.Tasks and Evaluation. Following previous work [7, 54] , we evaluate methods fine-tuned on the UltraFeedback Binarized dataset across general reasoning benchmarks (MMLU-PRO [55] , ARC [56] , IFEval [57] , BBH [58] , GPQA [59] ), and mathematical reasoning (GSM8K [60] and MATH [61] ).For training on the UltraFeedback Binarized dataset, we utilize the same chat template used in [44] for all methods. We also use AlpacaEval 2.0 [62] , a benchmark for assessing LLM alignment with human preference. The Anthropic HH dataset is used for dialogue generation to produce helpful and harmless responses [7] . For summarization, we use the Reddit TL;DR dataset. For the dialogue generation and summarization tasks, we use GPT-4 for zero-shot pairwise evaluation, which is consistent with human judgments (see prompts in Appendix C.2.1). In the IMDb sentiment dataset, the goal is controlled text generation to produce positive sentiments from movie review prefixes [7] .We train a binary sentiment classifier and define the oracle reward as its log odds, and evaluate the policy on the trained reward model [7] . The task and evaluation details are given in Appendix C.2.Models. For summarization and dialogue generation tasks, we use Pythia-2.8b [63] as our base model and the model after SFT as the reference model following [7] . For the IMDb controlled text generation task, both the policy and reward models are initialized from the GPT-2 Large model [64] .For tasks with the UltraFeedback Binarized dataset, we use the Zephyr-7b-sft model [54] as our base model to ensure alignment with previous work on LLM preference alignment [54] .Baselines We compare Cal-DPO with the following preference optimization methods: DPO [7] , IPO [8] , SLiC [9] , CPO [65] . We also compare Cal-DPO with other variants of DPO: f-DPO [66] , DPO-Positive (DPOP) [11] and DPO+NLL [67] which combine DPO loss over preference pairs and the negative log-likelihood (NLL) loss over chosen responses. We also compare with weighted MLE in Equation (12) . Besides DPO, we also implemented our calibration objective on top of SLiC [9] and IPO [8] (see Section 5.5). The implementation details are provided in Appendix B.1.Reasoning Benchmarks. Table 2 compares the performance of Cal-DPO against other alignment methods on the UltraFeedback Binarized dataset. Our results show that Cal-DPO exhibits remarkable effectiveness in enhancing DPO's performance. The improvements are particularly notable on the IFEval and Math benchmarks, with relative gains exceeding 63.1% and 12.5% compared to the best baseline, respectively. These findings underscore the efficacy of Cal-DPO. We hypothesize that these improvements can be attributed to the calibration of implicit rewards performed by Cal-DPO as part of its training objective. Without proper calibration, the likelihood of selected samples decreases, resulting in suboptimal performance, especially in mathematical reasoning tasks, where the chosen responses are very likely the ground-truth answers. Furthermore, Cal-DPO outperforms CPO, which combine DPO over preference pairs with the negative log-likelihood loss over the chosen response. We hypothesize that the superiority of Cal-DPO over CPO can be attributed to the conservative nature of the objectives optimized by the latter, which only affects the chosen response. In contrast, Cal-DPO calibrates implicit rewards for both chosen and rejected responses.Instruction-following Benchmarks. To assess the ability of Cal-DPO on align with human instruction, we compare the performance of Cal-DPO and DPO on AlpacaEval 2.0 [62] , an evaluator based on GPT-4 (version gpt-4-1106-preview) that produces the probability of preferring the evaluated model. Figure 2 shows the comparison in terms of both normal and length-controlled (LC) percentage of wins. We see that Cal-DPO demonstrates steady performance gains with the number of training iterations and outperforms SFT and DPO methods, which tend to produce longer responses. We also designed experiments to explore learning from real human preferences, focusing on summarization and dialogue generation tasks. Specifically, we used the Reddit TL;DR dataset for summarization and the Anthropic-HH dataset for dialogue generation. Table 3 summarizes the GPT-4 evaluation results. These results show that Cal-DPO demonstrates a notable improvement over DPO and its variants compared to both the SFT and the chosen responses. Remarkably, Cal-DPO aligns better with human preferences than baselines, achieving win rates of at least 60% against the chosen responses. This indicates that Cal-DPO shows strong promise in terms of aligning with human preferences. Additionally, we provide examples generated by both DPO and Cal-DPO for both tasks in Appendix D.1. These examples show that GPT-4 consistently prefers Cal-DPO over baselines and the chosen responses in the dataset, demonstrating that Cal-DPO significantly improves DPO in terms of helpfulness and harmlessness of the generated responses. We conducted experiments on the IMDB dataset to assess the generation of positive movie reviews. The task requires the model to provide positive and fluent completions of movie reviews based on given partial input texts. To perform a controlled evaluation, we trained a binary sentiment classifier on the IMDB dataset and defined the oracle reward as its log odds, following [7] . The reward score of the reward model then serves as an in-domain proxy for the unknown ground-truth reward used in evaluation. The results of IMDB sentiment generation are listed in Table 4 . We used the reward score of the reward model and the perplexity of GPT-2 [64] to assess alignment performance. We observe that (1) PPO, DPO, and Cal-DPO can align the SFT model with the preference of the reward model, as evidenced by the increasing reward score, and (2) Cal-DPO performs better in terms of reward score and perplexity than both PPO and DPO, which confirms our theoretical results that the policy trained by Cal-DPO can effectively maximize rewards.Training Dynamics. We also investigated the reward patterns during the training process of Cal-DPO. Figure 3 presents the reward patterns for Cal-DPO and DPO on the TL;DR summarization dataset. We observe that the rewards of the rejected data keep decreasing, and the margins between the chosen and rejected responses keep increasing. However, in the case of DPO, the rewards of the chosen responses fall below zero, whereas they continue to increase with Cal-DPO, underscoring the utility of reward calibration in LLM alignment. These results are similar to those on the UltraFeedback dataset shown in Figure 1 , verifying our motivation and the effectiveness of Cal-DPO.Generalization to other objectives. As mentioned in Section 4.3, our approach to calibrating the learned implicit reward from preference feedback generalizes in a straightforward manner to other pairwise preference optimization methods including IPO and SLiC. To show this, we implemented the calibration objective of Cal-DPO for IPO and SLiC, yielding their calibrated counterparts Cal-IPO and Cal-SLiC, respectively. Figure 3 shows the comparison on the Anthropic-HH dataset. We observe that combining our calibration objective can consistently improve standard OCPL methods for LLM preference alignment, demonstrating the broader utility of preferance calibration.Coefficient Parameter. We investigate the effect of the coefficient β and present an ablation study to analyze the performance of Cal-DPO on various tasks by varying β. Figure 4 in the Appendix shows the performance with different values of β. We observe that β plays an important role in Cal-DPO. A small β typically improves the model performance, whereas a too large β encourages the policy to remain close to the reference policy, leading to poor performance. 
conclusion and limitations: 
 We have presented Cal-DPO, a simple yet effective fine-tuning approach for aligning LLMs with human preference. Cal-DPO incorporates a simple preference calibration term that modifies the behavior of the objective of contrastive preference learning methods such as DPO. The key idea behind Cal-DPO is to ensure that the learned implicit rewards lie within the same range as the groundtruth rewards, yielding substantial gains in performance relative to the state-of-the-art baselines on several widely used benchmark data sets. We also demonstrate theoretically that Cal-DPO exhibits the desirable "negative gradients" and "mode-seeking" behavior.A limitation of Cal-DPO is that it is currently limited to offline methods and does not consider on-policy learning where the policy can interact with the reward model during learning. It would be interesting to explore how the reward calibration idea used in Cal-DPO performs in the on-policy learning scenario. We leave this as an interesting direction for future work. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: We provide the experiment results that support the main claims of the paper. Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [NA]Justification: The paper does not use existing assets.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: A.1 Derivations of Equation (11) We provide a detailed derivation of Equation (11) , demonstrating that the RLHF objective in Equation (2) optimizes a reverse KL-divergence.EQUATIONEQUATIONEQUATIONEQUATIONNote that the last line is exactly the RLHF objective with a negative sign. Therefore, maximizing the RLHF objective with respect to π θ is equivalent to minimizing the reverse KL-divergence. Because optimizing the reverse KL-divergence is mode-seeking, RLHF exhibits mode-seeking behavior.A.2 Derivations of Equation (13) In this section, we show that our proposed loss in Equation ( 10) is an empirical approximate estimation of the population loss in Equation ( 13) based on sampled preference feedback, i.e., y w and y l :EQUATIONUsing preference feedback y w and y l sampled from π ref (y | x) to approximate the above expectation gives us the following empirical estimation:LCal-DPO (θ) = -exp(r(x, yw)/β) exp(r(x, yw)/β) + exp(r(x, y l )/β) log π θ (yw|x)/πref(yw|x) π θ (yw|x)/πref(yw|x) + π θ (y l |x)/πref(y l |x) -exp(r(x, y l )/β) exp(r(x, yw)/β) + exp(r(x, y l )/β) log π θ (y l |x)/πref(y l |x) π θ (yw|x)/πref(yw|x) + π θ (y l |x)/πref(y l |x)EQUATION)Given pairwise preference feedback y w ≻ y l | x with reward r(x, y w ) = 1/2, r(x, y l ) = -1/2, setting the coefficient β → 0 (β is typically small and set as 0.001 in our experiments) gives us the following approximation (softmax weight becomes hard argmax weight):EQUATION)This is equivalent to the loss proposed in Equation (10) .Theorem 1. Minimizing the first term in our Cal-DPO in Equation ( 13) is equivalent to minimizing the forward KL divergence, or equivalently MLE in Equation (12), while maintaining the following contrastive negative gradient with respect to π θ :EQUATIONw(x, y) = exp(r(x, y)/β)EQUATIONProof. Recall that the first term in the population objective of Cal-DPO in Equation ( 13) is: L1(θ) = -E y∼π ref (Note that the second term in the last line does not depend on π θ . Therefore, minimizing the first term of Cal-DPO with respect to π θ is equivalent to optimizing the forward KL-divergence, as L MLE does.We know from the previous result that L 1 and L MLE have the same optimal. However their gradients differs. We first take the derivatives of L 1 with respect to π θ :EQUATIONEQUATIONEQUATIONEQUATIONEQUATION=E y∼πref(y|x) (w(x, y) -ŵ(x, y)∇ θ log π θ (y|x) .Similarly, we can take the derivatives of L MLE (θ) in Equation 12with respect to π θ :EQUATIONEQUATIONComparing Equations ( 37) and ( 39), we can observe that our Cal-DPO employs a negative gradient to push down the likelihood of bad responses under the learned policy compared to MLE.Theorem 2. Minimizing the Cal-DPO objective in Equation ( 13) with respect to π θ will encourage mode-seeking behavior by minimizing an upper bound of the reverse KL divergence, as RLHF does.EQUATIONProof. Recall that Cal-DPO contains two terms: the contrastive term and the calibration term:EQUATION)We begin by exploring the distinction between forward and reverse divergences. Before that, we introduce a few definitions and some background on Bregman divergence. For any two points r ∈ F and s ∈ F, the Bregman divergence D F , specified by the convex differentiable potential function F : F → R, is defined as follows [68, 69] :EQUATIONwhere f = ∇F , F ( p) is the convex conjugate of F , p = f (r), and q = f (s) [70] . By defining F (s) = log y exp(s(y)) as the Log-Sum-Exp operator, f (s) = s(y) y exp(s(y)) as the Softmax operator, and F (p) = -H(p), we get the KL divergence between two probability vectors q and p:EQUATION)By using Taylor expansions of F (p), we further have the following:EQUATIONwhere H F denotes the Hessian of F , and a = (1 -α/2)r + sα/2, (0≤ α ≤ 1/2), b = (1 -λ/2)s + rλ/2, (0 ≤ λ ≤ 1/2).It is well known that Hessian has the following form:EQUATIONTaking the above and Equation ( 44), we obtain the following:EQUATION) Plugging p = π * (y | x) = π ref (y|x) exp(r(x,y)/β) Z(x), q = π θ (y | x), s = log π θ (y | x), and r = log π ref (y | x) + r(x, y)/β into Equation ( 46) obtains the following:EQUATIONEQUATION)Theorem 1 demonstrates that the first contrastive term in Cal-DPO is equivalent to optimizing the forward KL-divergence. Consequently, we can directly obtain:EQUATIONwhich completes the proof.The pseudocode for Cal-DPO is provided in Algorithm 1. For the general hyperparameters, we closely followed the configurations used in [71] . The β of Cal-DPO is searched from [1e-3, 2e-3, 3e-3, 1e-2, 1e-1], the batch size for all methods is 128, and we use the RMSprop optimizer with a learning rate of 5e-6. We linearly warm up the learning rate from 0 to 5e-6 in 150 steps. The sampling temperature is set to 1 for all experiments. The experiments on are run on 4 Nvidia A100 GPUs with BF16 precision. We thoroughly tune the hyperparameters for each baseline following [44] .In this section, we provide detailed descriptions of datasets:UltraFeedback Binarized [53, 54] : This dataset1 consists of 64k prompts, where each prompt is accompanied by four model completions from a wide variety of open and proprietary models. GPT-4 is then used to assign a score to each completion based on criteria like helpfulness and honesty. It def loss(chosen_pi_logps, chosen_ref_logps, rejected_pi_logps, rejected_ref_logps, beta): """ chosen_pi_logps: policy logprobs for the preferred responses, shape (B, ) chosen_ref_logps: reference logprobs for the preferred responses, shape (B, ) rejected_pi_logps: policy logprobs for the dispreferred responses, shape (B, ) rejected_ref_logps: reference logprobs for the dispreferred responses, shape (B, ) beta: the parameterization coefficient that defines the residual model """ chosen_reward = chosen_pi_logps -chosen_ref_logps reject_reward = rejected_pi_logps -rejected_ref_logps dpo_losses = -F.logsigmoid(chosen_reward -reject_reward) # our method requires a simple modification on DPO with one additional line of code cal_losses = F.mse_loss(chosen_reward, 0.5*beta) + F.mse_loss(reject_reward, -0.5*beta) cal_dpo_losses = dpo_losses + cal_losses return cal_dpo_losses constructs binary preferences from UltraFeedback by selecting the highest mean score as the "chosen" response and one of the remaining three at random as the "rejected" response.: This Anthropic Helpful and Harmless dialogue dataset2 contains 170k dialogues between a human and an automated assistant. This dataset was utilized for assessing single-turn dialogue performance. Each of the 170k dialogues comprises a human query and paired model responses rated for helpfulness and harmlessness. Following DPO [7] , the preferred responses from this dataset were utilized for the supervised Fine-Tuning (SFT) phase, aligning the initial model behavior with desirable conversational outcomes.Reddit TL;DR summarization [14] : This dataset3 is a compilation of forum posts from the popular social media platform Reddit, specifically curated for summarization tasks with preference labels. Following [3, 72] , we use the same filtered version of this dataset to train the SFT policy and use their preference labels for the following alignment stage.IMDB Sentiment [13] : This dataset4 contains movie reviews from the IMDb with positive and negative sentiment, which contains 25k training samples and each 5k samples for validation and test.We evaluate methods fine-tuned on the UltraFeedback Binarized dataset across reasoning benchmarks (MMLU-PRO [55] , ARC [56] , IFEval [57] , BBH [58] GPQA [59] ), and mathematical reasoning (GSM8K [60] and MATH [61] ) provided by the Language Model Evaluation Harness library [15] .This task is a set of grade-school science questions.Massive Multitask Language Understanding Professional (MMLU-PRO): MMLU-PRO is an enhanced version of the MMLU dataset [61] , addressing previous shortcomings by increasing choice options in multiple-choice questions and refining question quality through expert review, making it more challenging and less prone to data contamination.Instruction-Following Evaluation (IFEval): IFEval is a benchmark evaluating a model's ability to follow explicit instructions, emphasizing adherence to formatting over content generation. Big Bench Hard (BBH): BBH is a selection of 23 challenging tasks from the BigBench, focusing on areas like multistep arithmetic, algorithmic reasoning, language understanding, and world knowledge.Graduate-Level Google-Proof Q&A (GPQA): GPQA is a challenging benchmark composed of advanced questions developed by PhD-level experts across various fields like biology, physics, and chemistry.GSM8K: This dataset consists of diverse grade school math word problems to measure a model's ability to solve multi-step mathematical reasoning problems.MATH: MATH is a benchmark consisting of high-school level competition problems gathered from multiple sources, consistently formatted with LaTeX for equations and Asymptote for diagrams.AlpacaEval 2.0: We also use AlpacaEval 2.0 [62] , a benchmark to assess model fine-tuned on UltraFeedback Binarized dataset with human preference.We consider the following tasks to evaluate the model fine-tuned on real human preference data.Single-Turn Dialogue: We employ the Anthropic HH dataset, which includes 170,000 interactions between humans and virtual assistants. Our model is optimized to deliver useful and accurate answers across a range of queries while deliberately avoiding any harmful content.Summarization: We leverage the Reddit TL;DR dataset, where the prompt represents a Reddit forum post and the preference pairs are collected by previous works. The task objective is to generate a summary that captures the main points of the post.For single-turn dialogue and summarization tasks, we query GPT-4 for zero-shot pair-wise evaluation, which has been shown to be consistent with human judgments.Controlled Sentiment Generation: We use the IMDb dataset [13] , which contains 25,000 reviews, where the prompt is a prefix of a movie review and the preference pairs are generated following previous works. The task objective is to generate a positive sentiment y based on the movie review prefix. To perform a controlled evaluation, we train a binary sentiment classifier on the IMDb dataset and define the oracle reward as its log odds, following [7] . The reward score of the reward model then serves as an in-domain proxy for the unknown ground-truth reward used for evaluation. The results of IMDb sentiment generation are listed in Table 4 . We used the reward score of the reward model and the perplexity of GPT-2 [64] to demonstrate the performance of alignment.For the generation, summarization, and dialogue tasks, we utilize GPT-4 for zero-shot pair-wise evaluation, which is shown to be consistent with human judgments. We use gpt-4-0314 for all our experiments. The specific prompts used for GPT-4 evaluation are detailed below.Figure 5 shows the performance of SLiC and IPO, and their calibrated counterparts Cal-IPO and Cal-SLiC by applying our proposed calibration objective on the Anthropic-HH dataset. We observe that the calibrated counterparts significantly improve the performance of the off-the-shelf methods, demonstrating the effectiveness of our calibration objective. Figure 6 provides additional results of the training dynamics of DPO and Cal-DPO on the Anthropic-HH and IMDb datasets. These results show the robustness of Cal-DPO to different methods and datasets for preference fine-tuning Human: What are the differences between a hamster and a gerbil? Assistant: Hamsters are generally larger than gerbils, and they're also more active and playful. They both make good pet choices, and they can each be taught simple tricks. Hamsters can be bought in many different sizes and colors, and gerbils are usually sold in white and brown, or some similar combination of colors. They can also be bought in different breeds. Human: Do they eat similar foods and stay in similar types of cages? Assistant: Yes, you can buy rodent cages in various sizes for both hamsters and gerbils. They can both eat healthy fresh vegetables and fruits, as well as several types of high-quality pet food. If you want to make sure they stay healthy and active, you might want to feed them a high-quality wet food once or twice a day, as well as some fresh vegetables, and you might want to make sure their cages are large enough for them to run around and play in.Neither hamsters nor gerbils are picky eaters. Human: Do they both like to run on wheels? Assistant: Yes, hamsters and gerbils both tend to like to run on wheels. Some types of hamsters like to curl up in balls inside their cages, and some types of gerbils like to run on wheels. So it's a good activity for both of them. I would recommend picking a cage that has at least two large wheels and one smaller wheel for each of them, as well as some blank wooden ramps or other objects they can run around on. I would recommend picking wheels with holes in the middle to give them something to run around on, and also some more solid wheels they can run in circles on. Human: What are their average lifespans? Assistant:Hamsters generally live between two and four years, and gerbils can live between two and four years. They both have a good chance of living to be around ten years old. They both have a good chance of living to be around ten years old.Cal-DPO The average lifespan for hamsters is around 2 to 3 years, and the average lifespan for gerbils is around 4 to 5 years. They both tend to live longer than most other pets, but they also tend to die from various causes, including old age, illness, and accidents.Judgment Comparison: Response 2 provides more accurate and realistic information about the lifespan of hamsters and gerbils, while Response 1 inaccurately suggests that both can live up to ten years, which is misleading. More helpful: 2 • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material.https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback_binarizedhttps://huggingface.co/datasets/Anthropic/hh-rlhfhttps://huggingface.co/datasets/openai/summarize_from_feedbackhttps://huggingface.co/datasets/stanfordnlp/imdb 


Decision: This paper introduces calibrated direct preference optimization (Cal-DPO), a preference-tuning algorithm that modifies DPO to address the limitation of ignoring the actual values of implicit rewards. The approach modifies DPO loss by adding two MSE terms that aim to "calibrate" the log-likelihood ratios of y_w and y_l to their respective reward values. The paper evaluates by training the Zephyr 7B model on Cal-DPO and DPO, and comparing the results on a few different benchmarks. Cal-DPO generally seems to outperform DPO and other related variants. The method seems easy to implement, showing good practical values. The paper is generally well-written. The work can be further improved by making the motivation and intuition of some technical choices clearer, and by adding empirical studies on more base models.

Reviews: review: paper_summary: paper_summary: This paper proposes a simple yet effective method called calibrated direct preference optimization (Cal-DPO), which addresses the limitation of ignoring the actual values of implicit rewards. The authors demonstrate the theoretical advantages of Cal-DPO over existing approaches and show the effectiveness on a variety of standard benchmarks.
strength_weakness: strength_weakness: 1. It is still not intuitive for me why the scale of the reward's actual value plays an important role in the generation performance. The proposed method seems to control this scale via an additional regularization term (Equation 10) and improve the empirical generation performance. But the relationship between them lacks explanations.

2. The authors should further justify why they choose the squared loss to constrain the learned implicit reward theoretically or empirically. Although they analyze some theoretical properties of the loss function, I believe that the squared loss is not the only form to possess these properties.

3. Equation 10 is over-simplified when directly assigning $r(x,y_w)%$ and $r(x,y_l)%$ to 1/2 and -1/2, respectively. The authors should provide intuitions and show whether the method is robust by assigning other values.
questions: questions: I have included my questions in the weaknesses part.
limitations: limitations: The authors have adequately addressed the limitations.
paper_summary: paper_summary: The authors propose Cal-DPO, a preference-tuning algorithm that modifies the DPO loss by adding two MSE terms that aim to "calibrate" the log-likelihood ratios of y_w and y_l to their respective reward values. They claim that this is advantageous because the DPO loss only maximizes the reward ratio, and does not constrain the respective reward values. They also claim that Cal-DPO exhibits a "negative gradient" (it pushes down the likelihood of undesirable outputs) and mode-seeking behavior. They evaluate their methods by training the Zephyr 7B model on Cal-DPO and DPO, and comparing the results on a few different benchmarks. Cal-DPO generally seems to outperform DPO and other related variants.
strength_weakness: strength_weakness: - **Unfair comparisons to other methods**: The authors show in Figure 4 that the best $\beta$ value for Cal-DPO is 0.001, and they choose to use this same value for their comparisons to all other methods (described in Appendix B.1, where they state that "all the hyperparameters are set to be the same"). However, this implies that hyperparameter tuning was conducted for Cal-DPO, but not for any of the other methods. In fact, past work indicates that the optimal $\beta$ value can vary widely depending on the combination of model and algorithm, in some cases reaching as high as 0.6 (see https://huggingface.co/blog/pref-tuning). The difference in accuracy/performance can be vast, depending on the hyperparameter configuration. In general, the fairer way to compare methods is to conduct a hyperparameter search separately for every method, rather than to fix a set of hyperparameters. This is especially important in this case, since the $\beta$ was already selected to be the optimal one for Cal-DPO. Without this procedure, it is unclear whether Cal-DPO truly outperforms the other methods, or if the other methods were sub-optimally trained.
- The authors train the model on a set of training datasets covering safety behaviors, summarization, and sentiment detection, but the evaluation benchmarks in Table 2 primarily cover other types of capabilities, such as math and abstract reasoning. These are not preference learning tasks, as they have exact answers, rather than subjective ratings of human preference. Even if a couple other papers have done this evaluation, this does not make as much sense as evaluating on more related benchmarks, such as MT-Bench. The Cal-DPO accuracy numbers in this table are also quite close to the other methods' -- and given the hyperparameter tuning issues, it is unclear how much of this difference is noise or is meaningful.
- The authors predicate much of their rationale on the claim that other techniques like DPO reduce the likelihood of the chosen response. However, since DPO also reduces the likelihood of the rejected response, it is unclear whether this effect is actually undesirable -- to answer this question, one would have to know where the probability mass gets moved to instead. Since we know that DPO does in fact improve win rate, it seems likely that there is probability mass moved to more preferred outputs. The authors also do not cite any evidence that this is necessarily undesirable.
    - Additionally, not all y_w's in preference datasets are high-quality -- it is often the case that both y_w and y_l are low-quality, but y_w is slightly less low quality than y_l. In this case, it is not necessarily desirable for the model to increase the probability mass on y_w.
   - Similarly, the authors state that "an undesirable consequence of this behavior is that the learned policy increases the likelihood of unknown out-of-distribution responses," but this is also unfounded. If there is past work or experimental evidence that supports this claim, it would be helpful to cite it here.
- The authors also claim that mode-seeking behavior is more desirable here -- however, one can also see cases where this is undesirable. Since mode-seeking encourages more probability mass to be placed on high-reward outputs, and DPO/Cal-DPO rewards are based off the current model's likelihood, this promotes a positive feedback cycle where the algorithm continuously places more mass on outputs that are already high-likelihood under the current model. This may cause fast reward over-optimization, which has been a frequently observed issue in RLHF-based algorithms.
- The experiments are only conducted on the Zephyr 7B model, and it would be much more convincing if the results reproduced to a couple other sizes and types of LLMs as well.
- The paper also does not seem to list the generation parameters used for any of the evaluation experiments.

Minor nits:
- The stated IPO objective in Eq. 18 is incorrect -- the constant term should be $\frac{1}{2\tau}$, not $\frac{1}{2}$. Also, there is no $\beta$ coefficient. Alternatively, one could rename $\tau$ as $\beta$ instead and write $L_{IPO}=(h_{\theta}(x,y_w,y_l)-\frac{1}{2\beta})^2$.
- What are the "Value"s on the y-axis in the left two plots of Figure 3?
questions: questions: - Do the authors have results where the baseline methods have also been hyperparameter tuned?
limitations: limitations: The stated limitations about Cal-DPO being limited to offline methods is reasonable. An additional limitation is that the experiments have only been conducted on the Zephyr 7B model, and evaluated for preference learning on only a couple of small datasets (e.g. TL;DR summarization and Anthropic HH).
paper_summary: paper_summary: This paper proposes a simple yet effective change to the DPO objective that acts as a regularizer on the implicit reward, $\beta \log \frac{\pi(y \mid x)}{\pi_\mathrm{ref}(y \mid x)} + \beta \log Z(x)$, that is maximized by the LM. Specifically, the implicit reward is encouraged to be appropriately scaled relative to the desired reward $r(x, y)$. The main change is to add an additional squared loss term to the loss function that encourages the implicit reward for positive examples to concentrate around $\frac{1}{2 \beta}$ and the implicit reward for negative examples to concentrate around $\frac{-1}{2 \beta}$. The paper also includes some theoretical analysis, and positive empirical results.
strength_weakness: strength_weakness: My main difficulty is in the presentation of the motivation behind the method. In particular, I don't think that "calibration" is the right term to be using here --- it's a bit of a misnomer as it doesn't seem to have much to do with the standard way calibration is referred to (e.g., one might say that the BT model is calibrated if it produces calibrated preference probabilities). See also questions below.

I also think that the theoretical results can benefit from being stated a bit more precisely---in particular the comment following Theorem 2  (i.e., "Theorem 2 also implies that DPO, RLHF, and Cal-DPO asymptotically converge to the same global optimal policy ....") should be stated clearly and proven. 

There are also a number of other grammatical / writing errors (line 197, line 355, etc) that can be cleaned up.

From the empirical side of things, I think that it would be good to also compare to IPO, as the style of loss is quite similar (e.g., this loss also enforces the margin between the implicit reward of positive / negative examples to be a constant, though without constant differences). It would be good to show the effect.
questions: questions: If I understand the main motivation correctly, Eq. (9) is essentially saying that the log partition term for the optimal policy should be 0 in this "calibrated" version? Or, in other words, one is simply defining a new, translated reward via the equality:

$$\pi^*(y \mid x) = \frac{1}{Z(x)} \pi_\mathrm{ref}(y \mid x) \exp( \frac{1}{\beta} r(x, y)) \Rightarrow \pi^*(y \mid x) = \pi_\mathrm{ref}(y \mid x) \exp( \frac{1}{\beta} r'(x, y)),$$

where $r'(x, y) = r(x, y) - \beta \log Z(x)$. This can then be arranged in the usual way to get $\log \frac{\pi^*(y \mid x)}{\pi_\mathrm{ref}(y \mid x)} = r'(x, y) / \beta$. Setting $r'(x, y) = \pm \frac{1}{2}$ is then equivalent to adding $\beta \log Z(x)$ to the original reward (assuming it was $\pm \frac{1}{2}$), which still preserves $\pi^*(y \mid x)$, as that term does not depend on $y$ . However, enforcing this equality (instead of just the difference equality), then removes the underdetermined aspect of the BT model, which I can imagine is helpful (in addition to the theoretical properties w.r.t. gradient dynamics analyzed).
limitations: limitations: Yes
paper_summary: paper_summary: This paper proposed Cal-DPO, a variation of DPO to address the issue of decreasing rewards of chosen answers. In addition to DPO loss, Cal-DPO add a pair of calibration terms, which aim to match the rewards induced by language model with some absolute ground truth reward value. Theoretical analysis shows that, during the training process of Cal-DPO, the likelihood of chosen responses will likely to increase and the likelihood of rejected responses will likely to decrease. The authors also prove that DPO loss can be upper bounded by Cal-DPO loss. Experiment on various benchmarks shows that Cal-DPO outperforms DPO and other baselines.
strength_weakness: strength_weakness: This paper does not have specific weakness.
questions: questions: 1. Have the authors conducted experiment with models other than zephyr-7b-sft-full since zephyr-7b-sft-full demonstrate a relatively inferior performance on OpenLLM Leaderboard (which includes all reasoning benchmarks considered in the paper). Also, have the author conducted experiment on the other two benchmarks in OpenLLM Leaderboard (MMLU, TruthfulQA)?
limitations: limitations: The limitation is adequately clarified by the authors

Comments: title: Thank you very much for reading our response
Further response and important clarifications to Reviewer s9Bq
Dear NeurIPS Reviewer EJ4r: we understand that you maybe busy, so we would greatly appreciate it if you could check out our rebuttal.
Thank you for your comments
Dear NeurIPS Reviewer f8MD, thank you very much for reading our rebuttal.
Dear NeurIPS Reviewer yAWB, discussion period is ending soon
Dear NeurIPS Reviewer f8MD, discussion period is ending soon
Thank you very much for your reply
Official Comment
Dear NeurIPS Reviewer s9Bq, there are indeed misunderstandings and discussion period is ending soon
Sincere Comments and Clarifications for Reviewer s9Bq: Our comparisons are indeed fair.
comment: Dear NeurIPS Reviewer EJ4r

Thank you very much for carefully reading our response. We are very glad to hear that our rebuttal has addressed your concerns and we believe that the your insightful comments can be easily and effectively addressed in the final version.

We sincerely appreciate your time and effort.
Thanks for your rebuttal, which solves most of my concerns. I will raise my overall score.
Dear NeurIPS Reviewer s9Bq,

Here, we would like to provide a detailed and further response to your comments for our rebuttal.

### **R1. Our reported DPO results are indeed optimal, and the comparison to our Cal-DPO is fair.**

- The main and only purpose of Figure 4 is to demonstrate the sensitivity of Cal-DPO to hyperparameters; importantly, we indeed did not conduct a hyperparameter search for Cal-DPO to cherry-pick parameters that would enhance performance in the main results table. **Furthermore, we have provided the source code to facilitate reproducibility.**

- Moreover, as shown in many recent works [2,5,6]—which state that extensive hyperparameter searches were conducted—our reported DPO results are close to, and even better than, their reported results (as the results may still vary due to differences in hardware configurations and CUDA versions, etc. as shown in SimPO [10]). There is strong evidence that our reported DPO is near optimal, although there is a trade-off between different benchmarks.

|   | GSM8K  | ARC | Winogrande | HellaSwag |
| --- | --- | --- | --- |--- |
| DPO 1.0    |  25.34   | 57.96  | &nbsp; 71.64 | 81.28 | 
| DPO 0.5    | 27.12   | 58.41    | &nbsp; 73.59 | 81.95 | 
| DPO 0.1      | 33.51   | 60.34  |  &nbsp; 74.11 | 83.10 | 
| DPO 0.01     | 34.36   |  61.53   |  &nbsp; 75.18 |  83.67 | 
|  **DPO 0.001 (reported in our paper)**  | 35.41   |    62.02   |  &nbsp; 76.22 | 84.51 | 
| **DPO  (reported in SimPO [2])**  | 21.76  |   61.26   |   &nbsp; 76.80 | 83.59 | 
|  **DPO  (reported in Zephyr [5]**  | &nbsp; -  |   62.03   |  &nbsp; - | 84.52 | 
|  **DPO  (HuggingFaceH4/zephyr-7b-beta reported in LLM Leadboard [6]**  | 29.04  |   62.03   |  &nbsp; 77.74 | 84.36 | 
| Cal-DPO 0.001  | **40.34**   |  **64.34**   |   &nbsp; **78.54**  |  **85.33** |

---
### **R2. We would like to provide further responses and clarifications on reward over-optimization.**

First, **we agree with the reviewer that addressing the challenge of over-optimization is a promising future direction. However, this topic indeed extends beyond the scope of one paper. We will ensure to include this discussion  in our final version.**

In addition, the observed over-optimization in Figure 1 in [4] suggests that an additional increase in the KL budget leads to decreased model performance. **We refer to the original sentences in [4]:** "*This indicates that under the standard DAA training pipeline, decreasing likelihoods are not necessarily an issue for performance and are even necessary for improvement, but they exhibit non-linear over-optimization dynamics.*" **Thus, we believe there is no evidence to support that continually increasing and choosing positive likelihoods will result in over-optimization; rather, it is the decreasing likelihoods that exhibit non-linear over-optimization dynamics.**

---
### **R3. We would like to provide further responses and clarifications on motivation.**

 **As shown in many recent and concurrent works [2,7,8,9], DPO and other preference optimization methods can not effectively increase the likelihood of preferred sequences despite increasing the reward margin. This phenomenon generally decreases downstream task performance, particularly on reasoning-heavy tasks [2,7,8,9]]**. The key intuition behind our Cal-DPO is very simple yet effective: If the implicit reward estimates from preference data are well calibrated relative to the actual ground-truth rewards, **we can prevent the reward (likelihood) of chosen responses from continually decreasing while ensuring that the learned policy theoretically converges to the optimal. Specifically, Cal-DPO pushes chosen rewards to be as large as 1/2 and rejected rewards to be as small as −1/2.**


---
[1] https://huggingface.co/blog/pref-tuning.

[2] SimPO: Simple Preference Optimization with a Reference-Free Reward. In Arxiv.

[3] Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models. In ICML.

[4] Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms. Arxiv 2024.

[5] Zephyr: Direct Distillation of LM Alignment. Arxiv 2023.

[6] https://huggingface.co/spaces/open-llm-leaderboard-old/open_llm_leaderboard. 

[7] Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive. Arxiv 2024

[8] 3D-Properties: Identifying Challenges in DPO and Charting a Path Forward. Arxiv 2024

[9] Iterative Reasoning Preference Optimization. Arxiv 2024

[10] https://github.com/princeton-nlp/SimPO
Dear reviewers,

The discussion period will end soon. If you haven't responded to the authors' rebuttal, please do so and kick off the discussion. 

Best, 
SAC
Dear NeurIPS Reviewer EJ4r

Regarding the initial review from reviewer EJ4r, we just want to reiterate that there are very clear-cut answers to every question and misunderstandings that was raised, and our rebuttal has carefully addressed each point-by-point.

**The reviewer's comments are mainly about some clarifications and misunderstandings and are indeed not fatal to the contributions of our manuscript; we believe that the reviewer's insightful comments can be easily and effectively addressed in the final version. We would be grateful if the reviewer could increase the score.**

Many thanks for your time; we are extremely grateful.

The authors of "Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment"
Dear NeurIPS Reviewer EJ4r,

We gratefully appreciate your time in reviewing our paper and your insightful comments.

**We made our greatest efforts to address your concerns in the rebuttal.  The reviewer's comments are mainly about some clarifications and misunderstandings and are indeed not fatal to the contributions of our manuscript;** We would appreciate it if you could consider increasing your score. 

Thank you very much once again; we are extremely grateful.

Best regards
Dear NeurIPS Reviewer f8MD,

Thank you very much for reviewing our paper and reading our rebuttal. We sincerely appreciate your recognition of our contribution! 

We are truly grateful for your time and your reply.
Thanks for the thorough response to my comments. I have raised my score from 6 to 7 as I do believe my concerns can be addressed as discussed in another revision.
Dear NeurIPS Reviewer yAWB,

We gratefully appreciate your time in reviewing our paper and your comments.

We have made extensive efforts to address your comments and believe that they adequately address all your concerns. The reviewer's comments are mainly about some clarifications and are not fatal to the contributions of our manuscript; we believe that the reviewer's insightful comments can be easily and effectively addressed in the final version.

We would like to confirm whether there are any other clarifications and would be grateful if the reviewer could increase the score.

Many thanks for your time; we are extremely grateful.
Dear NeurIPS Reviewer f8MD,

We gratefully appreciate your time in reviewing our paper and your comments.

We have made extensive efforts to address your comments and believe that they adequately address all your concerns. The reviewer's comments are mainly about some clarifications and are not fatal to the contributions of our manuscript; we believe that the reviewer's insightful comments can be easily and effectively addressed in the final version.

We would like to confirm whether there are any other clarifications and would be grateful if the reviewer could increase the score.

Many thanks for your time; we are extremely grateful.
Dear NeurIPS Reviewer s9Bq,

Thank you very much for reading our response. Thank you also for your additional comments to facilitate further discussion!

**Q1. I am still concerned by the fact that the rebuttal contradicts what is stated in the paper.**

**A1.**  We would like to provide further responses and clarifications on your misunderstandings:

- **The main and only purpose of Figure 4 is to demonstrate the sensitivity of Cal-DPO to hyperparameters**; importantly, we indeed did not conduct a hyperparameter search for Cal-DPO to cherry-pick parameters that would enhance performance in the main results table. Furthermore, we have provided the source code to facilitate reproducibility. 
- **Moreover, as shown in the Table 9 in SimPO [2]—which states that they conducted extensive searches on baselines (DPO, IPO, SLiC)—our reported baseline results  are even better than theirs.**
- **As indicated by the results in the Global Response,  $\beta=0.001$  is the optimal parameter for DPO on the Zephyr-7b-beta-sft model with the ultrafeedback-binarized dataset,. This is consistent with previous study [1], which demonstrate that a small $\beta$ leads to better results. Thus our comparisons with other methods are fair since the base methods are optimally configured.**

**Q2. Reward over-optimization in direct alignment algorithms does not refer to the trend of the model's likelihoods on the offline dataset -- it refers to the trend of the on-policy win-rate during training, as shown in Fig. 1 of the source that you cited ([2,3]). There is no evidence provided that Cal-DPO indeed avoids this issue.**

**A2.**  We would like to provide further responses and clarifications. **The observed over-optimization in Figure 1 in [4] suggests that an additional increase in the KL budget leads to decreased model performance. We refer to the original sentences in [4]:** "*This indicates that under the standard DAA training pipeline, decreasing likelihoods are not necessarily an issue for performance and are even necessary for improvement, but they exhibit non-linear over-optimization dynamics.*" **Thus, we believe there is no evidence to support that continually increasing and choosing positive likelihoods will result in over-optimization; rather, it is the decreasing likelihoods that exhibit non-linear over-optimization dynamics.**

**We agree with the reviewer that addressing the challenge of over-optimization is a promising future direction. However, this topic indeed extends beyond the scope of one paper. We will ensure to include this discussion  in our final version.**


[1] https://huggingface.co/blog/pref-tuning.

[2] SimPO: Simple Preference Optimization with a Reference-Free Reward. In Arxiv.

[3] Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models. In ICML.

[4] Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms. Arxiv 2024.
Thank you for your responses to my review.

I am still concerned by the fact that the rebuttal contradicts what is stated in the paper. In L585-586, the text says "All the hyperparameters are set to be the same for DPO and Cal-DPO for a fair comparison" and in the same paragraph, "Unless specified otherwise, the default parameterization coefficient $\beta$ is 0.001, the batch size is 64, and we use the RMSprop optimizer with a learning rate of 5e-6." Nowhere are there listed other hyperparameters for Cal-IPO and Cal-SLiC, and Figure 4 clearly shows that Cal-DPO was tried with multiple different $\beta$ values, which contradicts the part of the rebuttal that says "We chose to use $\beta=0.001$ for Cal-DPO without conducting further hyperparameter searches."

**"In contrast, Cal-DPO maintains increasing and positive chosen likelihoods, thus actually avoiding the rapid reward over-optimization seen with standard DPO."**
Reward over-optimization in direct alignment algorithms does not refer to the trend of the model's likelihoods on the offline dataset -- it refers to the trend of the *on-policy win-rate* during training, as shown in Fig. 1 of the source that you cited ([3]). There is no evidence provided that Cal-DPO indeed avoids this issue.

I still have strong concerns related to both the motivations and soundness of this work, but due to the new experiments provided, I will increase my score 3->4. (Concerning the motivations, some motivations make sense to me, such as adding information about the *magnitude* of the rewards, and most of the other ones appear to be unfounded / not supported by evidence.)
We sincerely hope that our responses can address your comments. Moreover, as noticed by the reviewer, our work presents some interesting findings, a simple yet effective framework, and some theoretical contributions. The reviewer's suggestions can be easily and effectively addressed, and we genuinely hope that the reviewer can consider increasing the score. Thank you very much for your time!

[1] Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive. Arxiv 2024

[2] 3D-Properties: Identifying Challenges in DPO and Charting a Path Forward. Arxiv 2024

[3] Iterative Reasoning Preference Optimization. Arxiv 2024

[4] Simpo: Simple Preference Optimization with a Reference-free Reward. Arxiv 2024

[5] A General Theoretical Paradigm to Understand Learning from Human Preferences. AISTATS 2024

[6] Nash Learning from Human Feedback. ICML 2024
Dear NeurIPS Reviewer s9Bq,

We sincerely appreciate your time and the insightful comments you have provided during the review of our paper.

**The concerns highlighted in your comments relate primarily to misunderstandings on the setting of hyperparameters. We have made extensive efforts to address the misunderstandings you pointed out in our responses.**

As the discussion period is drawing to a close, we would like to confirm if there are any further clarifications you require. We would be grateful if you could consider revising your score upward.

Thank you once again for your time and attention; we truly appreciate it.
We appreciate the reviewer's perception of our novelty and thank the reviewer for the insightful comments. 

### The key and important concern from the reviewer is about the unfair hyperparameter search on Cal-DPO. **As this is an important misunderstanding, We would like to first clarify that our comparisons to other methods are indeed fair using *Comments*. The detailed responses to reviewer's other comments and misunderstandings will be presented with *Rebuttal* later before the rebuttal deadline.**

---

**Q1. Unfair comparisons to other methods: The authors show in Figure 4 that the best 
 value for Cal-DPO is 0.001, and they choose to use this same value for their comparisons to all other methods (described in Appendix B.1, where they state that "all the hyperparameters are set to be the same").**

**A1.** **We apologize for your confusion and we believe there are some misunderstandings.**

### Our comparisons to other methods are indeed fair. We did not perform a hyperparameter search for our Cal-DPO to cherry-pick parameters for better performance, and then set the same hyperparameters for other methods.  **Instead, we directly set its hyperparameters based on its respective base method, such as DPO, following thorough searches for these base methods. Therefore, comparisons with other methods are fair since the base methods are optimally configured.**

Specifically, the value $\beta = 0.001$ is the optimal hyperparameter for DPO through an extensive search across $\beta \in [0.001, 0.01, 0.1, 0.5, 1.0]$ on Zephyr-7b-SFT. We chose to use $\beta = 0.001$ for Cal-DPO without conducting further hyperparameter searches. **When we state that "all the hyperparameters are set to be the same," we mean that the $\beta$ of Cal-DPO is set to match the optimal $\beta$ of the base DPO directly (not inversely as you mentioned)**. Similarly, for Cal-IPO and Cal-SLiC, we set their hyperparameters to match the optimal values found for IPO ($\beta=0.5$) and SLiC ($\beta=0.01$) through extensive searches of IPO and SLiC across $\beta \in [0.001, 0.01, 0.1, 0.5, 1.0]$. 

### We sincerely apologize for any confusion and we will clearly state this in the revision.  Nevertheless, we believe this is indeed a misunderstanding and not fatal to the major contributions of our manuscript. The responses to reviewer's other comments and misunderstandings will be presented soon.