Title: Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment
Venue: NeurIPS 2024
Accepted: yes
Keywords: Large Language Model

Alignment
Abstract: We study the problem of aligning large language models (LLMs) with human preference data. Contrastive preference optimization has shown promising results in aligning LLMs with available preference data by optimizing the implicit reward associated with the policy. However, the contrastive objective focuses mainly on the relative values of implicit rewards associated with two responses while ignoring
their actual values, resulting in suboptimal alignment with human preferences. To address this limitation, we propose calibrated direct preference optimization (Cal-DPO), a simple yet effective algorithm. We show that substantial improvement in alignment with the given preferences can be achieved simply by calibrating the implicit reward to ensure that the learned implicit rewards are comparable in
scale to the ground-truth rewards. We demonstrate the theoretical advantages of Cal-DPO over existing approaches. The results of our experiments on a variety of standard benchmarks show that Cal-DPO remarkably improves off-the-shelf methods.

Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment

Abstract

We study the problem of aligning large language models (LLMs) with human preference data. Contrastive preference optimization has shown promising results in aligning LLMs with available preference data by optimizing the implicit reward associated with the policy. However, the contrastive objective focuses mainly on the relative values of implicit rewards associated with two responses while ignoring their actual values, resulting in suboptimal alignment with human preferences. To address this limitation, we propose calibrated direct preference optimization ( Cal-DPO ), a simple yet effective algorithm. We show that substantial improvement in alignment with the given preferences can be achieved simply by calibrating the implicit reward to ensure that the learned implicit rewards are comparable in scale to the ground-truth rewards. We demonstrate the theoretical advantages of Cal-DPO over existing approaches. The results of our experiments on a variety of standard benchmarks show that Cal-DPO remarkably improves off-the-shelf methods. Code is available at https://github.com/tengxiao1/Cal-DPO .

1 Introduction

Aligning the behavior of large language models (LLMs) with human preferences is crucial for ensuring that the responses of a pretrained LLM are aligned with human or societal values and preferences [1, 2, 3]. In recent years, reinforcement learning from human feedback (RLHF)[2, 4] has become a standard approach for fine-tuning language models based on human preferences. RLHF involves first fitting a reward signal from human preference data and then using reinforcement learning (RL) algorithms such as PPO[5] to optimize language models to generate responses with high reward.

While RLHF shows impressive capabilities on diverse tasks ranging from programming to creative writing, its training process is unstable and complex [6, 7]. This potentially worsens the sample complexity and compromises efficient convergence. To address these issues, offline contrastive preference learning methods, which include DPO [7], IPO [8], and SLiC [9], have been proposed to replace RLHF with supervised learning on the preference data. These methods eliminate the need for explicit reward modeling by directly using the likelihood of the policy to define an implicit reward fitted to the preference data, and achieve notable efficiency and competitive performance [10].

While various contrastive preference learning methods employ different pairwise ranking losses, they share a common underlying motivation: Maximize the expected relative difference between the implicit rewards associated with the chosen and rejected responses. Because the ranking loss is invariant to various score transformations (e.g., subtracting a constant), these methods tend to ignore the absolute values of the rewards. Hence, while these methods learn to preserve the relative ordering between the likelihoods of the chosen and the rejected responses, they may reduce the likelihood of the chosen response. Figure 1 illustrates this behavior and its implications. In the case of DPO, a representative method of the contrastive methods, the likelihood of the chosen response

counter-intuitively continues to decrease despite remaining higher than the likelihood of the rejected response. An undesirable consequence of this behavior is that the learned policy increases the likelihood of unknown out-of-distribution responses, resulting in poor performance. Maximizing the likelihood of the chosen response can be important in many practical applications, e.g., reasoning and mathematical problem solving [11, 12], limiting the applicability of contrastive preference learning.

The preceding discussion raises an important question with significant implications for how we align LLMs with human preferences: How can we design a new objective that effectively alleviates this problem while ensuring that the learned policy theoretically converges to an optimal policy?

Our answer to this question is Cal-DPO , a simple yet effective framework for preference learning which optimizes the contrastive preference objective to maximize the relative differences between implicit rewards of chosen and rejected responses, while simultaneously ensuring that learned implicit rewards are calibrated to match the actual values of the ground-truth rewards (see Section 4.1 for a formal definition). The key intuition behind Cal-DPO is quite simple: If the implicit reward estimates from preference data are well-calibrated relative to the ground-truth rewards (meaning both lie on the same scale), we can prevent the likelihood (reward) of chosen responses from continually decreasing. Hence, Cal-DPO is designed to learn an implicit reward parameterized by the policy calibrated against the ground-truth reward. This can be achieved through a simple modification to the existing methods. For instance, Cal-DPO can be implemented on top of DPO with just one line of code and without any additional hyperparameters. Although we refer to our method as Cal-DPO , it notably generalizes to other preference optimization methods such as IPO and SLiC (see Section 4.3). In addition, we theoretically demonstrate that Cal-DPO possesses several properties that are desirable for fine-tuning LLMs based on preferences, such as mode-seeking behavior, negative preference optimization, or "negative gradient" to push down the likelihood of undesirable responses [10].

The main contributions of this paper are: (i) We propose Cal-DPO , a simple, effective, and intuitive framework for preference learning that facilitates alignment of language models. Cal-DPO aims to learn implicit reward functions for learning policy that are calibrated with respect to groundtruth rewards. (ii) We theoretically analyze the learning behaviors of Cal-DPO and prove that Cal-DPO is guaranteed to yield an optimal policy for preference learning. (iii) We present results of extensive experiments on a range of benchmark tasks, including controlled text generation [13], summarization [14], dialogue generation [1], and several reasoning tasks [15] that demonstrate that Cal-DPO consistently outperforms previous alignment methods for preference fine-tuning.

2 Related Work

Reinforcement Learning from Human Feedback (RLHF) is highly effective in aligning Large Language Models (LLMs) with human preferences [2, 4]. In RLHF, a reward model is trained from human preference data to map responses to a scalar reward, which aligns a policy using RL algorithms like PPO [5]. Although RLHF excels in instruction-following [2], safety alignment [1], and summarization [3], it requires a more complex training pipeline than supervised learning.

Recent work proposes simplifying RLHF by directly optimizing language models with contrastive learning on preference data, resulting in contrastive preference learning methods [10], such as DPO [7], IPO [8], and SLiC [9], NLHF [16], and their variants that incorporate rejection sampling,

e.g., RSO [17]. While each of these methods works with different loss functions, the primary objective is to increase the likelihood gap between preferred and dispreferred responses [10]. Other works [18, 19, 20, 21, 22] apply contrastive preference learning iteratively and on-policy. However, as shown in Figure 1, the likelihood of the preferred response frequently decreases during contrastive training, adversely impacting performance on tasks such as coding and mathematical question answering [11, 12]. This paper addresses this limitation by calibrating contrastive objectives for preference learning and supports our approach with extensive theoretical guarantees in practice.

There is a substantial body of work that analyzes contrastive preference learning methods from different perspectives. For instance, [23, 24] study the performance of DPO under distribution shift and show that DPO is more susceptible to out-of-distribution responses than PPO. [25] theoretically analyze DPO with noisy preferences. [26, 27, 28] revisit the training objective of DPO from the perspective of noise contrastive estimation [29]. [30] theoretically analyze the gradient vector field of DPO and show that DPO decreases the probability of disfavored responses faster than it increases the probability of generating preferred responses. [31, 32, 33, 34] derive DPO within a token-level MDP formulation. Concurrent work [10] shows that DPO employs a negative gradient to push down the likelihood of undesirable, e.g., rejected responses, and implicitly exhibits a "mode-seeking" behavior. In this paper, our results are complementary. Specifically, we introduce Cal-DPO , which explicitly calibrates the rewards learned by DPO to match the scale of ground-truth rewards, while exhibiting "mode-seeking" behavior by minimizing the reverse KL divergence similar to RLHF.

Also worth mentioning is a body of work on calibration in supervised classification [35], learningto-rank [36], unsupervised learning [35], and reinforcement learning [37, 38]. Recently, calibration has also been introduced into language models [39, 40]. In these works, calibration refers to the alignment of the model's assessed confidence scores with the likelihood of its responses being correct. In contrast, our focus is on calibrating the learned implicit rewards in contrastive preference learning to match ground-truth rewards in aligning language models with human preferences.

3 Notations and Preliminaries

Problem Setup. We consider the preference learning scenario as follows: let the text sequences x = [ x 1 , x 2 , . . . ] denote an input prompt, and y w = [ y 1 , y 2 , . . . ] and y l = [ y 1 , y 2 , . . . ] denote two responses, typically sampled from the reference policy π ref ( y | x ) . The response pairs are then presented to human labelers (or an oracle) who express preferences for responses given the prompt, denoted as y w ≻ y l | x , where y w and y l denote preferred and dispreferred responses, respectively. The preference distribution is commonly expressed using a latent reward model r ( x , y ) as:

where g : R → [0 , 1] is a monotone non-decreasing function (with g ( z ) = 1 -g ( -z ) ) that converts reward differences into winning probabilities). When g is the sigmoid function σ ( x ) = 1 1+ e -x , we get the Bradley-Terry (BT) preference model [41]. Given dataset D , containing feedback ( x , y w , y l ) , the goal is to learn a LLM policy π θ ( y | x ) to align human preference by generating high rewards.

RLHF. Typically, given the reward function r ( x , y ) , which dictates the human preferences, RLHF optimizes policy π θ for x to maximize reward with the following RL objective:

where β > 0 is an appropriate KL penalty coefficient. Due to the discrete nature of language generation, we typically optimize the RLHF objective in Equation (2) using RL algorithms, such as PPO [2, 5]. Although RLHF with PPO has achieved remarkable success, the training process of PPO is unstable because of the high variance of the policy gradient estimation [6].

Reward Modeling. One standard approach to reward modeling is to fit a reward function r ϕ ( x , y ) using the BT preference model in Equation (1). Specifically, the reward function r ϕ ( x , y ) can be estimated by maximizing the log-likelihood of the preference feedback ( x , y w , y l ) :

Contrastive Preference Learning. To simplify RLHF, contrastive preference learning [42, 7, 9, 8] uses the log-likelihood of the learning policy to implicitly represent the reward function:

With this parameterization, DPO [7] aims to optimize π θ based on the BT model in Equation (3):

Technically, minimizing the DPO objective or any other pairwise contrastive objective, such as those of IPO [8] and SLiC [9], essentially amounts to maximizing the relative reward differences between chosen and rejected responses as shown in [10]. However, as noted earlier, these pairwise ranking objectives are not scale-calibrated and ignore the absolute values of the rewards. Thus, the likelihood of the chosen response can continue to decrease during training as long as the relative difference in the likelihoods between the chosen and rejected responses remains large (see Figure 1). This property has resulted in suboptimal performance, especially on reasoning and mathematical problem-solving [11, 12]. In this paper, we address this limitation by proposing a simple yet effective calibrated objective to calibrate the behavior of contrastive preference learning methods such as DPO.

4 Calibrated Direct Preference Optimization

We proceed to introduce Calibrated Direct Preference Optimization ( Cal-DPO ), a simple and effective modification of DPO . The key intuition behind Cal-DPO is to calibrate the implicit reward function against the ground-truth rewards. Hence, Cal-DPO is designed to learn an implicit reward parameterized by the policy calibrated against the ground-truth reward. This can be achieved through a simple modification of DPO [7], assuming, as in the case of DPO , that the preferences adhere to the BT preference model. Thus, Cal-DPO can be implemented on top of DPO with just one line of code and without any additional hyperparameters. In principle, our idea of calibration can be applied to any contrastive preference learning algorithm such as IPO [8] and SLiC [9, 17] (see Section 4.3).

4.1 The Calibrated Objective for Preference Optimization

To fine-tune a policy on preference feedback, we propose learning a reward model implicitly and using the log-likelihood of the policy to represent the estimated reward [7, 9, 8]. Specifically, we start by defining a preference score of y w relative to y l , where the implicit reward is represented by π θ .

Here ̂ r θ ( x , y ) = log π θ ( y | x ) π ref ( y | x ) is an implicit reward defined by the training and reference policies π θ and π ref , allowing us to express that, for any θ ∈ Θ , the preference probabilities can be denoted as:

where we assume that preferences adhere to the BT preference model as in DPO [7]. With preference probabilities expressed in terms of the learning policy, we can find the maximum likelihood estimate by minimizing the preference optimization loss based on the preference feedback:

We can note that this pairwise loss is equivalent to the DPO objective (without β ) in Equation (5). As this contrastive pairwise loss is also not scale-calibrated, ignoring the absolute values of the rewards. Thus, we cannot guarantee that the estimated reward of the chosen response will increase and the reward of the rejected response will decrease, which tends to degrade the performance on math and reasoning benchmarks [11, 12, 43, 44]. We propose to address this limitation by explicitly constraining the implicit reward function log( π θ ( y | x ) /π ref ( y | x )) to a scale that matches the ground-truth reward r ( x , y ) /β . Intuitively, if the learned implicit rewards are constrained to lie in the range of the ground-truth reward, we can prevent the rewards of chosen responses from continually decreasing. Formally, we define "calibration" with respect to ground-truth reward as follows:

Definition 1. (Calibration). An estimated implicit reward log π θ ( y | x ) π ref ( y | x ) for the LM policy π θ is called scale calibrated with respect to the ground truth reward if log π θ ( y | x ) π ref ( y | x ) = r ( x , y ) β , ∀ ( x , y ) ∼ D .

Thus, in addition to the BT preference loss in Equation (8), we propose constraining the learned implicit reward to match the ground-truth reward through solving squared loss regression problems:

The calibration loss requires access to oracle rewards; however, in some scenarios, we may only have access to pairwise preference feedback. In such cases, we define the reward for preference feedback as follows: r ( x , y w ) = 1 / 2 and r ( x , y l ) = -1 / 2 , indicating y w ≻ y l | x . Empirically, we find that this works quite well in practice. Combining this calibration loss with the BT preference loss in Equation (8), we get the following full loss of Cal-DPO on human preference data:

where our modifications to the standard BT preference model are straightforward and depicted in blue. Intuitively, Cal-DPO learns an implicit reward parameterized by the LM policy that is "calibrated" against the ground-truth reward. Specifically, Cal-DPO attempts to push the reward of the chosen response toward 1 / 2 β and the reward of the rejected response toward -1 / 2 β , ensuring that π θ ( y w | x ) > π ref ( y w | x ) and π θ ( y l | x ) < π ref ( y l | x ) . This alignment keeps the estimate of the implicit reward consistent with the ground-truth reward. Our implementation of Cal-DPO builds directly on the DPO codebase with just one additional line (see pseudocode in Appendix B.1). Our experiments show that calibration via a simple square loss can consistently improve the performance of off-the-shelf DPO , demonstrating the potential of calibration for preference fine-tuning.

4.2 Theoretical Analysis

Next, we proceed to present a theoretical analysis of Cal-DPO . We show that Cal-DPO with reward calibration enjoys important properties that are desirable for fine-tuning LLMs with preferences, e.g., mode-seeking behavior, negative preference optimization ("negative gradient" property) to push down the likelihood of undesirable responses [10]. We also show that Cal-DPO minimizes an upper bound on the standard KL-regularized RLHF in Equation (2). All proofs are provided in the Appendix 4.2.

We start by presenting our theoretical framework from a distribution matching perspective. We first interpret the RLHF objective as the optimization of a reverse KL-divergence between π θ and π ∗ [7]:

where π ∗ ( y | x ) ∝ π ref ( y | x ) exp( r ( x , y ) /β ) . The derivation is given in Appendix A.1. Since reverse KL can be difficult to optimize [7, 10], one can instead optimize the forward KL divergence:

which is the weighted maximum likelihood loss. Although this objective provides a straightforward approach for preference fine-tuning [45, 46, 47], it leads to poor performance compared to RLHF , as shown by [10]. The poor performance of the preceding objective in Equation (12) relative to RLHF is due to the fact that it assigns a positive weight to all samples y for a given x . Since the rewards are always positive, the likelihood loss always tries to increase the probability of a response even if it receives a much smaller reward compared to other responses. We note that the recent work of [10] also empirically demonstrates that the maximum likelihood criterion lacks the 'negative gradient' property, resulting in poor performance of MLE in Equation (12) compared to RLHF and DPO .

Because KL-regularized RLHF and the MLE objective in Equation (12) are both population losses which involve the 'oracle' reward r ( x , y ) , we also define a population loss for Cal-DPO to facilitate a direct comparison between Cal-DPO with KL-regularized RLHF and the MLE loss of Equation(12):

Our proposed loss in Equation (10) is an empirical estimate of this population loss on preference feedback by sampling two responses, i.e., y w ≻ y l | x and setting a small β (see Appendix A.2).

In what follows, we theoretically show that our Cal-DPO enjoys the "negative gradient" property.

Theorem 1. Minimizing the first term in our Cal-DPO in Equation (13) is equivalent to minimizing the forward KL divergence, or equivalently MLE in Equation (12) , while maintaining the following contrastive negative gradient with respect to π θ :

This theorem establishes the relationship between the objectives of MLE and our Cal-DPO . Notably, the theorem shows that the first term in Cal-DPO also minimizes forward KL divergence with negative gradients. Specifically, we examine this update rule to illustrate how our objective produces a negative gradient [10]. If we sample response y , and w ( x , y ) -ˆ w ( x , y ) is positive (which happens more often when the reward r ( x , y ) is high), the update rule will increase the log-probability of this response. This leads to an increase in the probability of generating a response with a high reward. Conversely, if w ( x , y ) -ˆ w ( x , y ) is negative (which occurs more often when r ( x , y ) is low), we decrease the probability of response y , while increasing the probability of other responses due to normalization. This implies that Cal-DPO also exhibits a form of the 'negative gradient' [10].

Theorem 1 shows that the first contrastive term in Cal-DPO minimizes forward KL divergence similar to MLE but with negative gradients. In practice, minimizing either KL divergence results in policies with distinct properties due to limited data coverage [48, 49, 50]. Specifically, forward KL D KL [ π ∗ ∥ π θ ] promotes mode-covering behavior, whereas reverse KL D KL [ π θ ∥ π ∗ ] encourages modeseeking behavior [10, 49, 51, 52]. In other words, forward KL encourages all responses in datasets to have equal probability, resulting in an overestimation of the long tail of the target distribution, whereas reverse KL sharpens the probability mass on certain high-reward regions. Thus, alignment commits to generating a certain subset of high-reward responses, which is more effectively realized by minimizing the reverse KL, as the RL objective in Equation (11) does. This also explains why MLE , which optimizes forward KL, performs worse than RLHF with reverse KL as shown in [10].

In what follows, we show that Cal-DPO with calibration loss also theoretically encourages modeseeking behavior by minimizing an upper bound of the RL objective in Equation (11):

Theorem 2. Minimizing the Cal-DPO objective in Equation (13) with respect to π θ will encourage mode-seeking behavior by minimizing an upper bound of the reverse KL divergence, as RLHF does.

This theorem shows that, unlike DPO and MLE , Cal-DPO with calibration asymptotically minimizes a reverse KL divergence with respect to the policy, making it mode-seeking like RLHF . The first term, i.e., preference loss in Cal-DPO , corresponds to optimizing the forward KL as shown in Theorem 1. Hence, our proposed calibration loss can be understood as minimizing the gap between forward KL and reverse KL. Theorem 2 also implies that DPO , RLHF , and Cal-DPO asymptotically converge to the same global optimal policy in the limit given a sufficiently large dataset and model capacity. Thus, our calibration objective in Equation (9) is similar in some respects to RLHF . Table 1 presents a comparison of the different methods in terms of strengths and weaknesses compared to Cal-DPO .

4.3 Generalizations and Extensions

We observe that our approach to calibrating the learned implicit reward from preference feedback is not limited to the DPO algorithm or the BT preference model. As we shall see below, the general idea behind Cal-DPO extends to other methods such as IPO and SLiC and other preference models. Instead of the sigmoid loss in Equation (7), SLiC [9, 17] minimizes a pairwise hinge loss:

IPO [8] is a contrastive algorithm similar to DPO and minimizes the following pairwise square loss:

A potential advantage of SLiC and IPO over DPO is that they do not require that the preference model be BT and can work with general preference probabilities. By combining our calibration loss in Equation (9), it is straightforward to define the calibrated counterparts of both L SLiC and L IPO . Thus, our calibration approach can be generalized to work with SLiC and IPO as well (see Section 5.5).

5 Experiments

5.1 Experimental Setup

Datasets. We evaluate Cal-DPO on four widely used datasets for preference fine-tuning: the UltraFeedback Binarized dataset [53, 54], Reddit TL;DR summarization dataset [14], Anthropic-HH dataset [1], and the IMDb sentiment dataset [13]. Details of the datasets are provided in Appendix C.1.

Tasks and Evaluation. Following previous work [7, 54], we evaluate methods fine-tuned on the UltraFeedback Binarized dataset across general reasoning benchmarks (MMLU-PRO [55], ARC [56], IFEval [57], BBH [58], GPQA [59]), and mathematical reasoning (GSM8K [60] and MATH [61]). For training on the UltraFeedback Binarized dataset, we utilize the same chat template used in [44] for all methods. We also use AlpacaEval 2.0[62], a benchmark for assessing LLM alignment with human preference. The Anthropic HH dataset is used for dialogue generation to produce helpful and harmless responses [7]. For summarization, we use the Reddit TL;DR dataset. For the dialogue generation and summarization tasks, we use GPT-4 for zero-shot pairwise evaluation, which is consistent with human judgments (see prompts in Appendix C.2.1). In the IMDb sentiment dataset, the goal is controlled text generation to produce positive sentiments from movie review prefixes [7]. We train a binary sentiment classifier and define the oracle reward as its log odds, and evaluate the policy on the trained reward model [7]. The task and evaluation details are given in Appendix C.2.

Models. For summarization and dialogue generation tasks, we use Pythia-2.8b [63] as our base model and the model after SFT as the reference model following [7]. For the IMDb controlled text generation task, both the policy and reward models are initialized from the GPT-2 Large model [64]. For tasks with the UltraFeedback Binarized dataset, we use the Zephyr-7b-sft model [54] as our base model to ensure alignment with previous work on LLM preference alignment [54].

Baselines We compare Cal-DPO with the following preference optimization methods: DPO [7], IPO [8], SLiC [9], CPO [65]. We also compare Cal-DPO with other variants of DPO: f-DPO [66], DPO-Positive (DPOP) [11] and DPO+NLL [67] which combine DPO loss over preference pairs and the negative log-likelihood (NLL) loss over chosen responses. We also compare with weighted MLE in Equation (12). Besides DPO, we also implemented our calibration objective on top of SLiC [9] and IPO [8] (see Section 5.5). The implementation details are provided in Appendix B.1.

5.2 Performance Comparison on Benchmarks

Reasoning Benchmarks. Table 2 compares the performance of Cal-DPO against other alignment methods on the UltraFeedback Binarized dataset. Our results show that Cal-DPO exhibits remarkable effectiveness in enhancing DPO's performance. The improvements are particularly notable on the IFEval and Math benchmarks, with relative gains exceeding 63.1% and 12.5% compared to the best baseline, respectively. These findings underscore the efficacy of Cal-DPO . We hypothesize that these improvements can be attributed to the calibration of implicit rewards performed by Cal-DPO as part of its training objective. Without proper calibration, the likelihood of selected samples decreases, resulting in suboptimal performance, especially in mathematical reasoning tasks, where the chosen

responses are very likely the ground-truth answers. Furthermore, Cal-DPO outperforms CPO, which combine DPO over preference pairs with the negative log-likelihood loss over the chosen response. We hypothesize that the superiority of Cal-DPO over CPO can be attributed to the conservative nature of the objectives optimized by the latter, which only affects the chosen response. In contrast, Cal-DPO calibrates implicit rewards for both chosen and rejected responses.

Instruction-following Benchmarks. To assess the ability of Cal-DPO on align with human instruction, we compare the performance of Cal-DPO and DPO on AlpacaEval 2.0 [62], an evaluator based on GPT-4 (version gpt-4-1106-preview) that produces the probability of preferring the evaluated model. Figure 2 shows the comparison in terms of both normal and length-controlled (LC) percentage of wins. We see that Cal-DPO demonstrates steady performance gains with the number of training iterations and outperforms SFT and DPO methods, which tend to produce longer responses.

5.3 Performance Comparison with Human Preferences

We also designed experiments to explore learning from real human preferences, focusing on summarization and dialogue generation tasks. Specifically, we used the Reddit TL;DR dataset for summarization and the Anthropic-HH dataset for dialogue generation. Table 3 summarizes the GPT-4 evaluation results. These results show that Cal-DPO demonstrates a notable improvement over DPO and its variants compared to both the SFT and the chosen responses. Remarkably, Cal-DPO aligns better with human preferences than baselines, achieving win rates of at least 60% against the chosen responses. This indicates that Cal-DPO shows strong promise in terms of aligning with human preferences.

Additionally, we provide examples generated by both DPO and Cal-DPO for both tasks in Appendix D.1. These examples show that GPT-4 consistently prefers Cal-DPO over baselines and the chosen responses in the dataset, demonstrating that Cal-DPO significantly improves DPO in terms of helpfulness and harmlessness of the generated responses.

5.4 Performance Comparison on Controlled Evaluation

We conducted experiments on the IMDB dataset to assess the generation of positive movie reviews. The task requires the model to provide positive and fluent completions of movie reviews based on given partial input texts. To perform a controlled evaluation, we trained a binary sentiment classifier on the IMDB dataset and defined the oracle reward as its log odds, following [7]. The reward score of the reward model then serves as an in-domain proxy for the unknown ground-truth reward used in evaluation. The results of IMDB sentiment generation are listed in Table 4. We used the reward score of the reward model and the perplexity of GPT-2 [64] to assess alignment performance. We observe that (1) PPO, DPO, and Cal-DPO can align the SFT model with the preference of the reward model, as

SFT

DPO

Cal-DPO

Method

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

evidenced by the increasing reward score, and (2) Cal-DPO performs better in terms of reward score and perplexity than both PPO and DPO, which confirms our theoretical results that the policy trained by Cal-DPO can effectively maximize rewards.

5.5 Further Analysis

Training Dynamics. We also investigated the reward patterns during the training process of Cal-DPO . Figure 3 presents the reward patterns for Cal-DPO and DPO on the TL;DR summarization dataset. We observe that the rewards of the rejected data keep decreasing, and the margins between the chosen and rejected responses keep increasing. However, in the case of DPO, the rewards of the chosen responses fall below zero, whereas they continue to increase with Cal-DPO , underscoring the utility of reward calibration in LLM alignment. These results are similar to those on the UltraFeedback dataset shown in Figure 1, verifying our motivation and the effectiveness of Cal-DPO .

Generalization to other objectives. As mentioned in Section 4.3, our approach to calibrating the learned implicit reward from preference feedback generalizes in a straightforward manner to other pairwise preference optimization methods including IPO and SLiC. To show this, we implemented the calibration objective of Cal-DPO for IPO and SLiC, yielding their calibrated counterparts Cal-IPO and Cal-SLiC, respectively. Figure 3 shows the comparison on the Anthropic-HH dataset. We observe that combining our calibration objective can consistently improve standard OCPL methods for LLM preference alignment, demonstrating the broader utility of preferance calibration.

Coefficient Parameter. We investigate the effect of the coefficient β and present an ablation study to analyze the performance of Cal-DPO on various tasks by varying β . Figure 4 in the Appendix shows the performance with different values of β . We observe that β plays an important role in Cal-DPO . A small β typically improves the model performance, whereas a too large β encourages the policy to remain close to the reference policy, leading to poor performance.

6 Conclusion and Limitations

We have presented Cal-DPO , a simple yet effective fine-tuning approach for aligning LLMs with human preference. Cal-DPO incorporates a simple preference calibration term that modifies the

behavior of the objective of contrastive preference learning methods such as DPO. The key idea behind Cal-DPO is to ensure that the learned implicit rewards lie within the same range as the groundtruth rewards, yielding substantial gains in performance relative to the state-of-the-art baselines on several widely used benchmark data sets. We also demonstrate theoretically that Cal-DPO exhibits the desirable 'negative gradients' and 'mode-seeking' behavior.

A limitation of Cal-DPO is that it is currently limited to offline methods and does not consider on-policy learning where the policy can interact with the reward model during learning. It would be interesting to explore how the reward calibration idea used in Cal-DPO performs in the on-policy learning scenario. We leave this as an interesting direction for future work.

Acknowledgments

The work of Honavar and Xiao was supported in part by grants from the National Science Foundation (2226025, 2225824), the National Center for Advancing Translational Sciences, and the National Institutes of Health (UL1 TR002014).

References

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 , 2022.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems , pages 27730-27744, 2022.

Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems , 33:3008-3021, 2020.

Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems , 30, 2017.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep policy gradients: A case study on ppo and trpo. In International Conference on Learning Representations , 2020.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems , 36, 2024.

Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics , pages 4447-4455. PMLR, 2024.

Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slichf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.10425 , 2023.

Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Stefano Ermon, Chelsea Finn, and Aviral Kumar. Preference fine-tuning of llms should leverage suboptimal, on-policy data. arXiv preprint arXiv:2404.14367 , 2024.

Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with dpo-positive. arXiv preprint arXiv:2402.13228 , 2024.

Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, et al. Advancing llm reasoning generalists with preference trees. arXiv preprint arXiv:2404.02078 , 2024.

Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies , pages 142-150, 2011.

Michael Völske, Martin Potthast, Shahbaz Syed, and Benno Stein. Tl; dr: Mining reddit to learn automatic summarization. In Proceedings of the Workshop on New Frontiers in Summarization , pages 59-63, 2017.

Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 12 2023.

Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Andrea Michi, et al. Nash learning from human feedback. arXiv preprint arXiv:2312.00886 , 2023.

Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. In The Twelfth International Conference on Learning Representations , 2023.

Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXiv:2401.10020 , 2024.

Wei Xiong, Hanze Dong, Chenlu Ye, Han Zhong, Nan Jiang, and Tong Zhang. Gibbs sampling from human feedback: A provable kl-constrained framework for rlhf. arXiv preprint arXiv:2312.11456 , 2023.

Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie. Direct nash optimization: Teaching language models to self-improve with general preferences. arXiv preprint arXiv:2404.03715 , 2024.

Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. Direct language model alignment from online ai feedback. arXiv preprint arXiv:2402.04792 , 2024.

Teng Xiao, Yige Yuan, Zhengyu Chen, Mingxiao Li, Shangsong Liang, Zhaochun Ren, and Vasant G Honavar. Simper: Simple preference fine-tuning without hyperparameters by perplexity optimization. arXiv , 2024.

Ziniu Li, Tian Xu, and Yang Yu. Policy optimization in rlhf: The impact of out-of-preference data. arXiv preprint arXiv:2312.10584 , 2023.

Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for llm alignment? a comprehensive study. arXiv preprint arXiv:2404.10719 , 2024.

Sayak Ray Chowdhury, Anush Kini, and Nagarajan Natarajan. Provably robust dpo: Aligning language models with noisy feedback. arXiv preprint arXiv:2403.00409 , 2024.

Huayu Chen, Guande He, Hang Su, and Jun Zhu. Noise contrastive alignment of language models with explicit rewards. arXiv preprint arXiv:2402.05369 , 2024.

Han Zhang, Lin Gui, Yu Lei, Yuanzhao Zhai, Yehong Zhang, Yulan He, Hui Wang, Yue Yu, Kam-Fai Wong, Bin Liang, et al. Copr: Continual human preference learning via optimal policy regularization. arXiv preprint arXiv:2402.14228 , 2024.

Haozhe Ji, Cheng Lu, Yilin Niu, Pei Ke, Hongning Wang, Jun Zhu, Jie Tang, and Minlie Huang. Towards efficient and exact optimization of language model alignment. arXiv preprint arXiv:2402.00856 , 2024.

Zhuang Ma and Michael Collins. Noise contrastive estimation and negative sampling for conditional models: Consistency and statistical efficiency. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 3698-3707, 2018.

Duanyu Feng, Bowen Qin, Chen Huang, Zheng Zhang, and Wenqiang Lei. Towards analyzing and understanding the limitations of dpo: A theoretical perspective. arXiv preprint arXiv:2404.04626 , 2024.

Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From r to q*: Your language model is secretly a q-function. arXiv preprint arXiv:2404.12358 , 2024.

Han Zhong, Guhao Feng, Wei Xiong, Li Zhao, Di He, Jiang Bian, and Liwei Wang. Dpo meets ppo: Reinforced token optimization for rlhf. arXiv preprint arXiv:2404.18922 , 2024.

Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang, Haifeng Zhang, and Jun Wang. Tokenlevel direct preference optimization. arXiv preprint arXiv:2404.11999 , 2024.

Teng Xiao and Donglin Wang. A general offline reinforcement learning framework for interactive recommendation. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 4512-4520, 2021.

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning , pages 1321-1330, 2017.

Le Yan, Zhen Qin, Xuanhui Wang, Michael Bendersky, and Marc Najork. Scale calibration of deep ranking models. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 4300-4309, 2022.

Ali Malik, Volodymyr Kuleshov, Jiaming Song, Danny Nemer, Harlan Seymour, and Stefano Ermon. Calibrated model-based deep reinforcement learning. In International Conference on Machine Learning , pages 4314-4323. PMLR, 2019.

Mitsuhiko Nakamoto, Simon Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning. Advances in Neural Information Processing Systems , 36, 2024.

Chiwei Zhu, Benfeng Xu, Quan Wang, Yongdong Zhang, and Zhendong Mao. On the calibration of large language models and alignment. arXiv preprint arXiv:2311.13240 , 2023.

Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221 , 2022.

Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika , pages 324-345, 1952.

Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Rémi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Ávila Pires, and Bilal Piot. Generalized preference optimization: A unified approach to offline alignment. arXiv preprint arXiv:2402.05749 , 2024.

Teng Xiao, Mingxiao Li, Yige Yuan, Huaisheng Zhu, Chao Cui, and Vasant G Honavar. How to leverage demonstration data in alignment for large language model? a self-imitation learning perspective. arXiv preprint arXiv:2410.10093 , 2024.

Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. arXiv preprint arXiv:2405.14734 , 2024.

Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308.08998 , 2023.

Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu. Openchat: Advancing open-source language models with mixed-quality data. In The Twelfth International Conference on Learning Representations , 2023.

Banghua Zhu, Hiteshi Sharma, Felipe Vieira Frujeri, Shi Dong, Chenguang Zhu, Michael I Jordan, and Jiantao Jiao. Fine-tuning language models with advantage-induced policy alignment. arXiv preprint arXiv:2306.02231 , 2023.

Kevin P Murphy. Machine learning: a probabilistic perspective . MIT press, 2012.

Ofir Nachum, Mohammad Norouzi, and Dale Schuurmans. Improving policy gradient by exploring under-appreciated rewards. arXiv preprint arXiv:1611.09321 , 2016.

Teng Xiao, Zhengyu Chen, Donglin Wang, and Suhang Wang. Learning how to propagate messages in graph neural networks. In ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pages 1894-1903, 2021.

Rishabh Agarwal, Chen Liang, Dale Schuurmans, and Mohammad Norouzi. Learning to generalize from sparse and underspecified rewards. In International conference on machine learning , pages 130-140. PMLR, 2019.

Teng Xiao, Zhengyu Chen, Zhimeng Guo, Zeyang Zhuang, and Suhang Wang. Decoupled self-supervised learning for graphs. Advances in Neural Information Processing Systems , pages 620-634, 2022.

Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.01377 , 2023.

Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of lm alignment. arXiv preprint arXiv:2310.16944 , 2023.

Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01574 , 2024.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457 , 2018.

Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911 , 2023.

Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging bigbench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261 , 2022.

David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022 , 2023.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 , 2021.

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874 , 2021.

Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models, 2023.

Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning , pages 2397-2430. PMLR, 2023.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , page 9, 2019.

Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of llm performance in machine translation. arXiv preprint arXiv:2401.08417 , 2024.

Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. Beyond reverse kl: Generalizing direct preference optimization with diverse divergence constraints. In The Twelfth International Conference on Learning Representations , 2023.

Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. arXiv preprint arXiv:2404.19733 , 2024.

Mark Reid, Robert Williamson, et al. Information, divergence and risk for binary experiments. 2011.

Mohammad Norouzi, Samy Bengio, Navdeep Jaitly, Mike Schuster, Yonghui Wu, Dale Schuurmans, et al. Reward augmented maximum likelihood for neural structured prediction. Advances In Neural Information Processing Systems , 29, 2016.

Arindam Banerjee, Srujana Merugu, Inderjit S Dhillon, Joydeep Ghosh, and John Lafferty. Clustering with bregman divergences. Journal of machine learning research , 6(10), 2005.

Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01335 , 2024.

Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. Understanding the effects of rlhf on llm generalisation and diversity. arXiv preprint arXiv:2310.06452 , 2023.

A Derivations and Proofs

A.1 Derivations of Equation (11)

We provide a detailed derivation of Equation (11), demonstrating that the RLHF objective in Equation (2) optimizes a reverse KL-divergence.

Note that the last line is exactly the RLHF objective with a negative sign. Therefore, maximizing the RLHF objective with respect to π θ is equivalent to minimizing the reverse KL-divergence. Because optimizing the reverse KL-divergence is mode-seeking, RLHF exhibits mode-seeking behavior.

A.2 Derivations of Equation (13)

In this section, we show that our proposed loss in Equation (10) is an empirical approximate estimation of the population loss in Equation (13) based on sampled preference feedback, i.e., y w and y l :

Using preference feedback y w and y l sampled from π ref ( y | x ) to approximate the above expectation gives us the following empirical estimation:

Given pairwise preference feedback y w ≻ y l | x with reward r ( x , y w ) = 1 / 2 , r ( x , y l ) = -1 / 2 , setting the coefficient β → 0 ( β is typically small and set as 0 . 001 in our experiments) gives us the following approximation (softmax weight becomes hard argmax weight):

This is equivalent to the loss proposed in Equation (10).

A.3 Proofs of Theorem 1

Theorem 1. Minimizing the first term in our Cal-DPO in Equation (13) is equivalent to minimizing the forward KL divergence, or equivalently MLE in Equation (12) , while maintaining the following contrastive negative gradient with respect to π θ :

Proof. Recall that the first term in the population objective of Cal-DPO in Equation (13) is:

Note that the second term in the last line does not depend on π θ . Therefore, minimizing the first term of Cal-DPO with respect to π θ is equivalent to optimizing the forward KL-divergence, as L MLE does.

We know from the previous result that L 1 and L MLE have the same optimal. However their gradients differs. We first take the derivatives of L 1 with respect to π θ :

Similarly, we can take the derivatives of L MLE ( θ ) in Equation 12 with respect to π θ :

Comparing Equations (37) and (39), we can observe that our Cal-DPO employs a negative gradient to push down the likelihood of bad responses under the learned policy compared to MLE .

A.4 Proofs of Theorem 2

Theorem 2. Minimizing the Cal-DPO objective in Equation (13) with respect to π θ will encourage mode-seeking behavior by minimizing an upper bound of the reverse KL divergence, as RLHF does.

Proof. Recall that Cal-DPO contains two terms: the contrastive term and the calibration term:

We begin by exploring the distinction between forward and reverse divergences. Before that, we introduce a few definitions and some background on Bregman divergence. For any two points r ∈ F and s ∈ F , the Bregman divergence D F , specified by the convex differentiable potential function F : F → R , is defined as follows [68, 69]:

where f = ∇ F , F ( p ) is the convex conjugate of F , p = f ( r ) , and q = f ( s ) [70]. By defining F ( s ) = log ∑ y exp( s ( y )) as the Log-Sum-Exp operator, f ( s ) = s ( y ) ∑ y exp( s ( y )) as the Softmax operator, and F ( p ) = -H ( p ) , we get the KL divergence between two probability vectors q and p :

By using Taylor expansions of F ( p ) , we further have the following:

where H F denotes the Hessian of F , and a = (1 -α/ 2) r + s α/ 2 , (0 ≤ α ≤ 1 / 2) , b = (1 -λ/ 2) s + r λ/ 2 , (0 ≤ λ ≤ 1 / 2) . It is well known that Hessian has the following form:

Taking the above and Equation (44), we obtain the following:

Plugging p = π ∗ ( y | x ) = π ref ( y | x ) exp( r ( x , y ) /β ) Z ( x ) , q = π θ ( y | x ) , s = log π θ ( y | x ) , and r = log π ref ( y | x ) + r ( x , y ) /β into Equation (46) obtains the following:

Theorem 1 demonstrates that the first contrastive term in Cal-DPO is equivalent to optimizing the forward KL-divergence. Consequently, we can directly obtain:

which completes the proof.

B Algorithm

B.1 Algorithm and Implementation Details

The pseudocode for Cal-DPO is provided in Algorithm 1. For the general hyperparameters, we closely followed the configurations used in [71]. The β of Cal-DPO is searched from [1e-3, 2e-3, 3e-3, 1e-2, 1e-1], the batch size for all methods is 128, and we use the RMSprop optimizer with a learning rate of 5e-6. We linearly warm up the learning rate from 0 to 5e-6 in 150 steps. The sampling temperature is set to 1 for all experiments. The experiments on are run on 4 Nvidia A100 GPUs with BF16 precision. We thoroughly tune the hyperparameters for each baseline following [44].

C Experimental Details

C.1 The Details of Datasets

In this section, we provide detailed descriptions of datasets:

UltraFeedback Binarized [53, 54]: This dataset 1 consists of 64k prompts, where each prompt is accompanied by four model completions from a wide variety of open and proprietary models. GPT-4

Algorithm 1: A Pytorch-style Pseudocode of Cal-DPO

is then used to assign a score to each completion based on criteria like helpfulness and honesty. It constructs binary preferences from UltraFeedback by selecting the highest mean score as the 'chosen' response and one of the remaining three at random as the 'rejected' response.

Anthropic-HH [1]: This Anthropic Helpful and Harmless dialogue dataset 2 contains 170k dialogues between a human and an automated assistant. This dataset was utilized for assessing single-turn dialogue performance. Each of the 170k dialogues comprises a human query and paired model responses rated for helpfulness and harmlessness. Following DPO [7], the preferred responses from this dataset were utilized for the supervised Fine-Tuning (SFT) phase, aligning the initial model behavior with desirable conversational outcomes.

Reddit TL;DR summarization [14]: This dataset 3 is a compilation of forum posts from the popular social media platform Reddit, specifically curated for summarization tasks with preference labels. Following [3, 72], we use the same filtered version of this dataset to train the SFT policy and use their preference labels for the following alignment stage.

IMDB Sentiment [13]: This dataset 4 contains movie reviews from the IMDb with positive and negative sentiment, which contains 25k training samples and each 5k samples for validation and test.

C.2 The Details of Tasks and Evaluation

We evaluate methods fine-tuned on the UltraFeedback Binarized dataset across reasoning benchmarks (MMLU-PRO [55], ARC [56], IFEval [57], BBH [58] GPQA [59]), and mathematical reasoning (GSM8K [60] and MATH [61]) provided by the Language Model Evaluation Harness library [15].

AI2 Reasoning Challenge (ARC) : This task is a set of grade-school science questions.

Massive Multitask Language Understanding Professional (MMLU- PRO) : MMLU-PRO is an enhanced version of the MMLU dataset [61], addressing previous shortcomings by increasing choice options in multiple-choice questions and refining question quality through expert review, making it more challenging and less prone to data contamination.

Instruction-Following Evaluation (IFEval) : IFEval is a benchmark evaluating a model's ability to follow explicit instructions, emphasizing adherence to formatting over content generation.

Big Bench Hard (BBH) : BBH is a selection of 23 challenging tasks from the BigBench, focusing on areas like multistep arithmetic, algorithmic reasoning, language understanding, and world knowledge.

Graduate-Level Google-Proof Q&A (GPQA) : GPQA is a challenging benchmark composed of advanced questions developed by PhD-level experts across various fields like biology, physics, and chemistry.

GSM8K : This dataset consists of diverse grade school math word problems to measure a model's ability to solve multi-step mathematical reasoning problems.

MATH : MATH is a benchmark consisting of high-school level competition problems gathered from multiple sources, consistently formatted with LaTeX for equations and Asymptote for diagrams.

AlpacaEval 2.0 : We also use AlpacaEval 2.0 [62], a benchmark to assess model fine-tuned on UltraFeedback Binarized dataset with human preference.

We consider the following tasks to evaluate the model fine-tuned on real human preference data.

Single-Turn Dialogue : We employ the Anthropic HH dataset, which includes 170,000 interactions between humans and virtual assistants. Our model is optimized to deliver useful and accurate answers across a range of queries while deliberately avoiding any harmful content.

Summarization : We leverage the Reddit TL;DR dataset, where the prompt represents a Reddit forum post and the preference pairs are collected by previous works. The task objective is to generate a summary that captures the main points of the post.

For single-turn dialogue and summarization tasks, we query GPT-4 for zero-shot pair-wise evaluation, which has been shown to be consistent with human judgments.

Controlled Sentiment Generation : We use the IMDb dataset [13], which contains 25,000 reviews, where the prompt is a prefix of a movie review and the preference pairs are generated following previous works. The task objective is to generate a positive sentiment y based on the movie review prefix. To perform a controlled evaluation, we train a binary sentiment classifier on the IMDb dataset and define the oracle reward as its log odds, following [7]. The reward score of the reward model then serves as an in-domain proxy for the unknown ground-truth reward used for evaluation. The results of IMDb sentiment generation are listed in Table 4. We used the reward score of the reward model and the perplexity of GPT-2 [64] to demonstrate the performance of alignment.

C.2.1 GPT-4 Win Rate Judgments

For the generation, summarization, and dialogue tasks, we utilize GPT-4 for zero-shot pair-wise evaluation, which is shown to be consistent with human judgments. We use gpt-4-0314 for all our experiments. The specific prompts used for GPT-4 evaluation are detailed below.

D Additional Experimental Results

Figure 5 shows the performance of SLiC and IPO, and their calibrated counterparts Cal-IPO and Cal-SLiC by applying our proposed calibration objective on the Anthropic-HH dataset. We observe that the calibrated counterparts significantly improve the performance of the off-the-shelf methods, demonstrating the effectiveness of our calibration objective. Figure 6 provides additional results of the training dynamics of DPO and Cal-DPO on the Anthropic-HH and IMDb datasets. These results show the robustness of Cal-DPO to different methods and datasets for preference fine-tuning

For the following dialogue history to a chatbot, which response is more helpful and harmless? Dialogue history: {dialogue history} Response 1: {Response 1} Response 2: {Response 2}

FIRST provide a one-sentence comparison of the two responses and explain which you feel is more helpful and harmless. SECOND, on a new line, state only "1" or "2" to indicate which response is more helpful and harmless. Your response should use the format:

Comparison: More helpful: <"1" or "2">

<one-sentence comparison and explanation>

Table 5: Prompt for GPT-4 evaluation for the dialogue generation task on the Anthropic-HH dataset. {dialogue history}, {Response 1} and {Response 2} are placeholders.

Which of the following summaries does a better job of summarizing the most important points in the given forum post, without including unimportant or irrelevant details? A good summary is both precise and concise.

1} 2} FIRST provide a one-sentence comparison of the two summaries, explaining which you prefer and why. SECOND, on a new line, state only "1" or "2" to indicate your choice. Your response should use the format: Comparison: <one-sentence comparison and explanation> Preferred: <"1" or "2">

Table 6: Prompt for GPT-4 evaluation for the summarization task on the Reddit TL;DR summarization dataset. {post}, {Summary 1} and {Summary 2} are placeholders.

Anthropic-HH

IPO

Cal-IPO

SLiC

Cal-SLiC

Methods

30

45

60

75

90

57.3

62.2

56.2

61.7

68.3

72.1

67.6

70.8

vs Chosen

vs SFT

D.1 Case Study

Warning: The following case studies contain example data that may be offensive or harmful.

In this section, we present examples of comparisons between Cal-DPO and the baseline DPO. See Tables 7 for dialogue examples, and Tables 12 for summarization examples.

Table 8: GPT-4 chooses Cal-DPO over DPO. Responses to a prompt from the Anthropic-HH Harmless test set.

Table 10: GPT-4 chooses Cal-DPO over DPO. Responses to a prompt from the Anthropic-HH Helpful test set.

Table 11: GPT-4 chooses Cal-DPO over DPO. Responses to a prompt from the Anthropic-HH Helpful test set.

Table 12: GPT-4 chooses Cal-DPO over DPO. Summaries to a post from the TL;DR test set.

Table 13: GPT-4 chooses Cal-DPO over DPO. Summaries to a post from the TL;DR test set.

E Societal Impacts

We proposed Calibrated Direct Preference Optimization ( Cal-DPO ) to align LLMs with human preferences, leading to more reliable and human-aligned AI systems. By enhancing training stability and efficiency, Cal-DPO addresses limitations in contrastive preference learning methods. Its applicability across various tasks, from programming to problem-solving, underscores its versatility. Given the successful deployment of large language models (LLMs) in various human-related real-world applications, it is crucial to ensure that the responses of a pretrained LLM to prompts are aligned with human or societal values and preferences, which can potentially yield direct social impacts.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our main claim matches our theoretical and experimental results in Section 4.2 and Section 5.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Please see Section 6 for the discussion of limitations.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Please refer to Section 4.2 and Appendix A for our assumptions and a complete (and correct) proof.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide information needed to reproduce the main experimental result and the link which contains the code and dataset to reproduce our results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code of Cal-DPO is available

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide all the training and test details in Appendix B.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We provide the experiment results that support the main claims of the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide the computer resources in Appendix B.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We make sure to preserve anonymity and conform NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Please see Appendix E for broader impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: The paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer:[NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper proposed Cal-DPO, a variation of DPO to address the issue of decreasing rewards of chosen answers. In addition to DPO loss, Cal-DPO add a pair of calibration terms, which aim to match the rewards induced by language model with some absolute ground truth reward value. Theoretical analysis shows that, during the training process of Cal-DPO, the likelihood of chosen responses will likely to increase and the likelihood of rejected responses will likely to decrease. The authors also prove that DPO loss can be upper bounded by Cal-DPO loss. Experiment on various benchmarks shows that Cal-DPO outperforms DPO and other baselines.

STRENGTHS:
The strengths of the paper are listed below:

1. The decrease in chosen answers' reward is a notable issue in DPO. In this paper, the authors focus on this issue and proposed Cal-DPO. This observation and corresponding mitigation is interesting and deserves attention.
2. The paper provides theoretical analysis of Cal-DPO. The analysis looks right to me.
3. The paper is well developed. Motivations, method and analysis are clearly presented.

WEAKNESSES:
This paper does not have specific weakness.

QUESTIONS:
1. Have the authors conducted experiment with models other than zephyr-7b-sft-full since zephyr-7b-sft-full demonstrate a relatively inferior performance on OpenLLM Leaderboard (which includes all reasoning benchmarks considered in the paper). Also, have the author conducted experiment on the other two benchmarks in OpenLLM Leaderboard (MMLU, TruthfulQA)?

LIMITATIONS:
The limitation is adequately clarified by the authors
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a simple yet effective change to the DPO objective that acts as a regularizer on the implicit reward, $\\beta \\log \\frac{\\pi(y \\mid x)}{\\pi_\\mathrm{ref}(y \\mid x)} + \\beta \\log Z(x)$, that is maximized by the LM. Specifically, the implicit reward is encouraged to be appropriately scaled relative to the desired reward $r(x, y)$. The main change is to add an additional squared loss term to the loss function that encourages the implicit reward for positive examples to concentrate around $\\frac{1}{2 \\beta}$ and the implicit reward for negative examples to concentrate around $\\frac{-1}{2 \\beta}$. The paper also includes some theoretical analysis, and positive empirical results.\n\nSTRENGTHS:\nThe proposed change is nice in that it is simple to implement, and shows favorable empirical results. The motivation and presentation are a bit unclear to me (see next section), however, for the most part it seems like a practical, sound result that would be of interest to practitioners in RLHF.\n\nWEAKNESSES:\nMy main difficulty is in the presentation of the motivation behind the method. In particular, I don't think that \"calibration\" is the right term to be using here --- it's a bit of a misnomer as it doesn't seem to have much to do with the standard way calibration is referred to (e.g., one might say that the BT model is calibrated if it produces calibrated preference probabilities). See also questions below.\n\nI also think that the theoretical results can benefit from being stated a bit more precisely---in particular the comment following Theorem 2  (i.e., \"Theorem 2 also implies that DPO, RLHF, and Cal-DPO asymptotically converge to the same global optimal policy ....\") should be stated clearly and proven. \n\nThere are also a number of other grammatical / writing errors (line 197, line 355, etc) that can be cleaned up.\n\nFrom the empirical side of things, I think that it would be good to also compare to IPO, as the style of loss is quite similar (e.g., this loss also enforces the margin between the implicit reward of positive / negative examples to be a constant, though without constant differences). It would be good to show the effect.\n\nQUESTIONS:\nIf I understand the main motivation correctly, Eq. (9) is essentially saying that the log partition term for the optimal policy should be 0 in this \"calibrated\" version? Or, in other words, one is simply defining a new, translated reward via the equality:\n\n$$\\pi^*(y \\mid x) = \\frac{1}{Z(x)} \\pi_\\mathrm{ref}(y \\mid x) \\exp( \\frac{1}{\\beta} r(x, y)) \\Rightarrow \\pi^*(y \\mid x) = \\pi_\\mathrm{ref}(y \\mid x) \\exp( \\frac{1}{\\beta} r'(x, y)),$$\n\nwhere $r'(x, y) = r(x, y) - \\beta \\log Z(x)$. This can then be arranged in the usual way to get $\\log \\frac{\\pi^*(y \\mid x)}{\\pi_\\mathrm{ref}(y \\mid x)} = r'(x, y) / \\beta$. Setting $r'(x, y) = \\pm \\frac{1}{2}$ is then equivalent to adding $\\beta \\log Z(x)$ to the original reward (assuming it was $\\pm \\frac{1}{2}$), which still preserves $\\pi^*(y \\mid x)$, as that term does not depend on $y$ . However, enforcing this equality (instead of just the difference equality), then removes the underdetermined aspect of the BT model, which I can imagine is helpful (in addition to the theoretical properties w.r.t. gradient dynamics analyzed).\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose Cal-DPO, a preference-tuning algorithm that modifies the DPO loss by adding two MSE terms that aim to \"calibrate\" the log-likelihood ratios of y_w and y_l to their respective reward values. They claim that this is advantageous because the DPO loss only maximizes the reward ratio, and does not constrain the respective reward values. They also claim that Cal-DPO exhibits a \"negative gradient\" (it pushes down the likelihood of undesirable outputs) and mode-seeking behavior. They evaluate their methods by training the Zephyr 7B model on Cal-DPO and DPO, and comparing the results on a few different benchmarks. Cal-DPO generally seems to outperform DPO and other related variants.\n\nSTRENGTHS:\n- Cal-DPO has some solid motivations -- it is true that DPO does not constrain the reward margin or the absolute scale of the rewards, which often leads to issues with over-optimization. IPO was similarly motivated in that it explicitly constrains the reward margin.\n- The concept of the \"negative gradient\" is a well-established motivation for designing training objectives that decrease the likelihood of y_l.\n- Cal-DPO is easy to implement and can be straightforwardly combined with other training objectives.\n\nWEAKNESSES:\n- **Unfair comparisons to other methods**: The authors show in Figure 4 that the best $\\beta$ value for Cal-DPO is 0.001, and they choose to use this same value for their comparisons to all other methods (described in Appendix B.1, where they state that \"all the hyperparameters are set to be the same\"). However, this implies that hyperparameter tuning was conducted for Cal-DPO, but not for any of the other methods. In fact, past work indicates that the optimal $\\beta$ value can vary widely depending on the combination of model and algorithm, in some cases reaching as high as 0.6 (see https://huggingface.co/blog/pref-tuning). The difference in accuracy/performance can be vast, depending on the hyperparameter configuration. In general, the fairer way to compare methods is to conduct a hyperparameter search separately for every method, rather than to fix a set of hyperparameters. This is especially important in this case, since the $\\beta$ was already selected to be the optimal one for Cal-DPO. Without this procedure, it is unclear whether Cal-DPO truly outperforms the other methods, or if the other methods were sub-optimally trained.\n- The authors train the model on a set of training datasets covering safety behaviors, summarization, and sentiment detection, but the evaluation benchmarks in Table 2 primarily cover other types of capabilities, such as math and abstract reasoning. These are not preference learning tasks, as they have exact answers, rather than subjective ratings of human preference. Even if a couple other papers have done this evaluation, this does not make as much sense as evaluating on more related benchmarks, such as MT-Bench. The Cal-DPO accuracy numbers in this table are also quite close to the other methods' -- and given the hyperparameter tuning issues, it is unclear how much of this difference is noise or is meaningful.\n- The authors predicate much of their rationale on the claim that other techniques like DPO reduce the likelihood of the chosen response. However, since DPO also reduces the likelihood of the rejected response, it is unclear whether this effect is actually undesirable -- to answer this question, one would have to know where the probability mass gets moved to instead. Since we know that DPO does in fact improve win rate, it seems likely that there is probability mass moved to more preferred outputs. The authors also do not cite any evidence that this is necessarily undesirable.\n    - Additionally, not all y_w's in preference datasets are high-quality -- it is often the case that both y_w and y_l are low-quality, but y_w is slightly less low quality than y_l. In this case, it is not necessarily desirable for the model to increase the probability mass on y_w.\n   - Similarly, the authors state that \"an undesirable consequence of this behavior is that the learned policy increases the likelihood of unknown out-of-distribution responses,\" but this is also unfounded. If there is past work or experimental evidence that supports this claim, it would be helpful to cite it here.\n- The authors also claim that mode-seeking behavior is more desirable here -- however, one can also see cases where this is undesirable. Since mode-seeking encourages more probability mass to be placed on high-reward outputs, and DPO/Cal-DPO rewards are based off the current model's likelihood, this promotes a positive feedback cycle where the algorithm continuously places more mass on outputs that are already high-likelihood under the current model. This may cause fast reward over-optimization, which has been a frequently observed issue in RLHF-based algorithms.\n- The experiments are only conducted on the Zephyr 7B model, and it would be much more convincing if the results reproduced to a couple other sizes and types of LLMs as well.\n- The paper also does not seem to list the generation parameters used for any of the evaluation experiments.\n\nMinor nits:\n- The stated IPO objective in Eq. 18 is incorrect -- the constant term should be $\\frac{1}{2\\tau}$, not $\\frac{1}{2}$. Also, there is no $\\beta$ coefficient. Alternatively, one could rename $\\tau$ as $\\beta$ instead and write $L_{IPO}=(h_{\\theta}(x,y_w,y_l)-\\frac{1}{2\\beta})^2$.\n- What are the \"Value\"s on the y-axis in the left two plots of Figure 3?\n\nQUESTIONS:\n- Do the authors have results where the baseline methods have also been hyperparameter tuned?\n\nLIMITATIONS:\nThe stated limitations about Cal-DPO being limited to offline methods is reasonable. An additional limitation is that the experiments have only been conducted on the Zephyr 7B model, and evaluated for preference learning on only a couple of small datasets (e.g. TL;DR summarization and Anthropic HH).", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2024-07-12"}

comments: SUMMARY:
This paper proposes a simple yet effective method called calibrated direct preference optimization (Cal-DPO), which addresses the limitation of ignoring the actual values of implicit rewards. The authors demonstrate the theoretical advantages of Cal-DPO over existing approaches and show the effectiveness on a variety of standard benchmarks.

STRENGTHS:
1. The proposed method Cal-DPO can support the motivation and claim theoretically.
2. Experimental results on different benchmarks show the effectiveness of the proposed method.

WEAKNESSES:
1. It is still not intuitive for me why the scale of the reward's actual value plays an important role in the generation performance. The proposed method seems to control this scale via an additional regularization term (Equation 10) and improve the empirical generation performance. But the relationship between them lacks explanations.

2. The authors should further justify why they choose the squared loss to constrain the learned implicit reward theoretically or empirically. Although they analyze some theoretical properties of the loss function, I believe that the squared loss is not the only form to possess these properties.

3. Equation 10 is over-simplified when directly assigning $r(x,y_w)%$ and $r(x,y_l)%$ to 1/2 and -1/2, respectively. The authors should provide intuitions and show whether the method is robust by assigning other values.

QUESTIONS:
I have included my questions in the weaknesses part.

LIMITATIONS:
The authors have adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-08-04