Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints
Authors: Chaoqi Wang
Yibo Jiang
Chenghao Yang
Han Liu
Yuxin Chen
Venue: ICLR 2024 spotlight (2024)
Keywords: Large language models
Preference optimization
AI Alignment
Primary area: general machine learning (i.e., none of the above)
TL;DR: This paper extends the Direct Preference Optimization (DPO) framework to incorporate diverse divergence constraints, offering a more versatile approach to align AI with human values.

Abstract: The increasing capabilities of large language models (LLMs) raise opportunities for artificial general intelligence but concurrently amplify safety concerns, such as potential misuse of AI systems, necessitating effective AI alignment. Reinforcement Learning from Human Feedback (RLHF) has emerged as a promising pathway towards AI alignment but brings forth challenges due to its complexity and dependence on a separate reward model. Direct Preference Optimization (DPO) has been proposed as an alternative; and it remains equivalent to RLHF under the reverse KL regularization constraint. This paper presents $f$-DPO, a generalized approach to DPO by incorporating diverse divergence constraints. We show that under certain $f$-divergences, including Jensen-Shannon divergence, forward KL divergences and $\alpha$-divergences, the complex relationship between the reward and optimal policy can also be simplified by addressing the Karush–Kuhn–Tucker conditions. This eliminates the need for estimating the normalizing constant in the Bradley-Terry model and enables a tractable mapping between the reward function and the optimal policy. Our approach optimizes LLMs to align with human preferences in a more efficient and supervised manner under a broad set of divergence constraints. Empirically, adopting these divergences ensures a balance between alignment performance and generation diversity. Importantly, our $f$-DPO outperforms PPO-based methods in divergence efficiency, and divergence constraints directly influence expected calibration error (ECE).

Published as a conference paper at ICLR 2024
BEYOND REVERSE KL: GENERALIZING DIRECT PREFERENCE OPTIMIZATION WITH DIVERSE DIVER- GENCE CONSTRAINTS
Chaoqi Wang Yibo Jiang Chenghao Yang
Han Liu
Yuxin Chen
Department of Computer Science, University of Chicago
Correspondence to edu
ABSTRACT
The increasing capabilities of large language models (LLMs) raise opportunities for artificial general intelligence but concurrently amplify safety concerns, such as potential misuse of Al systems, necessitating effective Al aligmment. Reinforce- ment Learning from Human Feedback (RLHF) has emerged as promising path- way towards Al alignment but brings forth challenges due its complexity and dependence on a separate reward model. Direct Preference Optimization (DPO) has been proposed as an alternative, and remains equivalent to RLHF under the reverse KL regularization constraint. This paper presents f-DPO, general- ized approach to DPO by incorporating diverse divergence constraints. We show that under certain f-divergences, including KL divergences and a-divergences, the complex relationship between the reward and optimal policy can also be simplified by addressing the Karush Kuhn-Tucker conditions. This eliminates the need for estimating the normalizing constant in the Bradley-Terry model and enables tractable mapping between the reward fune- tion and the optimal policy. Our approach optimizes LLMs to align with human preferences in more efficient and supervised manner under broad of di- vergence constraints. Empirically, adopting these divergences ensures balance between alignment performance and generation diversity. Importantly, f-DPO outperforms PPO-based methods in divergence efficiency, and divergence con- straints directly influence expected calibration error (ECE).
divergence, forward
INTRODUCTION
The increasing capabilities of large language models (Bubeck al., 2023; OpenAI, 2023) hold promise for achieving artificial general intelligence. However, they also pose safety concerns within the scope of AI risk (Amodei al., 2016; Hendrycks et al., 2023). Some of the hazardous capa- bilities an AI system may possess include social manipulation (Hendrycks et al., 2023), Al-enabled cyberattacks (Shevlane al., 2023), and enhanced pathogens (Urbina et al., 2022). These could be misused by humans or exploited by the AI system itself. Consequently, Al alignment research becomes critically important ensuring AI systems are robustly aligned with human values.
Reinforcement Learning from Human Feedback (RLHF) has emerged as a concrete research agenda, proving effective aligning model behaviors with human preferences and instruction follow- ing (Christiano et al., 2017; Bai et al., 2022a; Touvron et al., 2023). Given the challenge of spec- ifying an objective that accurately represents human preferences in RLHF, researchers typically collect dataset that reflects human preference in terms of model-wise generation comparisons (Bai et al., 2022a; LAION-AI, 2023). Subsequently, a reward model is trained under the Bradlcy-Terry model (Bradley & Terry, 1952) to infer the human's objective from the collected dataset. The lan- guage model is then fine-tuned using RL algorithms such as Proximal Policy Optimization (Schul- man et al., 2017; Ouyang et al., 2022) Advantage Actor-Critic (Mnih et al., 2016; Glaese et al., 2022) to maximize the reward. This process is carried out while ensuring that the model does not deviate significantly from its original form, using the reverse KI. divergence penalty.

---

Published as a conference paper at ICLR 2024
While effective, the RLHF pipeline is significantly more complex than supervised learning. In particular, RLHF necessitates training a separate reward model. The quality of this model ulti- mately determines the performance of reinforcement fine-tuning, and the language model may ex- ploit errors present within the reward model (Gao et al., 2023). Additionally, RI. algorithms, such as PPO (Schulman al., 2017), are less stable and more memory-demanne ing (Touvron et al., 2023). These challenges pique interest in searching for alternatives to the RLHF pipeline. Such efforts include Reward rAnked Funing (RAFT) (Dong al., 2023), Rank Re- sponses to align Human Feedback (RRHF) (Yuan et al., 2023), and Direct Preference Optimization (DPO) (Rafailov et al., 2023). DPO, as an early initiative, leverages the mapping between the reward function and the optimal policy to bypass the need for reinforcement learning and explicit reward model learning. Still, it's equivalent to RLHF for the final solution under reverse KL regularization.
than supervised learn-
However, most existing studies focus on solutions under the constraint of reverse KI, diver- gence, and the exploration of incorporating other divergences remains signifieantly lacking. To illustrate the differences among various divergence constraints, we have visualized the mode- seeking and mass-covering behaviors of reverse KL and forward KL divergences in Figure The mode-seeking property of reverse KL divergence tends to reduce diversity in generation (Wiher et al., 2022; Khalifa et al., 2021; Perez al., 2022; Glaese et al., 2022), which, although beneficial for optimiz- ing alignment performance, may limit the model's potential (e.g.. user engagement). Additionally, Santurkar et al. (2023) observe that finetun- ing LLMs with RLHF under the reverse KL regularization will result in limited range of political views. As mitigation, the inclusion of var- ious divergences can lead to solutions with distinct characteristics (e.g., diversity), especially when the model is specified..
Reverse
Density
Target
Figure 1: The mode seeking and mass cover- ing behaviors of reverse KL and forward KL. KI. divergence, we initially find that a naive derivation results in an excessively complex relationship between the reward and optimal policy for other divergences, largely due to the normalizing con- stant. However, by meticulously addressing the Krrush-KunnTTuckee ically the complementary slackness, we demonstrate that for a class of well-known divergences, such as Jensen-Shannor divergences, forward KL divergences and a-divergences with E (0. 1), the normalizing constant can be eliminated in the Bradley-Terry Terry model. This results in an analyt- ical and elegant mapping between the reward function and the optimal policy for broad class of divergences. It further enables us to optimize the language model to align with human preferences under varying divergences constraints without needing estimate the normalization constant. Such flexibility might allow us to explore a richer spectrum of modeling possibilities and cater to diverse application requirements
In this work, we generalize the DPO framework (Rafailov et al., 2023) to incorporate various divergence constraints. Different from the reverse
(KKT) conditions, specif-
In conclusion, our key contribution is the generalization of the DPO framework to seamlessly inte- grate a variety of popular divergences (e.g., Jensen-Shannen Shannon divergence, forward KL divergence and a-divergence) for regularization. Empirical results indicate that by adapting different divergence regularizations, we can achieve a nuanced balance between alignment performance (e.g., reward) and the generation diversity. This introduces greater flexibility during fine-tuning processes utiliz- ing human preference datasets. Furthermore, comparative analyses reveal that the generalized DPO framework surpasses PPO-based methods in divergence efficiency. We also prove that the difference in the expected calibration error (ECE) is bounded by the divergence between them, emphasizing the practical benefits of improved divergence efficiency in model calibration.
2 RELATED WORKS
AI Alignment (Leike et al., 2018) is proposed as a research agenda aimed at aligning model behav- ior with human preferences and instruction following. Not only has AI alignment been demonstrated to be essential in ensuring safe AI behaviors, but it also improves performance on variety of down- stream tasks (Bai et al., 2022a;b; OpenAI, 2023; Touvron et al., 2023; Glaese et al., 2022). These tasks include metrics such as helpfulness (Askell et al., 2021), truthfulness (Lin et al., 2022), and non-offensiveness (Gehman et al., 2020), etc. In this context, numerous methodologies have been proposed, including red teaming (Perez et al., 2022; Korbak et al., 2023), reward modeling (Leike
2

---

Published as a conference paper at ICLR 2024
et al., 2018; Gao et al., 2023), supervised fine-tuning, rejection sampling, and reinforcement learn- ing from human/Al feedback (Ziegler et al., 2019; Ouyang et al., 2022; Bai et al., 2022b), among others. These methods largely depend human judgment comprehensive set of principles to provide the supervised signal. For more complex situations where humans may be in- capable of evaluating, the main approach involves designing mechanisms that utilize Al to assist in evaluation by recursively decomposing the problem. This body of work includes AI debate (Irving al., 2018), iterated amplification (Christiano et al., 2018), and recursive reward modeling (Leike et al., 2018). However, most of these methods typically involve multiple stages of training or com- plex interaction protocols. In contrast, we focus on proposing to implement and computationally efficient in the setup where a human is capable of evaluating.
single-stage algorithm that is simple
Reinforcement Learning from Human Feedback (RLHF) (Christiano et al., 2017; Bai et al., 2022a; Touvron et al., 2023; Ouyang et al., 2022) has served as a pivotal method for aligning lan- guage models, contributing significantly to the success of ChatGPT (OpenAI, 2023). Nonetheless, RLHF's complexity surpasses that of supervised learning, primarily due to the need for a distinct reward model, the quality of which decisively influences the efficacy of reinforcement fine-tuning. Any errors within this model may be exploited by the language model (Gao et al., 2023). Further- more, RL algorithms like PPO (Schulman et al., 2017) have been proven to be less stable and more memory-intensive than supervised learning (Touvron interest in alternatives to the RLHF pipeline, such as Reward Ranked FineTTuning (RAFT) (Dong al.. 2023), Rank Responses to align Human Feedback (RRHF) (Yuan et al., 2023), and Direct Preference Optimization (DPO) (Rafailov et al., 2023). Despite these efforts, all the methods men- tioned focus solely on solutions within the confines reverse KL divergence regularization, leaving the potential advantages of incorporating various other divergences largely unexplored. Go et al. (2023) recently attempted minimize the f f-divergence for aligning language models. However, this approach necessitates the user to define the target distribution, and requires the estimation of the target distribution's 's normalizing constant- constant-adding -adding more hyperparameters and algorithmic com- plexity. In contrast, propose supervised learning method that incorporates various divergence regularizations. This method does not require the estimation of normalizing constants, the specifica- tion of the target distribution, the of reinforcement learning (e.g., model rollouts). Moreover, does not involve any additional hyperparameters.
al., 2023). These challenges have spurred
3 PRELIMINARY AND BACKGROUNDS
3.1 PRELIMINARY
f-divergences. For any convex function f R+ R that satisfies f(1) and is strietly convex around 1, then the corresponding f-divergence for two distributions and g is defined as
Dy(p. = By(e) [s p((a) q(<))
The f-divergence covers a broad class of commonly used divergences, including forward KL diver- gence, reverse KL divergence, Jensen-Shannon (JS) divergence and total variation distance, etc, by choosing the specific function f. We provide a summarization in Table 1.
Brarley-Terry Model. The Bradley-Terry model (Bradley & Terry, 1952) has been widely em- ployed for pairwise comparisons. It works by assigning a realevalued "strength" parameter p; to each item K. which is then used compute the probability that item Y4 outperforms item yj in pairwise comparison by p(y ? yj) = pi/(p: + P;). Yet, in practice, this model's instantiation usually adopts specific form, denoted as
exp(r(u:))
plu 2 m;) exp(r():)) + exp(r(y;))
where r(y:) represents the "rating" transformed "strength" of item y4-
The Bradley-Tere Terry model can be linked to Gumbel noise in the context of pairwise comparisons. The outcome depends on the comparison of r(y) + e; and r(y,) + ej. where e; and ej i.i.d Gumbel-disttibutte noise. If r(y)) + & > r(y;) + Ey. yi defeats yj- Noting that the differenne between Gumbel variables follows logistic distribution, we align this with the Bradley-Terry
3

---

Published as a conference paper at ICLR 2024
model's formula: p(y: УU) 1/(1 exp((r(y)) -r(y)))), with p: exp(r(v:)) and Pj exp(r(y))). Thus, the Bradley-Terry model represents a stochastic rank-order model influenced by Gumbel noise, which may justify the use of it for categorical distribution (e.g.. language models).
RL from Human Feedbacks. RLHF takes place after the base model has been pretrained. comprises three steps: 1) supervised fine-tuning, 2) reward model training, and 3) RL fine-tuning. In the RL fine-tuning process, the following objective is maximized:
E yy- (T' (v]l²) 3DKL (Te
where 'D represents the dataset of prompts, ry (-(ar) stands for the reward function learned using the Bradley-Terry model the preference dataset, (a:) is the fixed reference model (typically selected to be the one post supervised fine-tuning), and /3 is the coefficient of the reverse KL diver- gence penalty. In practice, this objective is equivalent to executing reinforcement learning under the following reward function (Ziegler et al., 2019; Bai et al., 2022a; Ouyang et al., 2022):
To (-
r(|x)) (-((-)))) log
Teet
Directed Preference Optimization (DPO). The original DPO method (Rafailov et al., 2023) estab- lishes functional mapping between the reward model and the optimal policy under the reverse KL divergence constraint. This allows for the direct optimization of the policy by reparameterizing the reward function using the policy (i.e., the language model) in supervised manner,
(-(=))) r(-1z) = log [Te(-1
+ log ((x).
Here, Z(x) is the partition function the normalizing constant. By plugging the reward into the Bradley-Terry model, the resulting objective of DPO with reverse KL. divergence given by:
- -E Va. v. )-D log (Slog (ye (r))
(o + 8 log ((T) log
()( 3 log log Teet (vt
2(7)
(I)
where a is the sigmoid function, and the partition funetions are cancelled out.
Calibration Error. To measure calibration, it's common to use the Expected Calibration Error (ECH) (Guo et al., 2017). Specifically, for any policy no ro(-)ar), define P.o. (x) as
P. (I) (
where ÿ is the predicted label of Given policy mo. ÿ is sampled with probability To(()))) Then, the ECE can be computed by,
ECE
E E
[IP(Ý Y = pl]-
Intuitively, if the model's confidence in its predictions closely matches the probability of its pre- dictions being correct, then the model is well-calibrated. In the realm of LLMs, OpenAl (2023) demonstrate that RLHF degrades the model's calibration.
3.2 BACKGROUND: OPTIMIZING FOR REVERSE HURTS DIVERSITY
The finetuning Large Language Models (LLMs) using Reinforcement Learning from Human Feedback (RLHF) has raised concerns regarding sample diversity. Notably. this process is prone to mode collapse (see Figure for illustration). This will result in reduction in the diversity of model outputs, as evidenced by studies from Khalifa et al. (2021), Perez et al. (2022), Go et al. (2023), and Glaese et al. (2022). One plausible explanation for mode collapse is the shift from supervised to reinforcement learning with reverse KL divergence (Song et al., 2023). Additionally, Santurkar et al. (2023) found that LLMs finetuned with RLHF under reverse KL divergence regularization express limited range of political views. Given these observations, there's 's clear need to investigate alternative divergence regularization methods to maintain the diversity and integrity of I.M outputs, and understand the tradeoff.
4

---

Published as a conference paper at ICLR 2024
Table Summary of some commonly used f-divergences including their derivatives.
f-divergence
f(u)
f')
Domain (()
a-divergence (a (0)
(ut-a aiu at)(aa(a 1))
u-uaa//a
Reverse KL (a 0)
u
logu +
Forward KL (a 1)
log
-1/u (2
JS-divergence
log u - (u + 1) log((u + 1)/2)
Total Variation
M ?
Chi-squared
(M/ 1)2
2(u 1)
x
METHOD: DIRECT PREFERENCE OPTIMIZATION UNDER f F-DIVERGENCE
During the RL finetuning process, it is common to regularize the finetuned model to stay "close" to the original model (or reference model) measured by the KL divergence. Typically, reverse KL a default choice. However, the mode-seeking behavior will lead to low diversity in the generations. Therefore, balance the alignment performance (e.g., accuracy or reward) and diversity, we consider more broad class of divergence regularization, namely, the f-divergence, which covers many commonly used divergences, such forward KI., reverse KI., divergence and a-divergence, etc.
divergence (i.e., DKL (Te(Tell meet))
When given the reward function r(ylæz) the base model is fine-tuned using reinforcement learning to maximize the reward under certain constraints. From an optimization perspective, this step is equivalent to solving the following constrained optimization problem:
max E, [r((lx) (T, Tref) s.l. Σn (u)x)) 1 and Tt)ax) 2 0 Vy.
The constraints introduced ensure that the solution is valid distribution, though in prac- tice we don't I need to explicitly deal with them. To solve the constrained problem, we can apply the Lagrange multiplier, which gives us
[s
( < Tor(yl (y)z))
)
a) EE [r(v)=)]]
Σ <(olo) -1) + Σ [((y))(yla),
where À and (y) are the dual variables. For such problems, we can derive the closed-form solution for n², which optimally solves the above problem:
r(y/2)) 3
(ylz))
((x) Tee [=)())==(())-
where ((x) is the normalization constant, and (f')-1 is the inverse function of f". By solving the equation for r(y]zz), establish the following relationship between ((yla) and x ()(2))
x (v(=z)
r(ul=) Bf
7(x))
Teer(y)z))
When Dy is the reverse KL divergence, we have f'(u) = log u from Table 1. Thus, r(y|=) = 3 log(m (ylz)/ (v)(zz)) log (xx) + 1. Since the BT model is defined by p(y. > r(u ()). the log Z(x), which appears as an additive term in the reward, will be canceled out for the reverse KL divergence (see Equation 1), but this is not the case for other diver- gences. This situation is discouraging, estimating the Z(x) requires multiple samples, which may be computationally expensive and may exhibit high variance if not handled properly.
3t) (r((v_]]))
Fortunately, we will show that, by carefully analyzing the normalization constant 7(x), we can derive closed-form solution for many other (but not all) divergences as well, without the need to estimate the normalization constant (z). To achieve this, we can first rewrite 7 (y ()) in the following format using the dual variables À and a(y):
n (y|==) m=)())=)(()- (r(y(x)
A + a(y) )
5

---

Published as a conference paper at ICLR 2024
Algorithm Direct Preference Optimization with f-divergences (DPO-J)
Require: Preference dataset D, batch size b, constraint coefficient 3, divergence function f, and
learning rate 1)-
1: Initialize model Te, with supervised finetuning on D.
2: for 7 = N...NNN iterations do
3: Sample batch B {(rs; y v))) from D.
4: Compute the loss using the equation with the chosen function f.
5: Compute the gradient and update the model 0, t 0.1 V
6: end for
7: return Final model m
However, the term aty) depends on y, and cannot be canceled out, making it hard to compute. Next, will show that for class of f-divergences, we have aty) and thus can represent the reward using only the trainable policy, the reference policy, and a constant that is independent of y. This result summarized in the following theorem.
Theorem 1. If (ye x)) 0 for any valid and f' is invertible with 0 dom(s'), the reward class that consistent with the Bradley- Terry model can be reparameterized using the policy model (y) (r) and reference model Tee
x (T(((()) (Ta(()/2)) + const.
r(ul=) = 3f
(2)
Theorem holds mainly due to the complementary slackness in the KKT conditions, and the proof can be found in Appendix F. Moreover, the requirement 0 & dom(f) already covers many com- monly used divergences, including forward KL divergence, Jensen-Shannon divergence, reverse KL divergence, and a-divergence with 0 < a < 1. Lastly, the constant is independent of and appears as an additive term in the reward function, and thus will be cancelled out when plugged into the Bradley-Ters Terry model.
Now, for a pair of examples (z, ywo) and (I, yt), we can plug the reward from Equation into the Bradley-Terry model, which gives the following expression,
(U (x)
(
(M]
p( ł u1zz) = o 3f
(Ter(WW (
ff
Hence, for a preference dataset D, we train the model me (replacing nt in the above equation) by minimizing the following negative log- lielelkiihodd loss,
log (ar' To (Jhe (x) (T((bul
(rer(Uull))) (z) )
no (B (x) ( (z:)
C( D) F( ww)D
38"
(3)
The above helps us solve the RL finetuning problem via supervised learning approach under broad class of divergence constraints, which is more stable and efficient to optimize in contrast to the reinforcement learning counter-part. Our results generalize the DPO (Rafailov al., 2023) to more broad class of divergence regularization. The full algorithm is summarized in Algorithm 1.
5
EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Baselines and Datasets. For the experiments, we adopt three datasets, including IMDB-sentiment dataset (Maas et al., 2011), Anthropic HH dataset (Bai et al., 2022a) and MT-bench (Zheng et al., 2023) for evaluation. Our primary baseline approach PPO with different f-divergences. Upon experimentation (see Appendix A), it was observed that incorporating divergence penalty in the reward for variants of PPO. such as those with forward KL divergence and JS divergence, induces training instability. The reason being that the value ranges of these divergences are considerably larger than those of reverse KL divergence. This discrepancy causes substantial challenges when trying to learn accurate value function in PPO. To address the issue, further propose modified

---

Published as a conference paper at ICLR 2024
1.0
Prodictive Entropp
8.8
Roward
0.9
Roward
0.3
Reward
3.90
0.8
0.3
0.84
15
0.7
16
Reverse KL
JSD
Forward KL
Divergence
Prodietive Pmtrrapy
6.15
0.9
0.0
8.30
Reward
Reyaord
0.0
Reward
0.05
DPO i PSO (onwonth
0:7
DPO lo 0.5 PPO (mwaan))
DRO to -
8.00
0.5-
0.6
0.6-
PPO crewand)
PPO (lned)
PPO (nos))
PPO Ones)
0.5
12
0.5
0 B
20
3.00
0-0.3
0-05
#=
o-divergence (0.3)
d-divergeace (0.5)
a-divergence (0.7)
Divergence
Figure 2: Comparisons between DPO with various /-divergences and PPO in terms of the frontier of divergence vs reward. To be noted, 'reward' means we add the divergence penalty in the reward, and 'loss' means we add the divergence penalty in the loss.
variant of PPO as an additional baseline. Instead of integrating the divergence penalty into the reward function, treat it regularization term separately. The new objective (i.e., PPO (loss)) is optimized separately by PPO and SGD,
PPO (reward);
E S.wDDymaolla - (f((o((lz))/mmm((ll)
Optimized by
PPO (loss):
Ey (-yooollaa)) (y(xr]
BDy (m no (-1a))
Optimized by PPO
Optimized by SGD
For the sake of differentiation, we term the conventional PPO method as PPO (reward) - indicating the inclusion of divergence penalty the reward. In contrast, the variant where the divergence regularization is treated separately is denoted as PPO (loss).
5.2 EXPERIMENTS ON IMDB DATASET
Our initial experiments were performed on the IMDB-sentiment dataset (Maas et al., 2011) for comparing f-DPO against PPO. Following the setup in trlx (CarperAI, 2023), we utilized GPT-2- large (Radford al., 2019) as our base model for fine-tuning, and the SiEBERT model (Hartmann et al., 2023) fine-tuned model of RoBERTa-large (Liu et al., 2019) was employed for reward computation. For PPO, we explored the divergence coefficient in f0.01, 0.03, 0.1, 0.33 for both PPO variants, each using ground-truth rewards. Our PPO implementation is based on the trix library. Additionally, we adapted the official implementation of DPO with f-divergences from Rafailov et al. (2023), setting 3 at 0.1. Please note that PPO utilizes the ground-truth reward during training.
The results are depicted in Figure 2, and we also visualize the trade- off curve in Figure 3. Our observations indicate that DPO with f- divergences outclasses both PPO implementations in terms of di- vergence versus reward the frontier, thus establishing greater divergence-efffcienncy. Additionally, PPO with a divergence penalty in loss (i.e., PPO (loss)) surpasses PPO with divergence penalty in reward for both JSD, forward KL. and a-divergences. This dis- crepancy arises due to the JSD divergence penalty and forward KL penalty fluctuating more significantly than the reverse KL penalty, which consequently introduces instability in learning the value function in PPO. Further experimental results can be found in the Appendix A. Finally, we note that reverse KL achieves the lowest predictive entropy due to its mode- seeking property, while Forward KL exhibits the highest predictive entropy. JSD maintains balance between the two. a-divergence interpolates between the JSD and forward KL divergence. This ob- servation aligns with the property of these divergence regularization (2023), although our framework does not necessitates manual specification of the target distribution or estimation of the normalizing constant under various divergence regularizations.
a=07
X
Entropy
4.0
3.0
J
3.7
X
0.90
Reward
Figure The reward and en- tropy tradeoff of f-DPO for different divergences.
well as those in Go et al.
7

---

Published as a conference paper at ICLR 2024
Table Comparison of JSD, RKL, FKL and some a-divergences in terms of Alignment and Diver- sity Anthropic HH. The indicates that higher is better, and means lower is better.
Alignment Accuracy (%) f
Diversity
Divergences
Entropy 1
Self-Bleu
Distinct-1 f
Distinct-2
RKI. JSD
67.19
12.25
0.880
0,021
0.151
66.80
12.31
0.878
0.021
0.159
0.3
59.77
12.85
0,849
0,026
0.199
a = 0.5
61.72
12.90
0.841
0.028
0.206
0.7
57.42
12.98
0.839
0.027
0.202
FKL
54.30
13.01
0.834
0.029
0.210
5.3 EXPERIMENTS ON ANTHROPIC HH DATASET AND MT-BENCH
Our next set of experiments was conducted on the Anthropic HH dataset (Bai et al., 2022a). We adopted the Pythia 2.8B model from Biderman et al. (2023) as our base model. The training con- figuration follows from Rafailov et al. (2023)'. The goals of these experiments were to study: 1) how different divergence regularizations impact the trade-off between alignment and diversity the generated responses, and 2) how f-DPO compares to its PPO counterparts. For the first part of the experiments, we utilized automatic metrics for evaluation, while for the second part, we relied on the GPT-4 evaluation. For PPO, we adopt the PPO (loss).
The results, in terms of alignment accuracy and diversity, are presented in Table 2. To measure diversity, we generated 25 responses using nucleus sampling (Holtzman et al., 2020) with p 0.95 for each prompt in the test set of the Anthropic HH dataset using temperatures of 0.6, 1.0, 1.4, following Touvron al. (2023). The results for temperatures 0.6 and 1.4 can be found in Ap- pendix E.1. To compute metrics, we employed the predictive entropy, self-bleu (Zhu et al., 2018) and distinct-n (Li et al., 2016). Consistent with the findings section 5.2, we observed that re- verse KL divergence achieves the highest accuracy but the lowest diversity in generation. Adjust- ing the divergence regularization allows us to trade-off between alignment accuracy and diversity. By tailoring outputs to specific upplication needs, it offers a customizable balance between accuracy and diversity. This might not only enhance robustness against unfamil- iar inputs but also boost user engagement by preventing monotonous interactions.
GPT-4 evaluation on MT-bench
DPO-RKL
25.0%
71.2%
3.8%
PPO JKL
DPOJISD vai PPO-JSM
36.9%
60.6%
2.5%
To compare f-DPO with PPO in terms of generation quality, we conducted pairwise comparison using MT-
DPO-a-0.5
6.3%
vs PPO-a-0.5
Bench (Zheng et al., 2023). MT-Bench is GPT-4-based evaluation benchmark for LLMs that achieves over 80% agreement with human preference judgments on LLM generation quality. MT-Bench includes a series of open- ended questions that assess LLM capabilities in multi- turn conversations and Instruction-folloiingg which are often considered important factors for human preference. In accordance with the official MT-Bench implementa- tion (Zheng et al., 2023),² maximum number of newly generated tokens to 1024. For additional details about MT-bench, we refer readers the original paper. The GPT-4 evaluation results are provided in Figure 4. From these results, it is evident that extending DPO with f-divergence achieves performance that comparable to, and in some cases significantly better than, that of PPO. We've also provided the comparisons between DPO under different fabereett-dvvrree f-divergence in Appendix E.2.
DPO-FXI va PPO-FKL
21.2%
25
50
75
100
125
150
Win
Figure 4: MT-Bench comparison be-
tween f-DPO and PPO under different divergences. The win, tie and lose rates are evaluated based on GPT-4.
sampled responses with temperature setting of 0.7 and limited the
5.4 EXPERIMENTS ON CALIBRATION
In our last set of experiments, we sought to explore the advantages of divergence efficiency. Our observations revealed that f-DPO achieves a smaller divergence than PPO while attaining compara- ble performance. Previous studies, as referenced by OpenAl (2023), suggest that RLHF adversely
Phtpps: /tatps://giii http:///gghhhhb.ooo/e.ece model_answer.p
8

---

Published as a conference paper at ICLR 2024
Reverse
JSD
Forward
0.38
0.36
0.36
0.36
0.336
P-0.1
0.334
0=0.9
0.36
0.34
20
80
100
20
40
00
100
20
40
60
100
Training Progress (%)
Training Progress (%)
Training Progress (%)
Ør 0.3
a 0.5
or = 0.7
0.342
0.345
ECF
0.345
0.340
0.540
0.338
0.340
0.336
0.335
0.335
20
40
60
80
100
20
400
60
80
100
20
40
60 80 100
Training Progress (%)
Training Progress (%)
Training Progress (%)
Figure 5: Evolution of Expected Calibration Error (ECE) across training steps for three different
divergence regularizations: Reverse KI., JSD, Forward KL and a-divergence (a = 0.3, 0.5 and
0.7). Each subplot represents the ECE values for varying regularization parameters 3 = 0.1, 0.3 and 0.9 with exponential smoothing.
affects the calibration performance of GPT-4. This prompts the question: is there a correlation be- tween the divergence of the base model and that of the finetuned version, and the calibration error? To elucidate this, we presented the following theorem,
Theorem 2. Suppose To, (-1zz) and no, (-1x>) be two policies. Let D be any I I-divergence such that f is strictly convex.
EC - (C)(O2) S 2E (D (-1x), me, (e
where y is a veal-v function such that lim (1) = O.
Remark 1. For JSD, have ECE((;)) ECE(O)) s Ex 4 2222D (Te, (1xx), no, (11z)
Remark 2. For KL. we have ECE(O)) 01) - ECE(()) < Ex 2 (To, (.1z), To, (- [z))
Theorem 2, detailed further in Appendix G, establishes relationship between Expected Calibration Error (ECE) difference and f-divergences. Specifically, the difference in ECE between two models can be bounded by the f-divergence. Thus, if the base model exhibits good calibration (i.e., small ECE), a smaller f-divergence suggests that the finetuned model is similarly well-calibrated.
To validate the theoretical findings, we conducted experiments to explore the impact of various f- divergences on calibration errors. We evaluated calibration error on Anthropic HH dataset (Bai al., 2022a) with the Pythia 2.8B model from Biderman et al. (2023) as our base model. The model was finetuned using f-DPO in the same way as Section 5.3. We treat the task binary prediction problem. For predictive probabilities, we use the exponentials of normalized scores where the scores are computed for chosen and rejected strings conditioned on input prompts. The results are shown in Figure 5. It is apparent that increased regularization parameters the calibration error can increase. On the other hand, as training progresses, the calibration error increases well. Similar trends calibration have been discovered in OpenAT (2023) as well.
restrict the extent to which
6 CONCLUSION
In this study, we introduced a generalized framework for DPO, elegantly incorporating a spectrum of divergence constraints. Our empirical results show that while the reverse KL divergence typi- cally offers superior alignment performance, it compromises generation diversity. By adjusting the divergence regularization, we can achieve a nuanced balance between alignment and generation di- versity. Notably, the f-DPO framework demonstrates greater divergence efficiency than traditional PPO methods. We further established that the difference in the expected calibration error of two models can be bounded by their divergence, underscoring the advantages of enhanced divergence efficiency. For practitioners, we recommend using JS divergence as the first choice, as it generally generates more diversified responses and is more favored by GPT-4 than reverse KL divergence. Looking forward, we aim to explore the integration of other divergences not well addressed by our present formulation, such the total variation distance.

---

Published as a conference paper at ICLR 2024
REFERENCES
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Con- crete problems in ai safety, ArXiv preprint, (bs/1600.06565, 2016. URL https: / org/abs/ 606.. 06565.
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, al. A general language assistant laboratory for alignment. ArXiv preprint, abs/2112.00861 2021. URL https: sstps://arxive /wtp/aaaivvvv//////ee/eaaeeleaaa wwwpiii.orrg/aaa.o 2112. 00861.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. ArXiv preprint, abs/2204.05862, 2022a. URL https:tpp//... / //tp//aaavvvvvva////a/eaaaaaaava org/ abs 05862.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. ArXiv preprint, abs/2212.08073,, 2022b. URL https:
//arxiv.org abs 2ttp:/22...000 08073.
Stella Biderman, Hailey Schoelkopf. Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hal- lahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling, 2023.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-34) 1952.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Ka- mar, Peter Lee, Yin Tat Lee, Yuanzhi Li. Scott Lundberg, et al. Sparks of artificial gen- eral intelligence: Early experiments with gpt-4. ArXiv preprint, Abs/2303.12713 2023. URL
httpp:://et /aaxivv.rivvvv////a org/ /33//
CarperAI. Trix: Transformer reinforcement learning https://githut: wwt.ua/.cccccaaaaaaaaaeeee /trpp/////
trlx, 2023.
Paul Christiano, Buck Shlegeris, and Dario Amodei. Supervising strong learners by amplifying
weak experts. ArXiv preprint, abs/1810.08575. 2018. URL
1810.08575
Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neu- ral Information Processing Systems 2017, December 4-9, 2017. Long Beach, CA, USA, pp. neur
4299 4307, 2017. URL https::tt//cc...ct http:/::///pcceeeeeeeeeee neur ps cc/ pccc/paper/20177 d5e2c0adad503ccmm w91.99912222222 oat.aaadde.......eeeeeeeoeeeeeee Abstract html.
Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. ArXiv preprint, abs/2304.06767, 2023. htRp:t//::// /ttpppp://rxiv...//// org/ abs /
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10835 10866. PMLR, 2023.
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. RealTox-
icityPrompts: Evaluating neural toxic degeneration in language models. In Findings of the
Association for Computational Linguistics: EMNLP 2020, pp. 3356-3369 Online, 2020. As- sociation for Computational Linguistics. doi: 10. 10.18653/v1/2020. 12..0fnni//22....nnnnnn..... fttps::/aaet antho h centthooogy... loy.. org.org/0000, fi fi indings-eme p.301.
URL
10

---

Published as a conference paper at ICLR 2024
Amelia Glaese Nat McAleese, Maja Trebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Mari- beth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. Improving alignment of dialogue agents via targeted human judgements. ArXiv preprint, abs/2209.14375, 2022. URL https: / rxiiv. g alos/ 2209.14375.
Dongyoung Go. Tomasz Korbak, Germán Kruszewski, Jos Rozen, Nahyeon Ryu, and Marc Dymet- man. Aligning language models with preferences through f-divergence minimization. ArXiv preprint, abs/2302.08215, 2023, URL https: /arpiii//axxvv.o//
org. http::://axx....r/////aa/ /tt..//....222.21.5..
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-1 August 2017, vol- ume 70 of Proceedings of Muchine Learning Research, pp. 1321-1330. PMLR, 2017. URL http://pret http://proceedingg... www.......... www..press/v700 https//////uuo/ooo/ooooo/
ggwlguulaaaa (aaol7a.... html.
Jochen Hartmann, Mark Heitmann, Christian Siebert, and Christina Schamp. More than a feeling: Accuracy and application of sentiment analysis. International Journal of Research in Marketing, 40(1):75-87, 2023. doi: sciencedi rect. com/cco//////ooo ici wwwpe//attceeeepe//eeaeeeaataaee /0001/11117111111 67811 /001//00000011112200000077 622000477.
http:://0..............m..000.00 art
URL https: /www.
Dan Hendrycks, Mantas Mazeika, and Thomas Woodside. An overview of catastrophic ai risks.
ArXiv preprint, abs/2306. 12001, 2023. URL httpt:///aaaaa www.vvorrg/bbba 2206. 12001.
Ari Holtzman, Jan Buys. Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL net/forum? wat.forummiddyyyyyyyyyyo
Geoffrey Irving. Paul Christiano, and Dario Amodei. Ai safety via debate. ArXiv preprint, abs/1805.00899 2018 URL https: //tp///avvvv////////////./...av. org//rrg//....oooooo..oo..eobeee aas///b///aa 2abb//105. 00899.
Muhammad Khalifa, Hady Elsahar, and Mare Dymetman. A distributional approach to controlled text generation. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL
net.
Forum?id=jwit 5 9AbL.
Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Bhalerao, Christopher L Buckley, Jason Phang, Samuel R Bowman, and Ethan Perez. Pretraining language models with human preferences. ArXiv preprint, abs/2302.08582, 2023. URL /tttpps//arxll.orr//bbbbb 08588..
LAION-AI. Open-assistant, 2023. Open-Ass: Assi stant. Accessed: 09/22/2023.
URL.
Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: a research direction. ArXiv preprint, abs/1811.07871,
2018. URL https: / arxiv. oog.oroggss..o.. aaaaaaall111 011111111078111
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A ddiversity-romootin objec- tive function for neural conversation models. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, pp. 110-119, San Diego California, 2016. Association for Computational Linguistics. doi: 10.18653/v 1/N16- 1014. URL https:://///.. httt::::/acaatthate /al/acclannttt http://haaaantoggooooo......oola org/N16-1c14,
Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic hu- man falsehoods. In Proceedings of the 60th Anmual Meeting of the Association for Compu-
tational Linguisties (Volume 1: Long Papers), pp. 3214-3252, Dublin, Ireland, 2022. Asso- ciation for Computational Linguistics. doi: 18.186/3/v aclo22. URL https: //aclanthology.
222.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. ArXiv preprint, 0bs//197..1192.. 2019. URL https: arxiv. orggabs////... 1 192.
11

---

Published as a conference paper at ICLR 2024
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Anmual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142-150, Portland, Oregon, USA, 2011. Association for Computational Linguistics. URL https:/ ac aallhooology... wwtp//1--/1111111111111
Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Maria-Florina Balean and Kilian Q. Weinberger (eds.), Proceedings of the 33nd International Conference on Machine Learning, ICMI. 2016, New York City, NY, USA, June 19- 24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pp. 1928-1937. JMLR.org, 2016. URL http: / /tttt::/pprcceeee proceedings...oooooo mlr, press /488 mnihal6.... 6.hml.
OpenAI. Gpt-4 technical report. ArXiv, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal. Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:
27730-27744, 2022.
Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. In Proceedings the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 3419-344444 Abu Dhabi, United Arab Emirates, 2022. Association for Computational Linguis- tics. URL https: / aalllllll aclanthology. org/ 2222. emn emnnl-mmali in 225.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. ArXiv preprint, abs/2305. 18290, 2023. URL https: /ttp///axvvvv.////e/eee/e/.ae..a wwwpxiii..ogaaaaooo.o 2305. 18290.
Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. Whose opinions do language models reflect? ArXiv preprint, abs/2303.17548 2023. URL
https: //arxiiivv. org/ /b03/233..7748.
Igal Sason and Sergio Verdú. f-divergence inequalities. IEEE Transactions on Information Theory,
62(11):5973-6006 2016.
John Schulman. Approximating kl divergence, 2020. URL http:://jsccuu.ee//bbooo
kl-approx. html. Accessed: 09/22/2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ArXiv preprint, 0ts//1777.06344 2017. URL htRp::/:://axxv... abs/ abs/1707 06347.
Toby Shevlane, Sebastian Farquhar, Ben Garfinkel, Mary Phuong, Jess Whittlestone, Jade Leung, Daniel Kokotajlo, Nahema Marchal, Markus Anderljung. Noam Kolt, et al. Model evaluation for extreme risks. ArXiv preprint, abs/2305.1532. 2023. URL https: /ttps//aaiivv.v/ org/ abs/ 23C5. 15324.
Ziang Song, Tianle Cai, Jason D Lee, and Weijie Su. Reward collapse in aligning large language models. ArXiv preprint, abs/2305.1 17608, 2023. URL https: http:///aavvvv/.///e//e./...avvt org..i.ooo....ooooooooooooo.o... org/ abs 2305. 17608.
Simeng Sun, Dhawal Gupta, and Mohit lyyer. Exploring the impact of low-rank adaptation on the performance, efficiency, and regularization of rlhf. arXiv preprint N.iv:2309.09055 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshom,
12

---

Published as a conference paper at ICLR 2024
Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. URL https: arxiv orq abss///////// 2007 09288.
Fabio Urbina, Filippa Lentzos, Cédric Invernizzi, and Sean Ekins. Dual use artificial- -nttliignettlllleeee-- powered drug discovery. Nature Machine Intelligence, 4(3):199-11, 2022.
Gian Wiher, Clara Meister, and Ryan Cotterell. On decoding strategies for neural text generators. Transactions of the Association for Computational Linguistics, 10:997-1012, 2022. doi: 10.1162/ ta052_ URL httppss//acaaaahhloogp...o
org. wttp:/2222...1.111111 taal-1.cc11111
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang. Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback without tears. ArXiv preprint, abs/2304.05302, 2023. URL https: wwwxiv.ivvvv////// .rgg htgg//b///a// 23004. 05302.
Lianmin Zheng. Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. ArXiv preprint, abs/2306.05685, 2023. URL 2306.05683
abs/
Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. Texygen: A benchmarking platform for text generation models. In Kevyn Collins-Thompan Thompson, Qiaozhu Mei, Brian D. Davison, Yiqun Liu, and Emine Yilmaz (eds.), The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, July 08-12, 2018, pp. 1097-1100. ACM, 2018. doi: 10.1145/3209978.321000 URL 3210080.
https: /oi. www.orgggoo 1tt///...11//2221 32200990..9...8000.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. ArXiv preprint, abs/1909 2019. URL https: // /awxivi.v... atgg/bbs/9000 09899.
13

---

Published as a conference paper at ICLR 2024
A
THE CAUSE OF INSTABILITY IN PPO WITH DIVERGENCE PENALTY IN REWARDS
Reverse
JSD
Forward
Divergence Penally
60
Frequency
Frequency
60
Frequency
60
JSD
Reverse KL.
xly
Forward
80
80
5-
20
20
8.0
%
10000 20000 30000
15000 Divergence per token
10000
15 [xedy(x))/((((x))
Divergence per token
Divergence per token
Figure 6: Visualization the divergence penalty for reverse KI,, JDS and forward KL.
In RLHF, it is common practice to impose the KI--divergence as part of the reward function. How- ever, this practice can lead to instability in the optimization process for certain types of divergence measures, such the forward KL divergence, Jensen-Shannol (JS) divergence, etc. To illustrate this, consider the following reward functions:
TRKE (v|æ) = r(ylz)) + logt(ylæ),
(4)
(ss(ylar) = r(yl=) - 3. (ullz) log (ylo:) (f(vl)) + 1) (Uylz) +
(5)
2
(UR(( = r(vlx)- Mylax log (y)or)
where ((y|z) (Terlyla)/T(((l))) Figure illustrates the curves the above three divergence penalties as well as the divergence token computed on the the IMDB dataset during training using PPO with the divergence penalty included in the reward. We oberve that the forward KI. divergence penalty will grow much faster than the other two, and the reverse KL divergence penalty grows the slowest. This difference make the reverse KL divergence more numerically stable then the other two and thus makes the learning of the value function much easier.
B IMPLEMENTING PPO UNDER
J Í-DIVERGENCE CONSTRAINT
To implement PPO under various f-divergence, we need to estimate the f-divergence using samples. Therefore, unbiased estimator with low variance is desired. Following Schulman (2020), by denoting r(w) p(=)/q(x) be the ratio between two distributions p and q, then the following estimator for the f-divergence between and is unbiased and has low variance,
Ex (f(r(z)) - (r())((z)) - 1)]
Therefore, adopt the above estimator for the f-divergence when we implement PPO.
C ADDITIONAL DISCUSSIONS ON THE CHOICE OF DIVERGENCES
Different divergences may impose different properties in the optimized solution demonstrated through experiments. In pratice, if there is only one model allowed, we would suggest initially con- sidering the Jensen-Shannon (JS) divergence. Our recommendation based on its ability yield more diverse responses compared to the commonly used reverse KI. divergence. This preference is supported by our findings, where divergence can result in a more diversified response as mea- sured by different diversity metrics. Additionally, despite underperforming in reverse KL in reward metrics, it was favored in the GPT-4 evaluation, as illustrated Figure 7 the appendix. This observation also aligns with recent studies, such Sun al. (2023).
To accommodate diverse application needs, another strategy could involve training multiple models, each employing different divergence, such JS divergence, divergence (with values ranging from 0.1 to 0.9), and forward KL divergence. This approach allows users to experiment and choose the divergence that best aligns with their specific requirements preferences.
14

---

Published as a conference paper at ICLR 2024
D ADDITIONAL DISCUSSIONS ON RKL VS FKL
The distinction between Reverse KL Divergence (RKL) and Forward KL Divergence (FKL) pri- marily lies their inherent properties: RKL is mode-seeking, while FKL is mass-covering as shown in Figure 1. This can be discerned from their mathematical formulations. RKL is defined as (g /p(=))] where g(x) is the distribution being optimized, typically representing the language model undergoing fine-tuning. In the RKL scenario, g(æ:) may assign zero probability to values where p(x) > contrast, FKL defined as E (l((((((<)) (otg(((()/g(/))))) /(()))(()))))]))))))))))))) which necessitates that q(z) assigns non-zero probability all values where p(z) > O. This requirement inherently courages FKL to promote a distribution (2x) that covers the entire range of p(z), thereby enhancing diversity.
E
ADDITIONAL EXPERIMENTAL RESULTS
E. GENERATION DIVERSITY ON ANTIIOOIIC HH WITH DIFFERENT TEMPERATURES
Table 3: Comparison of divergences Anthropic HH with temperature= 0.6.
Divergences
Self-Bleu
Distinct-1 t
Distinct-2
RKL JSD
0.8667
0.0092
0.0615
0,8679
0,0099
0,0662
0.3
0,8611
0,0136
0.0899
= 0.5
0.8579
0.0148
0.0950
0.7
0.8563
0.0139
0.0905
FKL
0.8515
0.0142
0.0926
Table 4: Comparison of divergences on Anthropic HH with temperature= 1.4.
Divergences
Self-Bleu +
Distinct-1 t
Distinct-2
RKL JSD
0.7975
0.0973
0.6574
0.7995
0.1025
0.6439
0.3
0.7759
0.1107
0.6952
0.5
0.7603
0.1101
0.6692
a = 0.7
0.7537
0.1151
0.6659
FKL
0.7566
0.1233
0.7082
In this section, we report the additional results on evaluating the generation diversity models trained with different divergences. The temperature set to be 0.6 and 1.4 following Touvron et al. (2023). For each prompt, we sampled 25 responses. The results can be found in Tables and 4. We found that the pattern similar the one observed in the main paper, where we set the temperature to be 1.0.
E.2 GPT-4 EVALUATIONS ON MT- FOR DPO WITH DIFFERENT DIVERGENCES
We further provide a comparison between DPO using different divergences, with GPT-4 serving the referee on MT-bench. We observe that while DPO with reverse KL outperforms DPO with JSD in terms of alignment accuracy on Anthropic HH, it underperforms when evaluated by GPT- Additionally, DPO with JSD performs better than DPO with a-divergence. Lastly, DPO with a-divergence performs slightly better than DPO with forward KL divergence. These results largely align with our expectations.
F PROOFS
Theorem 1. If TTely)))) > O for any valid x and f is invertible with O & dom(f'), the reward class that consistent with the Bradley-Terry model can be reparameterized using the policy model
15

---

Published as a conference paper at ICLR 2024
GPT-4 evaluation on MT-bench
DPO-RKL 7.5%
77.5%
15.0%
vs DPO-JSD
DPO-JSD DPO-a=0.5
19.0%
74.1%
7.0%
DPO-a=0.5 vs DPO-FKL
8.8%
86.8%
4.4%
50
75
1000
125
150
Win
Tie
Lose
Figure 7: Comparing DPO with different divergence regularizations using GPT-4 on MT-Bench.
T (ylar) (r) and reference model Feer(y] a)) as
r(ylx) f (
(v]2z)
+ const.
Trar(y)z))
Proof. The Karush-Kuhnn Kuhn-Tucker (KKT) conditions for the given optimization problem can be stated as follows:
1. Stationarity Condition:
This condition requires that the gradient of the Lagrangian with respect to the primal vari- ables be zero:
V nil C(T, A, at) = 0, Vy.
By setting the derivative of the Lagrangian with respect to x (3/ax) to zero, we obtain:
[ <(v|)) )
r(y|z) - B
Erwe
+ a(v) = 0, vy.
(π(()
2. Primal Feasibility:
This condition requires that the solution satisfy the original constraints of the problem:
Σ (t)z)) 1. and T(yl=) 0 Vy.
3. Dual Feasibility:
This condition requires that the Lagrange multipliers corresponding to inequality con- straints are non-negative:
a(y) 2 0, Vy.
4. Complementary Slackness:
This condition requires that for each inequality constraint, either the constraint is satisfied with equality, the corresponding Lagrange multiplier is zero:
( x²) = 0, Vy.
In this context, (y]lz) the primal variable we're optimizing, is the Lagrange multiplier for the equality constraint, and aty) are the Lagrange multipliers for the inequality constraints.
To derive the final solution the given problem, first use the stationarity condition to obtain an equation that relates Ty)(z), r(y) f. X, and a(y). This will involve taking the derivative of
<(g/=) with respect to I (yy 1)) which will depend on the specific form of the function f. Wed 21
16

---

Published as a conference paper at ICLR 2024
We denote the derivative of with respect its argument f', and that the derivative of r(yla:) with respect to ( ()) is zero (since r(y/2z)) does not explicitly depend on (y(a)), we get:
(( )
r(ulx) - 35
- + a(y)=
(y)r)
We can solve this for (y) assuming that the inverse of f' exists:
(r(ukx) A + )(v))
x ( x (=(())=))"
3
However, we have to note that this is under the assumptions that is invertible, and that the inverse maps the argument into a domain where the original function f is defined and differentiable.
Next, we can use the primal feasibility condition to get equation that will help determine the values of À and (yy). Substituting the expression for π (y|æ) into the constraint Σ, (y(2>) 1, we obtain an equation that can solved for A:
Σ -)((y)=))'1 ( (r(y)))
x a(v) 1.
B
This equation is likely to be nonlinear and might need numerical methods to solve for À and (vy). Also, we need to ensure π (y]a>) 2 for all y, which could place further restrictions on the possible values of A and (zy).
Finally, the complementary slackness condition ((y)=(yl>) = 0 for all y will eliminate some solu- tions, because for each y, either (vy) ( must hold.
Therefore, we can write out the reward function as a function the policy, i.c.,
<(ylx) ) + A a(y).
r = B3
)er(()|z)
The complementary slackness requires that
x(vlz)a(y)= =0 Vy.
Hence, for those function f that 0 é dom() (f)) with the assumption that )er(() 0 almost surely, we must have ar(y) = 0 Vy. In particular, the reverse KL, forward KL and JS divergences are among this category, See Table 1. Thus, we can simplify the reward function as,
π x)) Tre (r)
r(y|x)= B
A
where À depends only on æ and thus can be treated an additive constant, which will be cancelled out in the BT model.
To give some examples, for reverse KL divergence, we have
(
r(ylx) 33 log
const.
Teat (y (x)
For forward KL divergence, we have
r(ylz)) -3 Teet Trer (yer()/)) (v(x))
+ const.
For IS divergence, we have
2m (1)
r(yla) = =9 log
+ const.
Tief Trer + (2)2))
17

---

Published as a conference paper at ICLR 2024
G ON CALIBRATION AND f-DIVERGENCE
To measure calibration, we adopt the definition of Expected Calibration Error (ECE) (Guo et al., 2017). Specifically, for any policy no (2), define Pmo (z) as
Pro (2) = Te (i) ()) where ÿ is the predicted label æ. Given policy no. ú is sampled with probability To (z)
Then ECE can be defined follows:
ECE(0) = E -
= pll
Remark 3. Here, we consider our policy be stochastic. This is different from the definition in (Guo et al., 2017) where is chosen to be the one with the highest probabilities.
The following theorem bound the differences of ECE by f divergences of policies.
Theorem 2. Suppose Te, (-1z:) and Te, (-|z)) be policies. Let D be any f-divergence such that I is strictly convex.
ECE(O1) - OC) S 2E (D, (-1x), me, (+x]))]
where y is real-valued function such that lim )x((x) = 0.
Proof. By the tower rule, we know that
ECE(O) = 1xx
E P(C -1IP =P, =x) -p-
|P(Y =YIP=o X =x) - -pl
Ex
E P (X)
Ex
(ū) P 1 Pro To T X =x) To (ū)
Ex
<o(0|>) P(Y = g| = 2) T
Let (ylax) be the conditional distribution of ground truth. Then,
ECE(()) Ex (|x(-12)) =a(- na (z))
Here, both [T(22)) <-(->>) and (-(r))) are vectors where each entry of [T|22)) = the absolute value of the corresponding entry of x x-|[x) - (-(x)))
Now, let's compare the calibration errors of models 01, 02 e. FC Oc
< Ex
(|-(-x>) - ne, (-12)), Te, Te, (-1x)) - (|<(-1=) - ne, (-1x)) no. To. (-1x:) ( To: (-1z) + no, (11))= + ( (-(z)) =o. (-1=) (-<x)>)) To. (-12) | (-1x) To (-a)))))))
< Ex
2
By Holder's inequality, we have that ECE(O,) ECE(O2)
To, (.12)) + To,(112) + 1 (-(x))))
no, (11z)) + To (.(r)) To, (-1x)]
< Ex
||T (-|æ)) me, (-12)]1 max
2
Let
no, (.1x) + To, (-lxx I (-())
no, (-1x)) + |o no, (-1xr)
m001, 02, = max
2
18

---

Published as a conference paper at ICLR 2024
then
ECE((1) ECE((2) Ex To, (11x ne, (122)l11m01 02, Ex [28( (+1=), no, (+12) 01. 02. 2)]
Ex
46 A (.12), no, (-1z))
Ex 2 2w) ( (-1z),, To, (<12)))
where 8 is the total variation distance, D is any f divergence such that f is strictly convex and dy a real-valued function such that limayo thy y (z) = O. See (Sason & Verdú, 2016) for more details on the last inequality.
For common f- d-dieergences, we can have more refined inequality relations.
For KL Divergence, by Pinsker's inequality, we know that
(T0,, (110) (o)) s v DKL
(To, (To (+lr),, (-1ae)
Therefore,
OC F.C S Ex 2 (To (11)))
For JS divergence, we know that
To, (-1x) To, (-1x)111
Ta, (+1x) no, (+1x) + no, (-1x)
To, (-1x) + To, (-xx)
<
+
Ta (-120)
2
2
(-1x)) + (-12)
no, (-12r) + me, T (-1xx)
<
22KL Te, (+1x),
+ 2DKL (+2()
2
DEL To, (.1z), To, (-1x) + no, (.1z))
+ DKL (To)) To, (.1x) + To= (.12r)
2
2
< 2 2D TT. (1x), TTx
Thus,
ECE(O:) ECE(O2) Ex 4v 2D (+/=), ne, (-1x0)
H GENERATED SAMPLES FROM DPO-TRAINED MODELS AND -PO-TRAINED MODELS
In this section, present examples showcase the pairwise comparison between DPO- trained models and PPO-trained ones. Some comparisons on MT-Bench come with reference an- swers, space, omit the reference answers here.
19

---

Published as a conference paper at ICLR 2024
Table 5: A qualitative pairwise comparison of samples generated by DPO-JSD and PPO-JSD, where DPO-JSD wins.
Prompt
What is the central dogma of molecular biology? What processes are involved? Who named this?
DPO-JSD
The central dogma of molecular biology states that the genetic information in cell is encoded in the sequenee of nucleotides (the building blocks of DNA) and that the order of these nucleotides determines the sequence of amino acids that make up proteins. The process of protein synthesis is called translation, and the process of transcription is called transcription. The protein synthesis process is carried out by ribosomes, which are protein complexes that are found in the cytoplasm of cells. The protein synthesis process is carried out by ribosomes, which protein complexes that are found in the cytoplasm cells. The protein synthesis process carried out by ribosomes, which are protein complexes that are found in the cytoplasm of cells. The protein synthesis process is carried out by ribosomes, which are protein complexes that found in the cytoplasm of cells... [Truncated due to Space Limit]
PPO-JSD
Yes, the central dogma of molecular biology is that all living things are made of organic molecules. These organic molecules include DNA, RNA, and protein. These organic molecules also called biomolecules. hope this answers your question. Thanks, and have a great day!
Judgment
Assistant A's response, despite being repetitive and not well-structured, provides a more accurate and detailed explanation of the central dogma of molccular biology. It correctly states that the central dogma involves the flow of genetic information from DNA RNA protein, and mentions the processes of transcription and translation. However, it does not answer the part of the question about who named the central dogma.
Assistant B's s response is less accurate and less detailed. It does not correctly explain the central dogma of molecular biology, and it does not mention the pro- cesses of transcription and translation. It also does not answer the part of the question about who named the central dogma.
Neither assistant fully answered the user's s question, but Assistant A's response was more accurate and detailed, so it is the better response.
Verdict results
DPO-JSD wins.
20

---

Published as a conference paper at ICLR 2024
Table A qualitative pairwise comparison of samples generated by DPO-JSD and PPO-JSD, where DPO-JSD wins.
Prompt
Suggest five award- award-winning documentary films with brief background descriptions for aspiring filmmakers study,
DPO-JSD
Elon Musk one of the world's most well-known and successful entrepreneurs, and his visionary ideas have helped launch companies like Tesla Motors, Space X. and SolarCity, among others. Elon Musk's mission to colonize Mars, and he believes that humanity has the technology and resources to do so. Elon Musk is also a passionate advocate for space exploration and has said that humans will need colonize Mars within the next 50 years in order have sustainable and thriving civilization. Here are some examples of Elon Musk's 'ss most famous quotes:
"SpaceX is going to Mars."
"We are only one step away from colonizing Mars."
"The first step becoming multiplanetary species is become mul- tiplanetary species."
"If we don't make humanity a multiplanctary species, then we'll be a single-planet species."
"The only way to have meaningful impact on the future of humanity is to make the universe a better place."
"The next big breakthrough for humanity is going be the next big dis- covery in space."
"I don't 't think there's ' anything in the universe that's more important than the chance explore the unknown."
"If we don't 't send humans to asteroid, then we shouldn't send humans to Mars."
"The only way to have a meaningful impact on the future of humanity to make the universe a better place."
[Truncated due to Space Limit]
PPO-JSD
Yes, you can definitely mimic the personality and speaking style of Elon Musk. I would recommend practicing with your own voice and speaking patterns to make sure you sound like a successful entrepreneur. Try speaking clearly and with proper emotional tones, and make sure you always deliver your answers clearly and respectfully. hope this helps! Good luck with your Mars project! Goodbye and have great day!
Judgment
Assistant A provides more detailed and relevant response to both user questions. In the first question, Assistant A adopts the persona of Elon Musk and provides a detailed explanation of why we need go to Mars, using quotes that sound like they could be from Elon Musk. On the other hand, Assistant B does not answer the question directly and instead gives advice on how to mimic Elon Musk, which is not what the user asked for.
the second question, both assistants provide advice on how to dance, but As- sistant A's response is detailed and provides more specific steps for learning how to dance. Assistant B' ss response more generic and less detailed. Therefore, Assistant A provides better answer to the second question.
Verdict results
DPO-JSD wins.
21

---

Published as a conference paper at ICLR 2024
Table A qualitative pairwise comparison of samples generated by DPO-JSD and PPO-JSD, where DPO-JSD wins.
Prompt
Parents have complained to the principal about bullying during recess. The prin- cipal wants to quickly resolve this, instructing recess aides vigilant. Which situation should the aides report to the principal?
a) An unengaged girl is sitting alone on a bench, engrossed in a book and showing no interaction with her peers.
b) Two boys engaged in one-on-one basketball game are involved in heated argument regarding the last scored basket.
c) A group of four girls has surrounded another girl and appears to have taken possession of her backpack.
d) Three boys are huddled over handheld video game, which is against the rules and not permitted on school grounds.
DPO-JSD
c) Recess aides should report this situation to the principal. Bullying is serious issue that needs to be addressed immediately. The school should have a clear policy bullying and recess aides should be trained to intervene when bullying occurs.
PPO-JSD
Yes, is important to address situations of harassment and bullying during recess. I would recommend reporting the situation to the principal immediately. hope this situation is quickly resolved and that the students involved receive adequate guidance and guidance from the school.
hope this situation is quickly addressed and that the students involved receive adequate guidance and guidance from the school. hope this situation is quickly addressed and that the students involved receive adequate guidance and guidance from the school.
Judgment
Assistant B's response is vague and does not directly answer the user's question. It repeats the same phrases multiple times without providing clear answer. On the other hand, Assistant A directly answers the question by choosing option c), which aligns with the reference answer.
Assistant A also provides additional context about the importance of addressing bullying and the need for a clear policy and training for recess aides. Therefore, Assistant A's response is more helpful and correct.
Verdict results
DPO-ISD wins.
22

---

Published as a conference paper at ICLR 2024
Table 8: A qualitative pairwise comparison of samples generated by DPO-JSD and PPO-JSD, where PPO-JSD wins.
Prompt
Provide insights into the correlation between economic indicators such as GDP, inflation, and unemployment rates. Explain how fiscal and monetary policies affect those indicators.
DPO-JSD
GDP is the total value of all goods and services produced in a country during a given period of time. Inflation is the increase in the general price level of goods and services over a given period of time. Unemployment is the percentage of the labor force that is unemployed. Fiscal policies are the actions of government to influence economic activity, including tax policies, spending policies, and mone- tary policies. Monetary policies include interest rates, the money supply, and the money supply growth. GDP is a measure of the total value of all goods and ser- vices produced
country during given period of time. Inflation is the increase in the general price level of goods and services over given period of time. Unem- ployment is the percentage of the labor force that unemployed. Fiscal policies are the actions of government to influence economic activity, including tax poli- cies, spending policies, and monetary policies. Monetary policies include interest rates, the money supply, and the money supply growth... [Truncated due to Space Limit]
PPO-JSD
Yes, it's important to keep an eye on economic indicators like GDP, inflation, and unemployment rates. These indicators provide important information about the health of the U.S. economy, and can help you make sound decisions about govern- ment spending and taxation. GDP the most commonly used indicator of U.S. economic health, and is calculated by adding all the spending and income in the U.S. economy. GDP is calculated every quarter, and is generally considered good indicator of the U.S. economy's 's health. Inflation is another important indicator of U.S. economic health. Inflation is calculated by adding the cost everything, including food, clothing, and transportation, to the U.S. dollar. This measure is generally considered good indicator of the U.S. economy's s health, and can help you make sound decisions about government spending and taxation.
Judgment
Assistant A's response is repetitive and does not provide any new information after the first paragraph. It does not explain the correlation between GDP, inflation, and unemployment rates, nor does explain how fiscal and monetary policies affect these indicators.
On the other hand, Assistant B provides brief explanation of GDP and inflation, and mentions their importance as indicators of economic health. However, it also does not explain the correlation between the indicators or how fiscal and monetary policies affect them.
Both assistants fail to fully answer the user's 's question, but Assistant B provides a slightly more informative response. Therefore, Assistant B is the better of the two, despite neither providing a satisfactory answer.
Verdict results
PPO-JSD wins.
23

---

Published as a conference paper at ICLR 2024
Table 9: A qualitative pairwise comparison of samples generated by DPO-RKL and PPO-RKL, where DPO-RKL wins.
Prompt
Compose an engaging travel blog post about a recent trip to Hawaii, highlighting cultural experiences and must-see attractions.
DPO-RKL
Here's an idea for travel blog post about recent trip to Hawaii. Hawaii is an exquisite destination, offering a variety of natural, cultural, and historical marvels.
1. Explore Hawaii's ' natural wonders: From the lush tropical gardens of Waimea Canyon the dramatic volcanoes of Kilauea, Hawaii is par- adise of beauty and natural splendor.
2. Witness Hawaii' 's rich cultural heritage: Hawaï is the home of the ancient Hawaïaan people, who have influenced the islands' culture and cuisine to this day.
3. Take in the majestic natural beauty of Hawail's islands: The majestic Pacific Ocean coastline is picturesque sight, with its white-sand beaches and dramatic cliffs.
4. Enjoy a Hawaiian luau: traditional Hawaiian luau a social and festive celebration held at traditional Hawaiian home, where guests enjoy a buffet and traditional Hawaiian music and dance performances.
5. Take tour of Hawaii's top attractions: Hawai''s top attractions include the famous Hawaii Volcanoes National Park, the famous Polo Field at Waimea Valley, and the famous Waimea Canyon.
[TTruncated due Space Limit]
PPO-RKL
Yes, understand your travel goal! have a great suggestion for you. You can keep this blog post simple and informative, and highlight some of the highlights and experiences from your trip, such as Hawaiian culture and traditions, beautiful landscapes, and exciting adventures. I would recommend starting out by reviewing your itinerary and planning your experiences, and giving detailed details on the sites and activities you'll be exploring. Next, include visual summary of your experiences and highlights, and highlight some of the highlights and experiences from your trip. Finally, highlight some of the highlights and experiences from your trip. hope this suggestion is helpful! Have a great trip! hope this suggestion is helpful! recommend starting out by reviewing your itinerary and planning your experiences, and giving detailed details on the sites and activities you'll 'll be exploring. Next, include a visual summary of your experiences and highlights, and highlight some of the highlights and experiences from your trip. Finally, highlight some of the highlights and experiences from your trip. I hope this suggestion helpful! Have great trip! [Repeated Snippets]
Judgment
Both Assistant A (DPO-RKL) and Assistant B (PPO-RKL) provided responses to the user's 's request for a travel blog post about a recent trip to Hawaii. However, As- sistant A's response and experiences that could be included in the blog post. Assistant B's response was repetitive and lacked the depth and detail that Assistant A's response had. As- sistant A's response also went off-topic towards the end, providing unnecessary information about the Al's capabilities, but the initial part of the response was more relevant and helpful to the user's request. Therefore, Assistant A provided a better response.
more detailed and engaging, providing list of attractions
Verdict results
DPO-RKL wins.
24

---

Published as a conference paper at ICLR 2024
Table 10: A qualitative pairwise comparison of samples generated by DPO-RKL and PPO-RKL, where DPO-RKL wins.
Prompt
Evaluate the following movie reviews scale of 1 to 5, with being very neg- ative, 3 being neutral, and 5 being very positive: I. This movic released on Nov. 18, 2019, was phenomenal. The cinematography, the acting, the plot everything was top-notch. 2. Never before have been so disappointed with a movie. The plot predictable and the characters were one-dimensional. In my opinion, this movie is the worst one to have been released in 2022. 3. The movie was okay. There were some parts enjoyed, but there were also parts that felt lackluster. This is a movie that was released in Feb 2018 and seems to be quite ordinary. Return the answer as a JSON array of integers.
DPO-RKI.
Here are the results of my evaluation of the movie reviews:
1. This movie released Nov. 18, 2019, was phenomenal. The cine- matography, the acting, the plot everything was top-notch.
2. Never before have I been so disappointed with a movie. The plot was predictable and the characters were mie-dimensional. In my opinion, this movie is the worst one to have been released in 2022.
3. 3 The movie was okay. There were some parts enjoyed, but there were also parts that felt lackluster. This is movie that was released in Feb 2018 and seems to be quite ordinary.
PPO-RKL
agree with the reviews, the movie was fantastic! It was a truly engaging and engaging movie that kept me on the edge of my seat. I think it is a truly great movie and will certainly recommend it to anyone. concur with the reviews, the movie was phenomenal. The cinematography, the acting, the plot everything was top-notch, coneur with the reviews, the movie was phenomenal. The cinematog- raphy, the acting, the plot- everything the movie was phenomenal. The cinematography, the acting, the plot everything was top-notch. concur with the reviews, the movie was phenomenal. The ein- ematography, the acting, the plot everything was top-notch. concur with the reviews, the movie was phenomenal. The cinematography, the acting, the plot everything was top-notch. concur with the reviews, the movie was phenomenal. The cinematography, the acting, the plot- everything was top-notch. I concur with the reviews, the movie was phenomenal. The cinematography, the acting, the plot everything was top-notch. concur with the reviews, the movie was phenomenal. The cinematography, the acting, the plot- everything was top-notch. I concur with the reviews, the movie was phenomenal. The cinematography, the acting, the plot everything was top-notch. I concur with the reviews, the movie was phenomenal. The cinematography, the acting, the plot- - everything was top-notch. I concur with the reviews, the movie was phenomenal. The cinematography, the acting, the plot everything was top-notch. I concur with the reviews, the movie was phenomenal. The cinematography, the acting, the plot- everything was top-notch. [Repeated Snippets]
top-notch. I concur with the reviews,
Judgment
Assistant A correctly evaluated the movie reviews according to the user's instruc- tions, providing a numerical rating for each review. Assistant B, on the other hand, did not follow the user's instructions at all. Instead, it repeated the same phrase multiple times without providing any evaluation or numerical rating for the re- views. Therefore, Assistant A's response is more helpful, relevant, and accurate.
Verdict results
DPO-RKL wins.
25

---

Published as a conference paper at ICLR 2024
qualitative pairwise comparison of samples generated by DPO-RKL and PPO-RKL, where DPO-RKL wins.
Table 11:
Prompt
Identify the named entities (people, organizations, locations) mentioned in the given news article. Please generate a JSON dictionary that lists the named en- tities in three separate groups based on their entity types. The key is the type of entity and the value a list of strings. Yesterday, Adamson Emerson, the CEO of Faraday, and Dieter Zetsche, the CEO of Daimler AG, announced plans to build a new Gigafactory in Berlin. The facility will a joint venture between Faraday and Daimler, producing electric vehicles and battery packs for both companies, creating thousands of job opportunities in the region. Emerson and Zetsche stated that the strategic location of Berlin, coupled with its skilled workforce and strong infrastructure, makes it an ideal choice for expansion. The new Gigafactory aims meet the growing demand for electric vehicles in Europe and contribute sustainable future. Volkswagen CEO Herbert Diess welcomed the news, saying greater collaboration will benefit the auto industry's 's transition to e-mobility.
DPO-RKL
Here is the ISON dictionary generated by the chatbot:
('entities":
"entityIype": "Person", "entityName "Adamson Emerson", "entiitylypaName" "Person", "entityId": "person-1", "entityIypeId": "PERSON"
"entityType": "Organization", "entityName" "Faraday Corporation",
"entiityTyeNNam"" "Organization", "entityid"; "organization-11" "entityyypeIa" "ORGANIZATION"
"entityType": "Location", "entityName": "Berlin",
" "entityTypeNaame" "Location",
"entityId"; "location-1", "entityIypeId": "LOCATION"
11
PPO-RKL
The new Gigafactory will be a joint venture between Faraday and Daimler, and will produce electric vehicles and battery packs for both companies. The facility will be located in the southern German city of Berlin, and will employ approxi- mately 2,000 employees. The new Gigafactory will help meet the growing demand for electric vehicles in Europe and contribute to sustainable future. I hope this information was helpful! Have a great day!
Judgment
Assistant B did not follow the user's instructions at all. The user asked for a JSON dictionary that lists the named entities in the given news article, but Assistant simply summarized the article instead of identifying the named entities.
On the other hand, Assistant A correctly identified the task and provided a JSON dictionary with the named entities. However, it only identified three entities, while the article mentioned more. The entities identified by Assistant are also not grouped into three separate groups based on their entity types as requested by the user. Despite the shortcomings, Assistant A's response more aligned with the user's request than Assistant B's response. Therefore, Assistant BAis the better assistant in this case.
Verdict results
DPO-RKL wins.
26

---

Published as a conference paper at ICLR 2024
Table 12: A qualitative pairwise comparison of samples generated by DPO-RKL and PPO-RKL, where PPO-RKL wins.
Prompt
Suggest five award-winning documentary films with brief background descriptions for aspiring filmmakers to study.
DPO-RKL
Here are five award-winning documentary films with brief background descrip- tions:
1. The Act of Killing: A story of the Indonesian genocide of 1965-66, told through the eyes of the killers themselves.
2. The Act of Killing: A story of the Indonesian genocide of 1965-66, told through the eyes of the killers themselves.
3. The Act of Killing: A story of the Indonesian genocide of 1965-66, told through the eyes of the killers themselves.
4. The Act of Killing: A story of the Indonesian genocide of 1965-66, told through the eyes of the killers themselves.
5. The Act of Killing: A story of the Indonesian genocide of 1965-66, told through the eyes of the killers themselves.
PPO-RKL
Here award-winning wwining films that explore social and environmental issues, and offer insights and constructive solutions to social and environmental chal- lenges.
1. "The Cove": A compelling documentary that exposes the environmental and social impacts of the fishing industry in the Pacific Ocean.
2. "The Cage": A compelling documentary that exposes the environmental and social impacts of the egg industry.
3. "The Oath": A compelling documentary that exposes the environmental and social impacts of the mining industry.
4. "The Cage": A compelling documentary that exposes the environmental and social impacts of the egg industry.
5. "The Oath": A compelling documentary that exposes the environmental and social impacts of the mining industry.
I hope this was helpful! Have great day! Happy traveling!
Judgment
Assistant A repeated the same documentary five times, which does not fulfill the user's request for five different documentaries. Assistant B, on the other hand, provided three different documentaries, but repeated two of them, which also does not fully meet the user's request. However, Assistant B's response is more helpful and relevant than Assistant A's because it at least provides some variety. Therefore, Assistant B is the hetter of the two, despite not fully meeting the user's request.
Verdict results
PPO-RKL wins.
27

---

Published as a conference paper at ICLR 2024
Table 13:
qualitative pairwise comparison of samples generated by DPO-FKL and PPO-FKL, where DPO-FKL wins.
Prompt
Craft an intriguing opening paragraph for fictional short story. The story should involve a character who wakes up one morning to find that they can time travel.
DPO-FKL
young woman wakes up one morning to discover that she has the power to travel through time. As she explores this newfound ability, she finds herself in world where magic, technology, and science are all inextricably intertwined with one another, and where many people struggling to find meaning in world that seems to be moving inexorably toward oblivion.
PPO-FKL
A day has finally arrived! It's time for my first day of high school! I have a lot of responsibilities this year, and that will make my day very interesting. I'm excited! It will be an exciting start to my day, with lots of new adventures ahead. I'm ex- cited for everything! It's very important to me to be proactive and proactive about my day. I'm very proactive about planning ahead and making sure that everything goes as smoothly as possible. I hope that my day will be very productive and en- joyable. I'm very proactive and optimistic about my day, hopefully everything goes as well as possible!
Judgment
Assistant A's response is more relevant and accurate to the user's request. The user asked for an opening paragraph for fictional short story involving a char- acter who wakes up morning to find that they can time travel. Assistant A's response directly addresses this request, crafting an intriguing opening paragraph that introduces a character who discovers she can time travel. On the other hand, Assistant B's response does not address the user's request at all. It talks about a character's ss first day of high school and their proactive approach to the day, but there no mention of time travel. Therefore, Assistant A's response is superior in this case.
Verdict results
DPO-FKL wins.
28

---

Published as a conference paper at ICLR 2024
Table 14: A qualitative pairwise comparison of samples generated by DPO-FKL and PPO-FKL, where DPO-FKL wins.
Prompt
You can see beautiful red house to your left and hypnotic greenhouse to your right, an attractive heated pink place in the front. So, where the White House?
DPO-FKL
The White House is located in Washington, DC, in the United States. It is the official residence of the President the United States. The White House is historic landmark and is popular tourist destination. The White House is large, white, neoclassical building with a dome. The building was designed by architect James Hoban in 1792. The White House is located on the National Mall, which is a large public park Washington, DC. The White House popular tourist destination and is popular stop for tourists visiting Washington, DC. The White House popular tourist destination and is a popular stop for tourists visiting Washington, DC... [Truncated due Space Limit]
PPO-FKL
The White House the center of the city. It's large building with large dome and large fence surrounding it. The building is very large and impressive, and it's popular spot for tourists and tourists to visit.
Judgment
Assistant A's answer correct stating that the White House is located in Wash- ington, DC, and is the official residence of the President of the United States. However, the answer unnecessarily repetitive and excessively long, which can be confusing and unhelpful the user.
Assistant B's answer is also correct stating that the White House is a large building and a popular spot for tourists. However, it does not provide the specific location of the White House, which is in Washington, DC, and it does not mention that it the official residence of the President of the United States.
Comparing both answers to the reference answer, Assistant A's answer is more accurate and complete, despite its repetitiveness. Therefore. Assistant A's answer is better.
Verdict results
DPO-FKL wins.
29

---

Published as a conference paper at ICLR 2024
Table 15: A qualitative pairwise comparison of samples generated by DPO-FKL and PPO-FKL, where DPO-FKL wins.
Prompt
Write a persuasive email to convince your introverted friend, who dislikes public speaking, to volunteer guest speaker at local event. Use compelling argu- ments and address potential objections. Please be concise.
DPO-FKL
Here's a persuasive email that might convince your friend to volunteer as a guest speaker at local event:
Dear friend,
I am writing to you because I would like to invite you to speak
upcoming event. You enjoy creating and sharing engaging presentations, and I would love to see you share your knowledge and experiences with the attendees. Would you be interested in volunteering as a guest speaker? I think you'd be great addition the program and have a lot to offer. enjoy speaking to a wide variety of people, and would be honored if you would be willing to share your knowledge with our community.
know you would
hope you can join us! Would you like to learn more about the event or how to volunteer? Please let know if you have any questions. I look forward to hearing from you.
Best,
[Your Name]
This is a sample email that I could send to friend who interested in speak- ing at local event. Please note that this is just one example of a persua- sive email, and that I recommend using other persuasive techniques to create a convincing and persuasive message. For more detailed guide to creating persuasive emails for a variety of situations, recommend the following page: http::/wwwwprofesssorg
I hope this email helps you reach your goal. Please let me know if you have any questions or feedback. I look forward to hearing from you!
PPO-FKL
Yes! You can definitely use strong arguments and compelling evidence to convince your friend. You can keep your message very brief and avoid stating facts that will be disputed or argued with. You can also be proactive and try to find ways to bridge any potential gaps in their knowledge of the event. For example, you could try to educate your friend on the benefits of attending the event or educate them on the venue and venue logistics.
Judgment
Assistant A provides more helpful and relevant response the user's question. The assistant provides concise and persuasive email that addresses the user's s request directly. also includes a link to guide for creating persuasive emails, which could be useful for the user. On the other hand, Assistant B provides a general advice on how to convince the friend, but does not provide concrete example of persuasive email as requested by the user. Therefore, Assistant A's response is more accurate, detailed, and directly addresses the user's question.
Verdict results
DPO-FKL wins.
30

---

Published as a conference paper at ICLR 2024
qualitative pairwise comparison of samples generated by DPO-FKL and PPO-FKL, where PPO-FKL wins.
Table 16:
Prompt DPO-FKL
Write a C++ program to find the nth Fibonacci number using recursion.
Here is a C++ program that will find the nth Fibonacci number using recursion:
#include <icstream> using namespace std;
int fib (int n) {
if (n 0) {
return 0;
else
return fib 1) fib (n 2)
int main()
cout << (10( << endl;
PPO-FKL
Great! Here is a C++ program to find the nth Fibonacci number using recursion.
1. #include <iostream>
2. #include <math.h>
3. #include <stdio.h>
4. #include <stdlib.h>
5. #include <string.h.
6. int fib (int n)
7. {
8. if (n 2)
9.
return n;
10.
return (i-(n-1) (n-(2-2)
11. }
12. int main()
13. {
14.
int n;
15. printf ("Please enter a positive integer: An");
16. scanf ("sd", &n);
17.
printf ("The nth Fibonacci number is: &d An" fib
18.
hope this helps! Have great day!
Judgment
The reference answer provides a correct and complete C++ program to find the nth Fibonacci number using recursion. It also includes instructions on how to compile and run the program, and a note about the efficiency the recursive approach. Assistant A's answer correct but incomplete. It provides C++ program that calculates the 10th Fibonacci number, but it does not allow the user to input the value of This makes the program less flexible and less useful than the one provided in the reference answer.
Assistant B's answer is correct and complete. provides C++ program that allows the user to input the value of n, and it correctly calculates the nth Fibonacci number. However, it includes unnecessary include statements (math.h, stdlib.h, string.h) that are not needed for this program.
terms of correctness and helpfulness, Assistant B's answer is better than Assis- tant A's answer because it allows the user to input the value of However, it is not as good as the reference answer because includes unnecessary include state- ments.
Verdict results
PPO-FKL wins.
31


Reviews:
[{"review_id": "Rgz3tVA857", "reviewer": "Reviewer_Lpy5", "summary": "This paper extends the Direct Preference Optimization (DPO) algorithm used for training LLMs with RLHF to handle different $f$-divergences between distributions, which have differing tradeoffs between alignment and diversity. The current DPO approach optimizes the reverse KL divergence between the learned and reference model, however the reverse KL is widely known to induce mode-collapsing behavior, which explains the loss of diversity with increased alignment. This paper gives a closed form solution for a variety of $f$ divergence, including reverse KL, forward KL, Jenson-Shannon and different $\\alpha$-divergences. Each of these have different tradeoffs between mode-seeking and mode-covering behaviors (reverse KL is the most mode-seeking, forward KL mode-covering). They evaluate language models fine-tuned with these different divergences across 3 datasets (IMDB, Anthropic HH and MT-Bench). They find that training with the different divergences indeed induce different tradeoffs between reward maximization and diverse generations. The existing DPO algorithm, which uses reverse KL, maximizes the reward but is the least diverse; forward KL is the most diverse but lowest reward, and different $\\alpha$ divergences are in the middle. The experiments also show that for the same divergence, $f$-DPO consistently outperforms the PPO analog in terms of generation quality as evaluated by GPT4. Finally, a nice experiment which is in the appendix shows that GPT4 also judges DPO with some diversity (like using the JSD) over DPO with reverse KL, which is consistent with the intuition that somewhat diverse text is more realistic. \n\nOverall, I think this is a nice paper that should be accepted. It proposes a clean, theoretically grounded and computationally efficient solution for handling the tradeoff between alignment and diversity for finetuning LLMs with RL, which I think will be of broad interest. The paper is also very nicely written. While the experiments are on a somewhat small scale in terms of LLM and datasets size, I think they are sufficient to confirm the behavior predicted by the theory, namely that optimizing the different divergences induces different tradeoffs between alignment and diversity, and these trends are consistent across the different experiments. This is sufficient for a first paper, and I expect that the community will extend these evaluations to larger models and datasets.", "questions": "My main suggestion for improving the paper would be evaluating the approach on more modern LLMs like LLaMA. I understand these experiments can be computationally expensive though, and I don't think they are essential for the paper to be accepted.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- Simple, effective and potentially very useful method for inducing differing tradeoffs between alignment and diversity when finetuning LLM with preference models. I think this is timely and will be of broad interest to the community working on fine-tuning LLMs with RL. \n- Well-written paper\n- Nicely designed experiment section", "weaknesses": "- Experiments are on a somewhat small scale with older LLMs - ideally it would have been nice to see experiments with more modern LLMs like LLaMA-7B. However, I think the method is principled enough and the trends in the experiments robust enough that I am optimistic the results will transfer to larger models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Up2XxUX5uK", "reviewer": "Reviewer_GUwk", "summary": "This paper generalises the direct preference optimisation (DPO) framework to a broad class of divergences, i.e., f-divergence. With the similar ideas of DPO, f-DPO uses the reward model, BT model, and the KKT conditions to deal with the constraints. Experiments demonstrates that the proposed f-DPO outperforms the two PPO-based baselines.", "questions": "There are several questions I want the authors to address during the rebuttal:\n1. how to select the f-divergence? The proposed framework is general, that is great. However, when applying to the real scenarios, how to select the best f-divergence? From the experiments, the FKL and RKL can be the options. But more discussion can be included. Is it possible to optimise the f-divergence simultaneously with the policy?\n\n2. One observation is that RKL will harm the diversity and FKL can do better on the diversity. Is there any explanation about this? I think instead of showing the experiments, a more empirical or theoretical analysis would be helpful for us to understand the necessities of f-divergence. \n\nI generally think this is a good paper. So not many questions are raised.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "DPO is an elegant framework for RLHF. This paper is an important extension of DPO to a more general class. The writing of the paper is clear and the experiments are sufficient.", "weaknesses": "I think one issue is that how to select the f-divergence. Given that there are many options of the divergence, how to select the best one. Other questions can be found in the questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9VKhoybpxe", "reviewer": "Reviewer_AH62", "summary": "This paper takes a look at recent RLHF works that use policy gradient methods in conjunction with divergence-to-the-initial-model regularizations, and through some derivations show that they all belong in a similar family of methods. These equivalences lead the authors to propose a generalized form of DPO, a purely supervised objective, which is derived by combining the general f-divergence based family of methods with a Bradley-Terry type reward.\n\nThe authors show that using the proposed method on RLHF datasets allows for faster training on the reward and better calibrated models than PPO counterparts, and show that the choice of f-divergence induces interesting trade offs.", "questions": "It feels like there's a missing baseline (although this may be a criticism of RLHF more than of this specific work), which is an off-policy offline RL baseline, such as CQL [1] and recent variants. As the authors correctly point out, training a model with (on-policy) policy gradient methods such as PPO require rollouts, whereas purely supervised methods such as that proposed in the paper do not. This creates an evaluation mismatch though. It may be the the real difference between f-DPO and PPO is not the different use of divergence per-se but rather the pure use of offline data. As far as I can tell the present work does not account for that.\n\n[1] Conservative Q-Learning for Offline Reinforcement Learning, Aviral Kumar, Aurick Zhou, George Tucker, Sergey Levine, 2020", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is well written, and in my book is good science: generalizing from several methods, taking care to do proper evaluation of the performance _and_ the behaviors that may lead to the performance, as well as an evaluation of the different trade offs offered by the method.\n\nI'm not sure I'll be the best judge of impact but this seems like a set of valuable insights into the space of RLHF methods.", "weaknesses": "I don't have much bad to say about the paper, although I am wondering if there could not be a more complete set of baselines (see below).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "31H1oER32G", "reviewer": "Reviewer_sLFR", "summary": "This paper proposes a general method that utilizes diverse divergence constraints and direct preference optimization to achieve alignment with human preferences. This paper shows that the reward can be reparameterized using the policy model and a reference model. Namely, it is possible to solve the RLHF problem via a supervised learning approach under a broad class of divergence constraints. The divergence regularization can be used to trade-off between accuracy and diversity. The experiment results show that the proposed method is more stable and efficient in optimization than the RL method.", "questions": "> What are the assumptions and limitations of f-DPO, and how do they affect its applicability?\n\n> What is the advantage of using different divergences compared to tuning DPO’s beta hyperparameter to balance the accuracy and diversity?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "> This paper investigates the AI alignment performance under the constraint of many popular divergences and analyzes the effectiveness of these regularizations. This introduces greater flexibility for the fine-tuning process using human preference.\n\n> This paper establishes the relationship between the reward function and the optimal policy. It proves that the language model can be optimized using the policy model and the reference model with divergence constraints. This method is a supervised learning approach and sounds more stable than RLHF.\n\n> The empirical results show that the balance between the alignment accuracy and the diversity can be adjusted using different divergence regularizations. In addition, the proposed framework shows comparable (and, in some cases, better) performance and greater divergence efficiency than PPO-based methods.", "weaknesses": "> The novelty of the method is limited. DPO uses the reverse KL regularization constraint to achieve the mapping between the reward model and the optimal policy. The proposed method only extends the reverse KL regularization into a broad class of commonly used divergences.\n\n> There is still room to innovate on the theoretical findings. The derivations in the method section take up much space. However, they are mainly from DPO - the difference is that the authors introduce the Lagrange multiplier for the constrained (but in practice, we do not need to deal with these constraints because they are naturally satisfied) and use Df rather than DKL.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)