Direct Density Ratio Optimization: A Statistically Consistent Approach to Aligning Large Language Models
Authors: Rei Higuchi
Taiji Suzuki
Venue: ICML 2025 poster (2025)
Keywords: LLM Alignment
Statistical Consistency
Density Ratio Estimation
Unpaired Preference Data
Primary area: social_aspects->alignment

Abstract: Aligning large language models (LLMs) with human preferences is crucial for safe deployment, yet existing methods assume specific preference models like Bradley-Terry model.
This assumption leads to statistical inconsistency, where more data doesn't guarantee convergence to true human preferences.
To address this critical gap, we introduce a novel alignment method Direct Density Ratio Optimization (DDRO).
DDRO directly estimates the density ratio between preferred and unpreferred output distributions, circumventing the need for explicit human preference modeling.
We theoretically prove that DDRO is statistically consistent, ensuring convergence to the true preferred distribution as the data size grows, regardless of the underlying preference structure.
Experiments demonstrate that DDRO achieves superior performance compared to existing methods, showcasing its effectiveness and potential for significant improvement.
DDRO unlocks the potential for truly data-driven alignment, paving the way for more reliable and human-aligned LLMs.

Direct Density Ratio Optimization:
A Statistically Consistent Approach to Aligning Large Language Models
Rei Higuchi 12
Taiji Suzuki 12
Abstract
Recently, alignment methods for LLMs, such as Reinforce- ment Learning from Human Feedback (RLHF) (Ouyang et al., 2022; Christiano et al., 2017; Stiennon et al., 2020) and Direct Preference Optimization (DPO) (Rafailov et al., 2024), have yielded significant progress. RLHF learns a reward model based on human feedback and optimizes the LLM's policy via reinforcement learning to maximize this reward. DPO directly optimizes the LLM's policy from human preference data without explicitly learning a reward model, and it has been reported to be more computationally efficient and stable than RLHF.
Aligning large language models (LLMs) with hu- man preferences is crucial for safe deployment, yet existing methods assume specific preference models like Bradley-Terry model. This assump- tion leads to statistical inconsistency, where more data doesn't guarantee convergence true human preferences. To address this critical gap, we in- troduce a novel alignment method Direct Density Ratio Optimization (DDRO). DDRO directly esti- mates the density ratio between preferred and un- preferred output distributions, circumventing the need for explicit human preference modeling, We theoretically prove that DDRO statistically con- sistent, ensuring convergence to the true preferred distribution as the data size grows, regardless of the underlying preference structure. Experiments demonstrate that DDRO achieves superior perfor- mance compared to existing methods on many major benchmarks. DDRO unlocks the potential for truly data-driven alignment, paving the way for more reliable and human-aligned LLMs.
However, these existing methods rely on the assumption that human preferences follow a specific model (c.g., the Bradley-Terry model (Bradley & Terry, 1952)). This as- sumption may not accurately capture the complexity of real human preferences (Wu et al., 2022), leading to a funda- mental issue of a lack of statistical consistency. Statistical consistency refers to the property that, as the amount of data increases, the learned model converges to the true optimal model ( (Vapnik, 1998; Mohri al., 2012). Without statistical consistency, increasing the amount of data does not guaran- tee performance improvement, raising concerns about the model's 's reliability and safety. More concretely, even if one perfectly optimizes the loss function based on the Bradley- Terry model, it does not necessarily guarantee the accurate learning of the true underlying human preferences.
1. Introduction
Large language models (LLMs) have achieved remarkable performance across various natural language processing tasks, including question answering, text generation, and translation (Achiam et al., 2023; Chowdhery et al., 2023; Touvron et al., 2023). However, they can also generate outputs containing harmful information, biased opinions, and misinformation, posing potential risks to society (Ben- der et al., 2021; Weidinger et al., 2021; Bai et al., 2022). To develop safe and reliable LLMs, it is essential to align their behavior with human intentions, values, and ethics (Christiano et al., 2017; Gabriel, 2020; Glaese et al., 2022).
In this work, we aim address the statistical consistency problem in existing methods and develop more theoreti- cally justified alignment method that benefits from increased data. Specifically, we propose novel alignment method, "Direct Density Ratio Optimization (DDRO)," which directly estimates the distribution of preferable outputs (Sugiyama et al., 2012a; 2007b; Kanamori et al., 2009) without relying on human preference models. DDRO takes an approach that directly estimates the density ratio between the distri- butions of preferable and unpreferable outputs. This allows for direct alignment from data without depending on cer- tain human preference models. Our contributions can be summarized as follows:
'University of Tokyo. Tokyo, Japan "Center for Advanced Intelligence Project, RIKEN, Tokyo, Japan. Correspondence to: Rei Higuchi <higuchi-reic rei7 14@ wwr.g.cccc.uooo. tp.............pp...
Methodological Contribution: We propose a new pref- erence optimization method called DDRO that does not rely on any human preference model. The idea of our
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

DDRO: A Statistically Consistent Approach to Aligning Large Language Models
proposal is frame the preference optimization prob- lem into a direct density ratio estimation problem be- tween the preferred and unpreferred data distributions, and solve it by matching the true ratio and our estima- tor's ratio through minimizing their Bregman divergence. DDRO does not require paired data, which contrasts with such a method as DPO that requires paired data. We also discuss how our method is connected to existing work by investigating how the preferred and unpreferred data distributions interact in the loss function.
volves learning a reward model rg y) that predicts the quality of response y to a given prompt s. based on human preference data. This is achieved using a dataset of human comparisons D paired { V,", 3. ))²= where each data point (2,, V,+ V/ indicates that human annotators prefer response y," over y; for prompt T.. The objective of re- ward model learning is to acquire a reward function that consistent with human preferences.
In RLHF, the learning of the reward model employs a loss function (RV(Φ) based on the Bradley-Tere model. The Bradley-Terry model is used to model human preference probabilities. For responses VI: V/ to a prompt æ, the proba- bility Pr(y1 > y2/z) that response y/2 is preferred over y/2 is defined as
2. Theoretical Contribution: We theoretically prove that DDRO is statistically consistent, ensuring convergence to the true distribution as the number of data points increases, without dependence on any preference model. This stems from the usage of the Bregman divergence for density ratio estimation because it gives an upper- bound of the dis-dintece from the true distribution of the preferred data.
Pr(y1 y2|z)= = (ro (z, v1) rT(az y2))
where (2) = (1 + + is the sigmoid function.
3. Practical Contribution:
We validate DDRO across mul- tiple real-world benchmarks, showing that it achieves performance on par with or surpassing widely-adopted methods such KTO and BCO. Even when converting paired datasets into an unpaired format, DDRO nearly matches or outperforms DPO, which is a paired method.
Based on this model, the loss function (Ø) is formu- lated to minimize the expected binary cross-entropy loss in predicting human preferences in the paired dataset 'D puiredd
(Ø) = pun [log (T rT( y)]]
In the second stage, the learned reward model is used to optimize the LLM policy po x) using reinforcement learning algorithms. This is typically done using Proxi- mal Policy Optimization (PPO) (Schulman et al., 2017) by maximizing the following objective function:
The remainder of this paper is organized as follows. Section 2 provides background knowledge on LLMs and alignment, as well as existing alignment methods and their limitations. Section 3 describes the detailed theoretical framework of our proposed method, Direct Density Ratio Optimization (DDRO), including its simplified version, and discusses its statistical consistency. Section 4 presents the experimental setup and results to validate the effectiveness of our pro- posed method. Section 5 discusses the experimental results, the strengths and weaknesses of our method, and the signifi- cance and contributions of this research. Finally, Section 6 concludes this paper and outlines future directions.
max
-)]--KL[ppp x))lper(y | [x]].
0
(1)
Here, Per(y | x)) is the reference policy (often the initial LLM policy). and / is a hyperparameter that controls the strength of KL divergence regularization. The KL diver- gence term contributes to stabilizing training and mitigating reward hacking (Ziegler et al., 2019).
2. Related Work: Limitations of Existing Alignment Methods
While RLHF has been empirically successful, it has lim- itations such as high computational costs due to separate reward model and policy learning, and instability inherent in reinforcement learning (Rafailov al., 2024). To ad- dress these issues, Direct Preference Optimization (DPO) (Rafailov et al., 2024) was proposed. DPO aims to stream- line the alignment process by directly optimizing the policy from preference data, bypassing explicit reward modeling. DPO leverages the theoretical connection between reward functions and optimal policies, demonstrating improved stability and computational efficiency compared to RLHF. However, DPO inherits critical assumption from the under- lying Bradley-Terry model: that human preferences can be accurately modeled by this specific parametric form. This reliance on the Bradley-Tere Teery model raises concerns about statistical consistency. Specifically, even DPO converges to
Large language models (LLMs) have achieved remarkable progress in natural language processing. However, they also present risks, notably the generation of harmful or in- appropriate content. Aligning LLMS with human values and preferences is therefore a critical challenge (Christiano et al., 2017; Gabriel, 2020; Glaese et al., 2022). Reinforce- ment Learning from Human Feedback (RLHF) (Ouyang et al., 2022; Christiano et al., 2017; Stiennon et al., 2020) has emerged as promising paradigm for this alignment, demonstrating effectiveness in enhancing helpfulness, harm- lessness, and alignment with complex instructions (Zheng et al., 2024; Dai et al., 2023).
RLHF consists of two-stage process. The first stage in-
2

---

DDRO: A Statistically Consistent Approach to Aligning Large Language Models
the optimal policy under the Bradley-Terry model assump- tion as paired comparison data increases, convergence to the true optimal policy reflecting actual human preferences is not guaranteed if human preferences deviate from this model.
3. Direct Density Ratio Optimization (DDRO)
To avoid relying on a specific preference model such as Bradley-Terry model, we propose a new method called Direct Density Ratio Optimization (DDRO) that estimates the aligned model through density ratio estimation. The basic idea of existing preference optimization methods (DPO, PPO, KTO) is to estimate the density ratio between the aligned model and the given reference model. In- deed, the optimal solution for PPO (1) can be given as p' x exp(-ra/3)Pet that is, PPO obtains the solution p' via the density ratio p" /Picf which is proportional to exp(-ra/3) The same argument applies to DPO and KTO. Based this observation, our idea is to directly estimate the density ratio without going through any intermediate step such as estimating an artificial preference model (like Haudley-Terry model) or constructing a reward model. For that purpose, we utilized a technique developed in the field of direct density ratio estimation (Sugiyama et al., 2012b). To introduce our method, we begin by describing the prob- lem setting. We then derive the DDRO objective through Bregman divergence, explain its advantage from theo- retical viewpoint, and conclude by providing a practical, simplified variant of the approach.
Proposition 2.1. There exists a class of preferences, none of which can be obtained by minimizing the negative log- likelihood loss under the Bradley-Tery model assumption.
See Appendix A for the proof. Proposition 2.1 suggests that the performance of DPO may be fundamentally lim- ited by the validity of the Bradley-Terry model assumption. Furthermore, both RLHF and DPO rely on paired compar- ison data. Collecting such data at scale can be expensive and time-consuming, potentially hindering the scalability of these alignment approaches.
To mitigate the reliance on paired comparison data and po- tentially improve scalability, Kahneman-Te -aversya--Teersk Optimiza- tion (KTO) (Ethayarajh et al., 2024) introduced the use of unpaired preference data. Unpaired data, where individual responses are labeled with scores or binary feedback (e.g., "good" or "bad"", is often more readily available and scal- able as it can be collected from real- world user interactions. KTO aligns LLMs using such unpaired data by leveraging prospect theory (Kahneman & Tversky, 2013; Tversky & Kahneman, 1992). which captures the asymmetry in human value functions, particularly the higher sensitivity to losses compared to gains. While KTO offers a practical approach by utilizing more accessible unpaired data, it also assumes preference model based on prospect theory. Similar to DPO's reliance on the Bradley-Terrs model, the perfor- mance and statistical consistency of KTO are contingent on the validity of prospect theory assumption in capturing human preferences.
3.1. Problem Setting
The alignment of large language models can be formulated as an optimization problem aimed at bringing the model's output distribution closer to a desirable human preference distribution. Specifically, given a prompt xr from a prompt set X, our goal is to align the conditional distribution of responses y denoted as ps z)) with the unknown true preferred response distribution p² (y [x). This alignment is achieved through density ratio estimation in our method. We also denote pte (z) as the probability distribution of a prompt x E X. Here, we consider responses y to be elements of a response set y.
Assumption 3.1 (Assumption on Response Set). We as- sume that the prompt set X' and response set y are finite.
In contrast to DPO and KTO, which depend on specific parametric models for human preferences, our proposed approach aims to achieve statistical consistency without as- suming particular preference model. While we also utilize unpaired preference data as in KTO, the key novelty of our approach lies in its ability to learn a policy that is guaranteed to converge to the distribution of preferred outputs as the amount of unpaired data increases, regardless of whether hu- man preferences strictly adhere to Bradley-Terry, prospect theory, other predefined models. Such statistical consis- tency is crucial for ensuring the reliability and robustness of LLM alignment. Especially in real-world scenarios where human preferences are complex and diverse, and may be poorly modeled by simplistic assumptions, our approach offers significant advantage.
This is practically motivated assumption. For both prompts and responses, large language models operate on sequences of tokens from finite vocabulary V. Given the context window length C and generation length limit T, the size of possible sequences is bounded by D/|C for X' and DD for ), ensuring both sets are finite. Technically, this assumption is particularly required to ensure that we can calculate the density function (or more precisely, the probability mass function). As long we can obtain a density or mass function, then we may remove this condition.
Our goal is to align this model distribution with a true preferred distribution p (y | z), which represents the responses that humans find desirable. We also denote
3

---

DDRO: A Statistically Consistent Approach to Aligning Large Language Models
| a)) pT
the corresponding true unpreferred response distribution. Although p x) is unknown, we assume access to unpaired preference data D+ = (x v, > and D_ - (x; ) sampled from p- and p- respectively, where n = | and n- [D-|- We remark that our method requires only unpaired data and does not require a human preference modeling that describes the relation between pairs. This property is more like KTO rather than DPO which requires paired data.
be differentiable strictly convex function with f(1) 0. then we define the Bregman divergence loss for the density ratio estimator as
Bro = Be-ppe [Bregy (00 [19")]
where
Breg ((9l)) == f(ğ) - f(g) f' -)), g),
Pref
t
96 == 1-t pe
Here we consider a situation where the data are generated through the reference model Peet and human taggers give labels of "preferred" and "umpreferred." Then, we have the following relation between Pret. p* p":
1-t
From the convexity of J, Breg (0) is always non-negative and equals to O if and only if ge = g". (pt x px) px)-almost surely. Moreover, from Eq. (3), we can also notice that py = pt if and only if go g" This observation ensures that we may estimate the preferred data distribution g by minimizing the Bregman divergence loss CBreg Now, we also define pa - Per T- po. which corresponds to an estimator of the unpreferred data distribution p Then, we have the following statement.
Pref(y x)) = pp= (v 1))((+ppp())))p))pp) | +) 12)p(-11p1 (2)
where p(+ x)) and p(- | x) are the probability of pre-
ferred and unpreferred labels, respectively. As we have de- scribed above, the preference optimization can be reduced to a problem of estimating the density ratio r" ("(yzz) =
Pre((yl>e)) (y)=))) Indeed, if know r" (ylar), then
Proposition 3.3. Suppose that ê minimizes the Bregman divergence loss so that Lireg (0) = O. then it holds that
we can recover pt (y/zz as pt (ylx) Par(y/lz//"" (y/x).
From Eq. (2), the density ratio can be rewritten as r" (y
2) = pp(+ x) + p(- | x>) (vlz))
[ylx)) which means that es-
P^(y|z) pt (ulz), po(v|z) = p (ylæ)) ((p xpx)
timating r" is reduced to estimating another density ratio (M(l)) Hence, DDRO aims to estimate this density ratio ptiyla)
This motivates us to use the Bregman divergence loss for the preference optimization.
g'(v z) = p (vlي) p [y]=] to obtain r" using D4 and D_. Here,
we place the assumption that p(+ x)) is a constant, say I E (0,1). 1), for all input x.
Empirical estimate of the Bregman divergence loss. AL- though we know that minimizing the Bregman divergence loss leads to estimating the preferred data distribution pt, it is not obvious how to calculate the loss from the observed data. However, expanding the definition of the Bregman divergence, we see that
Assumption 3.2 (Assumption on Reference Distribution). The reference distribution Pret can be expressed as a convex combination of pt and p
Piet = tp- + t)p E 1).
f ((00)) - f(go) - ('(g0) - go0))p+(p)
In other words, the probability of preferred data is constant: p(+ x) = (Vx E X²).
- / - -((0) f'(90) p
(P--go) --0)
90 dp+ + (const.)
Under this condition, we can see g" and r" satisfy the fol- lowing relationship:
p+
-Ep+ 1-5( f' (go)go) - -Ep [g0 + (const.),
where Ep, I.] denotes expectation with respect to P= (x). Therefore, the Bregman divergence is repre- sented by the expectations with respect to the preferred and unpreferred data distributions, which can be well approxi- mated by the empirical average over the fine-tuning training datasets: The empirical risk Livy (0) is given by
g
(3)
1
Therefore, the true density ratio r" can be estimated by estimating g" from data. In the next section, we derive a Bregman divergence loss to estimate g", which is the key component of our proposal.
Livey (0) = n²
Σ
(-f(00 ))) I' (9o (10()))))
3.2. DDRO: The Proposed Method
2;
ED+
Following the argument described above, DDRO directly matches the ratio between po and Peet to the true ratio g" by minimizing the Bregman divergence loss: Let f TR. - R
(F(co(=7)))
7. ED_
4

---

DDRO: A Statistically Consistent Approach to Aligning Large Language Models
where we abbreviated the notation as go(z;) (o | mr)) for 2 (xi, 9t). This enables us to directly estimate the density ratio with the consistency guarantee (Proposition 3.3).
information approximation (Suzuki et al., 2008; 2009), and causal inference (Yamada & Sugiyama, 2010).
The Bregman divergence framework that we employed for DDRO was originally established in (Sugiyama et al., 201 2b) and is one of the most general framework for den- sity ratio estimation. This framework encompasses various density ratio estimation methods depending on the choice of Bregman divergence (i.e., the convex function f) and how the density ratio model is parameterized. Notable examples include Least Squares Importance Fitting (LSIF) (Kanamori et al., 2009), Kernel Mean Matching (KMM) (Gretton et al., 2009), and Kullback-Leibler Importance Estimation Pro- cedure (KLIEP) (Sugiyama et al., 2007b). Basically, we may employ any density ratio estimator for the preference optimization. However, we employ the logistic loss for our implementation due to its simplicity.
Since just minimizing the empirical risk Livg (f) will lead to catastrophic forgetting, we include a Kullback- divergence (KI-divergence) regularization in the objective:
(8) := Breg (0) + 7 KL (pe Pret):
(4)
where KT g)) = F., [og(p/)))- This is the population
version of our DDRO objective.
Analogously, its empirical version is also defined as
EDDRO (0) =
Σ (-S(go(z)) + ('(g((2)) ()900())))0(23
n+
EDED+
3.3. Practical modification of DDRO
+r(ga(*,t) (=lg0ee)='-og,00(tt))) logg)o(z))))
While the DDRO is effective in terms of consistency as shown in Theorem 3.3 and Theorem 4.1 below, we found that its training becomes unstable in preliminary experi- ments due to the appearance of gradient spikes during train- ing (see Appendix E). To mitigate this instability, we pro- pose a practical variant of the DDRO loss that introduces a monotonically increasing function S in the loss of each data point:
Σ ())()(zz))
TTED
+7(1 '))0=(27 log go((z)))
where the KI-divergence is also replaced by its empirical estimate based on Assumption 3.2 to represent Piet- We can see that various types of preference optimization meth- ods can be derived from our DDRO formulation (5) by examining various convex functions/ I for the Bregman di- vergence (sec Appendix F for an empirical investigation of this choice). This offers modeling flexibility depending on the data property.
EDDRO
1
Σ
(log(1 + g0(22) (2)
=
n
x ED+
+ - log (z
1
Σ
|s (log(1 + go" (2+)))
Example 3.4. Here, we give one particular example of the Bregman divergence. Suppose that we employ fx) = I log x - (1 + x)) log(1 + .t). Then the Bregman loss is reduced to the logistic loss:
n
27 ED
+ - t)gg 0) log go (6)
We employed S(x) = log(a(x)) for the sigmoid function a in our experiments, which showed good empirical results. We note that we may choose other possibilities for S. We provide a comparison of different S in Appendix E. The rationale behind the choice of log-sigmoid function is as follows: Applying S clips large values of go and its gra- dient, which prevents gradient spikes and also works as regularization, so that Pe does not go far away from the original reference model Pret- Indeed, we observed that this modification can mitigate gradient spikes (see Appendix E),
BBeg (0) = Ept [log(1 go)l + Ep- [log(1 + 90 2
Indeed, if we let "classifier" fo be fo log(1/9o). then the loss becomes Live(()) Ep- [log(1 + px)] + E, [log(1 + exp(fo))], which coincides with the loss of the logistic regression. We employ this logistic loss for our implementation. See Sugiyama et al. (2012a) for more choices of density ratio estimators.
Related work on density ratio estimation Direct density ratio estimation has been extensively studied in the literature (Sugiyama et al., 2012b). The idea behind this technique is to directly model the ratio between two densities p(x)/g(x) is more efficient and robust, especially in high-dimensional spaces. Density ratio estimation has been applied to various machine learning tasks, including covariate shift adapta- tion (Shimodaira, 2000; Sugiyama et al., 2007a), mutual
4. Theoretical Analysis
In this section, we give some theoretical analysis of our proposed method. First, we give the error bound of the estimated distribution via the density ratio estimator, which gives justification to use the Bregman divergence loss. Next, we investigate a connection to existing approaches by study-
5

---

DDRO: A Statistically Consistent Approach to Aligning Large Language Models
Table 1. Unified perspective on preference optimization objectives. Existing methods apply (partially) increasing and (partially) convex
functions such as sigmoid a (.), quadratic. - log or (const.. - For simplicity, omit the regularization terms.
Method
Reference
Loss Function
DPO
Rafailov al. 2024
log (a b) (a - b 1)2
IPO
Azar et al. 2024
SPPO
Wu et al. 2024
(a 1/2)2 + (-bb- 1/2)2
KTO
Ethayarajh et al. 2024
a(-a) + obb)
BCO
Jung et al. 2024
log o(a - 5) - log a (-b 6)
DDRO
Ours
-0 - b + (const.)
ing how the preferred data distribution and the unpreferred data distribution interact in the loss.
et al., 2012), the right hand side can be evaluated O /minfal _ + infoge Lerey
The term
4.1. Estimation error bound
inféce [greg(ge) represents how far the model is misspec- ified from the true distribution. If the true preferred data distribution is included in the model, this term vanishes. Hence, we see that the bound represents the bias and vari- ance trade-off to estimate an appropriately aligned model. As the data size increases, DDRO becomes more accurate even though man preference model.
First, we give an estimation error bound of our method with respect to the L2 that includes the model misspecifi- cation error and a generalization error. Here, let θ be the set of parameters of our model and its corresponding set of distributions be H. = {pe 0 E 6}. Let the Rademacher complexity of the model H. be denoted by R2 (H):
not putting any assumption the hu-
1 Σ (2') ADD E
R. (H):= max EDupp pEp*.P" -P-
Eg
sup OEE
4.2. Connection to existing methods
To elucidate the connection between our DDRO framework and existing methods, we first consider a simplified version of our loss by setting t = 1/2 and 7 0. Under these conditions, the DDRO loss (4) is reduces to
where (a,) an i.i.d. Rademacher sequence, that is, P(o; = = P(a. = -1) 1/2 (see Mohri al. (2012) for reffernncce is known that the Rademacher complexity measures the variance of the empirical risk minimizer that reflects the complexity of the model. Let p P-rPret T-P T-PP for p E H. and let h= R - R be h+ (p) -f (p/p) (p/p) I' (5/p) and h (p) f (7/p). Then, we have the following theorem.
(D)= 1Ept log 2 log
Pe Pref
+ +Ep
log 2 - -log
Pe Pref
To establish unified comparison, we follow the single-pair setup from Wu et al. (2024) where only one preference pair (x, "yt."" y") is available. Define the log-density- terms on the data points y+ and y
Theorem 4.1. Let = arg minsee LDDRO(() be the op- timal solution of the DDRO loss function with 7 = O. We assume that there exists some open bounded interval such that p(y|z)) p(y]x)) E I for all p E H, x E X. and y E y. We also assume that h- and h- are Lipschitz con- tinuous the interval I with Lipschitz constant Lip(h). Then, the estimation error of Pa with respect to the true preferred distribution nt satisfies:
IR
a= log Po |x)
Po x:) Prrrly
b log
Perrly
xx)²
x)
Then, the existing approaches can be summarized in Table 1. We see that existing methods can expressed as S(-a or S(-a) + S(b) using specific (partially) increasing and (partially) convex functions S(-).
ED [Igg - 10-1p112- p01122001]
1)2
pm
On the other hand, the simplified DDRO can be expressed as - b b+ (const.), where b is defined by
(Lip(h)(R (H) R,- -?() + inf Eureg(go) BE0
x
b log
where the expectation is taken over the realization of the pervised data D4, m4 minggxx. (-2)) p² (y a)) > 0. and == inf.er (s) > 0.
Pref (y
x))
The biggest difference from existing methods is that, while existing ones try to reduce fitting of py to unpreferred data by decreasing b, DDRO tries to increase fitting of po to the unpreferred data by increasing b. Without any assumption
The proof is given in Appendix C. Since usual Rademacher complexity bounds yield (H) = ((1/n)) (Mohri
6

---

DDRO: A Statistically Consistent Approach to Aligning Large Language Models
BBH (UF-G)
GSM8K (UF-G)
MMLU (UF-G)
ddm (DuPS)
boz
saro (AAPS)
cdin (puns)
0.1
py.ha-1.4b
py 96
9b
lbma-76
pythla22.80 pytr
pytrbb6.5bb
lana-7b
py/Ne-1.4b
pyfin..88
syfii-6 St
lamo-76
TruthfulQA (UF-G)
AlpacaEval LC Winrate (UF-G)
6.15
0.00
25 Ab
2yl1i-2.00
py210-..00
Barre-7b
Pyttab11.4
p/260-200
Tumr-77
Figure I. Performance comparison of three methods (KTO, BCO, DDRO) on various benchmarks (BBH, GSM8K, MMLU, TruthfulQA, and AlpacaEval LC Winrate). The methods are applied four different model sizes: Pythia 1.4B, Pythia 2.8B, Pythia 6.9,, and LLaMA 7B, each using the UF-G dataset.
on the human preference model, decreasing the value of b does not necessarily lead to good fit on the preferred data. On the other hand, our method gives good fit of both Pa and Pe to preferred and unpreferred data respectively, leading to its better explanation of the whole data. This property stems from the proper specification of the loss derived from the Bregman divergence that yields statistical consistency as shown in the previous section.
dataset is transformed into an unpaired dataset. The UF-B dataset is preference dataset built from the GPT-4 scored UltraFeedback dataset (64k prompts, each with 4 comple- tions). It is created by binarizing the dataset, where the highest scoring completion is labeled as "chosen" and a random completion is labeled as "rejected." This makes UF-B a paired dataset.
Models. We train the following base pre-trained LLMs: Pythia (1.4B, 2.8B, 6.9B) (Biderman et al., 2023), LLaMA 7B (Touvron et al., 2023).
5. Experiments
In this section, we evaluate our proposed method on two datasets: UF-G (umpaired) and UF-B (paired). We first demonstrate that DDRO can achieve performance compa- rable or better than existing unpaired methods (KTO, BCO) UF-G. Next, we investigate how well DDRO can approach the performance of a paired method (DPO) by converting the paired UF-B dataset into an unpaired format for DDRO training. Surprisingly, DDRO remains compa- rable or superior to DPO on all benchmarks, despite losing information when transforming paired data into unpaired data.
Benchmarks. We evaluate performance on several down- stream benchmarks that assess helpfulness, correctness,
or factual consistency. These benchmarks include BBH (Suzgun et al., 2022), which consists of subset of BIG- Bench (Srivastava et al., 2022) tasks; GSM8K (Cobbe et al., 2021), which features grade-school math problems; MMLU (Hendryeks et al., 2021), a multi-task language understanding benchmark; TruthfulQA (Lin al., 2022), which evaluates the truthfulness of responses; and Al- pacaEval LC Winrate (Dubois et al., 2024), which focuses on length-controlled preference evaluation. We employ Ll ama 3- 700 Instruct (AI@ Meta, 2024) as an evalu- ator to determine preferences between responses.
5.1. Experimental Setup
Datasets. We conduct experiments on two publicly avail- able datasets for LLM alignment: altrafeedback-gptt3... turbo-helpfulness (UF-G) and Sttrafeebback_binrrtee
5.2. Results on UF-G (Unpaired Dataset)
(UF- B). Both of these datasets available through dat library (Lhoest et al., 2021). The UF-G dataset is a fil- tered version of the UltraFeedback dataset (Cui et al., 2023), specifically filtered by helpfulness. After filtering, the UF-G
We first compare DDRO with existing unpaired methods. Figure shows that DDRO achieves performance on par with or exceeding KTO and BCO across four out of five benchmarks. One exception is observed the AlpacaEval
7

---

DDRO: A Statistically Consistent Approach to Aligning Large Language Models
BBH (UF-B)
GSMBK (UF-B)
MMLU (UF-B)
Apo
Adm
App
aloo
0.1
1/.0a-1.4b
pyfia:2.00 86
syfia66.50
Hbma-76
Pytrla-14b
pyttlo-2.8
pytrib5..bb
lana-bb
py/Ne-1.4b
pyfio.2.0
lomo-
TruthfulQA (UF-B)
AlpacaEval LC Winrate (UF-B)
1.15
0:00
25 Ab
2y1hi-2.00
py21o-8.00
Barre-bb
Turre-7
Figure 2. Performance comparison of DDRO and DPO on various benchmarks (BBH, GSM8K, MMLU, TruthfulQA, and AlpacaEval LC Winrate). The methods are applied four different model sizes: Pythia 1.4B, Pythia 2.8B, Pythia 6.9B, and LLaMA 7B, each using the UF-B dataset.
LC Winrate benchmark, where KTO slightly outperforms DDRO. These results confirm that DDRO highly compet- itive in the unpaired setting.
density ratio estimation from data. This framework over- comes the limitations of existing methods that rely on spe- cific preference model assumptions. By directly optimizing the density ratio between preferred and unpreferred distribu- tions using Bregman divergence loss, DDRO theoretically guarantees statistical consistency (Theorem 4.
5.3. Results on UF-B (Paired Dataset) vs. DPO
Next, we examine how effectively DDRO, an unpaired method, can approach or match the performance of paired method (DPO). For DDRO, ples into unpaired form by taking each prompt-completion pair separately (and discarding explicit pairwise preference links). This artificially creates an unpaired dataset from originally paired data.
Experimental results on multiple benchmarks (BBH, GSM8K, MMLU, TruthfulQA, and AlpacaEval LC Win- rate) confirm that DDRO compares favorably with prior methods, achieving on-par or better performance. It note- worthy that in comparisons using paired datasets, it achieved results comparable to better than DPO, despite DPO's ability to leverage preference information.
convert the paired sam-
Figure presents an interesting result: despite converting paired dataset into an unpaired form and thus discarding direct pairwise preference information, DDRO consistently performs comparably to slightly better than DPO across all evaluated benchmarks (BBH, GSM8K, MMLU, Truth- fulQA, AlpacaEval). This outcome is noteworthy because DDRO, by design an unpaired approach, manages to achieve performance on par with or exceeding a paired method even under conditions of inherent information loss, suggesting that the direct density ratio estimation technique effectively leverages the remaining preference signals.
Furthermore, a key feature of our proposed framework is the flcxibility to select the convex function f used in the Breg- man divergence. By changing the choice of f. we can vary the estimated divergence (Sugiyama al., 2012b). A more comprehensive investigation of its impact performance could potentially reveal new benefits. Indeed, (Huang et al., 2024) has shown that preference optimization using the x² divergence as a regularization term instead of the conven- tional KL divergence offers specific advantages, such as improved robustness.
Overall, our findings highlight the potential of DDRO to address fundamental challenges in LLM alignment. By discarding parametric assumptions on human preferences. DDRO enables truly data-driven alignment, leading to more reliable and human-aligned L.MMs.
6. Conclusion
In this paper, we addressed the critical issue of statistical inconsistency in the alignment of large language models (LLMs) and proposed Direct Density Ratio Optimization (DDRO), novel framework for alignment based on direct
8

---

DDRO: A Statistically Consistent Approach to Aligning Large Language Models
Acknowledgements
In International Conference on Machine Learning, pp. 2397-2430. PMIR, 2023.
RH was partially supported by JST CREST (PPMJC22015).
TS was partially supported by JSPS KAKENHI (24K02905) and JST CREST (JPMICR2115), This research is supported by the National Research Foundation, Singapore, Infocomm Media Development Authority under its Trust Tech Funding Initiative, and the Ministry of Digital Development and In- formation under the AI Visiting Professorship Programme (award number and conclusions or recommendations expressed in this ma- terial are those of the author(s) and do not reflect the views of National Research Foundation, Singapore, Infocomm Media Development Authority, and the Ministry of Digital Development and Information.
Boucheron, S., Lugosi, G., and Massart, P. Concentration Inequalities: Nonasymptotic Theory of Independence. OUP Oxford, 2013. URL https: /books....///////////..///////// google. co - jp/ books deriidkkoNNgMM http/boksi..kkooooo.r... uhPOC.
Bradley, R. A. and Terry, M. E. Rank analysis of incom- plete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 -45334444,1111 1952.
Any opinions, findings
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023.
Impact Statement
Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information pro- cessing systems, 30, 2017.
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Cobbe, K., Kosaraju, V., Bavarian, M., Hilton, J., Nakano, R., Hesse, C.,, and Schulman, J. Training verifiers to solve math word problems, 2021.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S.,
Cui, G., Yuan, L., Ding, N., Yao, G., Zhu, W., Ni, Y., Xie, G., Liu, Z., and Sun, M. Ultrafeedback: Boosting language models with high-quality feedback. 2023.
Anadkat, S., et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Dai, J., Pan, X., Sun, R., Ji, J., Xu, X., Liu, M., Wang, Y., and Yang, Y. Safe RLHF: Safe reinforcement learning from human feedback. arXiv preprint arXiv:2310.127755 2023.
AI@ Meta. Llama model card. 2024. URL https://github: wewpittuu.coo//eeeeee cww/meta- -attalllamen 1el-1aaaaallll 1ammamllaaaaa blob/mai httpmmaaii//////// /MOD//OODCCARARR md.
Azar, M. G., Guo, Z. D., Piot, B., Munos, R., Rowland, M., Valko, M., and Calandriello, D. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pp. 4447 4455. PMLR, 2024.
Dubois, Y., Galambosi, B., Liang, P., and Hashimoto, T. B. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint 240.00244. 2024.
Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kicla, D. KTO: Model alignment prospect theoretic optimization. arXiv preprint arXiv:2402.01300 2024.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training helpful and harmless assistant with rein- forcement learning from human feedback. arXiv preprint arXiv:2204.055822, 2022.
Gabriel, Artificial intelligence, values, and alignment. Minds and Machines, (033):41-433) 2020.
Bender, E. M., Gebru, T., McMillan-Men Major. A., and Shmitchell, On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and trans- parency, pp. 610-623, 2021.
Glaese, A., McAleese, N., Trębacz, M., Aslanides, J., Firoiu, V., Ewalds, T., Rauh, M., Weidinger, L., Chad- wick, M., Thacker, P.,
L., Ue- sato, J., Huang, P.-S., Comaneseu, R., Yang, F., See, A.. Dathathri, S., Greig, R., Chen, C., Fritz, D., Elias, J. S., Green, R., Mokrá, S., Fernando, N., Wu, B., Foley, R., Young, S., Gabriel, I., Isaac, W., Mellor, J., Hassabis, D., Kavukcuoglu, K., Hendricks, L. A., and Irving, G. Im- proving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209, 14375, 2022.
Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O' Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al. Pythia: A suite for ana- lyzing large language models across training and scaling.

---

DDRO: A Statistically Consistent Approach to Aligning Large Language Models
Gretton, A., Smola, A., Huang, J., Schmittfull, M., Borg- wardt, K., and Schölkopf, B. Covariate shift by kernel mean matching. In Quiñonero-Candela, J., Sugiyama, M., Schwaighofer, A., and Lawrence, N. D. (eds.), Dataser Shift in Machine Learning, pp. 131-160. 2009.
the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Lonng Papers), pp. 3214-
3252, Dublin, Ireland, May 2022. Association for Com- putational Linguistics. doi: 10. (0.11653/11/2022..ccm
229. URL https: http//aaaantttt wtw.hh/ooooooooooo.....o... or0/2022...
acl long. 229.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021.
Loshchilov, I. and Hutter, F. Decoupled weight decay regu- larization. arXiv preprint arXiv:1711.05100 2017.
Mohri, M., Rostamizadeh, A., and Talwalkar, A. Founda- tions of Machine Learning. The MIT Press, 2012.
Huang, A., Zhan, W., Xie, T., Lee, J. D., Sun, W., Krish- namurthy, A., and Foster, D. J. Correcting the mythos of kl-regularization: Direct alignment without overopti- mization via chi-squared preference optimization. arXiv preprint wrxiv:2407.13399 2024.
Ouyang, L.,, Wu, J., Jiang, X., Almeida, D., Wainwright, C., Míshkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models follow instructions with human feedback. Advances in neural information processing systems, 35:27730-2777 27744, 2022.
Jung. S., Han, G.. Nam, D. W., and On, K.-W. Binary clas- sifier optimization for large language model alignment. arXiv preprint arXiv:2404.0466 2024.
Rafailov, R., Sharma, A., Mitchell, E., Manning, D., Er- mon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Ad- vances in Neural Information Processing Systems, 36, 2024.
Kahneman, D. and Tversky, A. Prospect theory: An analysis of decision under risk. In Handbook of the fundamentals of financial decision making: Part I, pp. 99-127. World Scientific, 2013.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint vrXXiv:/77.063322 2017.
Kanamori, T., Hido, S., and Sugiyama, M. A least-squares approach to direct importance estimation. Journal of Machine Learning Research, 10:1391-1445, 2009.
Shimodaira, H. Improving predictive inference under covari- ate shift by weighting the log-likelihood function. Jour- nal of statistical planning and inference, 90(2):227-244, 2000.
Kato, M. and Teshima, T. Non-negative Bregman diver- gence minimization for deep direct density ratio estima- tion. In International Conference on Machine Learning, pp. 5320-5333, PMLR, 2021.
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso A., et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint (xXiv:200.006111 2022.
Ledoux. M. and Talagrand, M. Probability in Banach Spaces. Isoperimetry and Processes. Springer, New York, 1991.
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. Learning to summarize with human feedback. Ad- vances Neural Information Processing Systems, 33: 3008-3021, 2020.
Lhoest, Q. Villanova del Moral, A., Jernite, Y., Thakur, A., von Platen, P., Patil, S., Chaumond, J., Drame, M., Plu, J., Tunstall, L., Davison, J., Šaško, M., Chhablani, G., Malik, B., Brandeis, S., Le Scao, T., Sanh, V., Xu, C., Patry, N., McMillan-Major, A., Schmid, P., Gugger, S., Delangue, C., Matussière, T., Debut, I.., Bekman, S., Cistac, P., Gochringer, T., Mustar, V., Lagunas, F., Rush, A., and Wolf, T. Datasets: A community library for natural language processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: Sysiem Demonstrations, pp. 175-184, Online and Punta Cana. Dominican Republic, November 2021. Association for Computational Linguistics. URL tps:: //a //aclanthoo //aclanthology org/ 2021 emnlp memnlp-ddmo.... 21.
Sugiyama, M., Krauledat, M., and Müller, K.RR. Covariate shift adaptation by importance weighted cross validation. Journal of Machine Learning Research, 8(5), 2007a.
Sugiyama, M., Nakajima, S., Kashima, H., Buenau, P., and
Kawanabe, M. Direct importance estimation with model selection and its application to covariate shift adaptation. Advances in neural information processing systems, 20, 2007b.
Sugiyama, M., Suzuki, T., and Kanamori, T. Density Ratio Estimation in Machine Learning. Cambridge University Press, 201 2a.
Lin, S., Hilton, J., and Evans, O. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of
10

---

DDRO: A Statistically Consistent Approach to Aligning Large Language Models
Sugiyama, M., Suzuki, T., and Kanamori, T. Density-ratio matching under the Bregman divergence: unified frame- work of density-ratio estimation. Annals of the Institute of Statistical Mathematics, 64:1009-1044 2012b.
Yamada, M. and Sugiyama, M. Dependence minimizing regression with model selection for non-linear causal inference under non-gaussian noise. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 24, pp. 643-648, 2010.
Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y. Chung, H. W., Chowdhery, A., Le, Q. V., Chi, E. H., Zhou, D., and Wei, J. Challenging big-bench tasks and whether chain thoutht can solve them. arXiv preprint arXiv:2210.09200 2022.
Zheng, C.,, Sun, K., Wu, H., Xi, C., and Zhou, X. Balancing enhancement, harmlessness, and general capabilities: En- hancing conversational LLMs with direct RLHF. arXiv preprint arXiv:2403.02513 2024.
Ziegler, D. M., Stiennon, N., Wu, Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning language models from human preferences, arXiv preprint AxXiv:990.089000 2019.
Suzuki, T., Sugiyama, M., Sese, J., and Kanamori, T. Ap- proximating mutual information by maximum likelihood density ratio estimation. In New challenges for feature selection in data mining and knowledge discovery, pp- 5-20. PMLR, 2008
Suzuki, T., Sugiyama, M., and Tanaka, T. Mutual informa- tion approximation via maximum likelihood estimation of density ratio. In 2009 IEEE International Symposium on Information Theory, pp. 463-467, IEEE, 2009.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F,, et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302.13977 2023.
Tversky, and Kahneman, Advances in prospect theory: Cumulative representation of uncertainty. Journal of Risk and uncertainry, 5:297-323, 1992.
Vapnik, V. N. Statistical Learning Theory. John Wiley & Sons, New York, 1998.
von Werra, L., Belkada, Y., Tunstall, L., Beeching, E., Thrush, T., Lambert, N., Huang, S., Rasul, K., and Gallouédec, Q. TRL: Transformer reinforcement learn- ing. /ettps:/githhe 2020.
Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato,
J., Huang, P.-S., Cheng, M., Glaese, M., Balle, B.,
Kasirzadeh, A., Kenton, Z., Brown, S., Hawkins, W., Stepleton, T., Biles, C., Birhane, A., Haas, J., Rimell, L., Hendricks, L. A., Isaac, W., Legassick, S., Irving, G., and Gabriel, Ethical and social risks of harm from language models. arXiv preprint arXiv:2 /2. 04399 2021.
Wu, W., Niezink, N., and Junker, B. A diagnostic frame- work for the Bradley-Tern model. Journal of the Royal Statistical Society Series A: Statistics in Society, 185 (Supplement_2)ss 2):S461-4453 2022.
Wu, Y., Sun, Z., Yuan, H., Ji, K., Yang, Y., and Gu, Q Self-play preference optimization for language model alignment. arXiv preprint Xrv::22005. 00675, 2024.

---

DDRO: A Statistically Consistent Approach to Aligning Large Language Models
A. Proof of Proposition 2.1
Proof. We aim to demonstrate the existence of a class of preferences for which no member can be obtained through optimization under the Bradley-Terry model assumption. To this end, let us consider set of prompts X' {2² and a set of responses y {yar yb. Be3.
We define class of preferences Cpret parametrized by 1 E [0, 1/2) U (1/2, Each preference Pa E Cpeet is characterized by the specific pairwise comparison probabilities:
Pr(ya > ya/r] = Pr(ys > yc|2]] = Pr(ye > Valat t.
a preference Pt were obtainable, there must exist a reward & such that
a (ro (z. Va) ro yo)) = (r yb) - (2 ye)) = a (ro - ra( yo)) =
Here a is a sigmoid funcction. Due to the strict monotonicity of o, follows that
(z Ma) r (z, Mb) (a yh) T (z, y.) r1 Me) ro(x,, Ma).
Since these differences add up by canceling each other out, we have r (xr Ma) (2; y/b r ub) ra(a, Me) r (x Me) ra( Ma) 0,
which implies rT(2, No) (z mo) ro (2 we). Then, the model generates probability:
Pr(ya > 30|=] (ra( Va) r (z. 1b)) (0) = 1/2.
This contradicts E [0, 1/2) U (1/2, 1]. which indicates that P. cannot be obtained by optimizing under the Bradley- model assumption. This completes the proof.
B. Proof of Proposition 3.3
Proof. We are given that minimizes the Bregman divergence loss, CBreg(0). such that CBee (0) = 0. The Bregman divergence loss defined as L (f) Besps.seesspt [Breg (90 (ollll")) where f a strictly convex function. A fundamental property of the Bregman divergence is that Breg, (ge "lg"""") 2 O, with equality holding and only ge = g* Also, since the expected value of non-negative random variable is zero if and only if the probability of the random variable taking non-zero value is zero, 9g(ulæz) = g" (y/x²) holds (pt x palmst surely. The paper establishes direct relationship. stating that pe = p+ if and only if = g" Since we have shown 9g = g" (p- x pz) pal-almsst surely, it follows that Pa(y<æ) p* ()) (p+ x p..-almost surely. This proves the first assertion of the proposition.
Next, we examine the estimator for the unpreferred distribution, po. defined as Po T-aPee - T Substituting Ô for 0 and using the established result P3 = p- we obtain (o(() = - - --p (y]xx) Assumption 3.2 provides the decomposition of the reference distribution as (ylr)) tp+ (y]æc) + (1 - t)p (z)(z). Inserting this into the expression for Pa yields:
Po(1/=) - Itp (y|x) + (1 (y|=))
- 1 = P (y)z))
This equality holds (p+ x pz Jammott surely, as it relies on the condition P& (g((x) = p= (v (r), which itself holds (p+ x px)- almost surely. This demonstrates the second assertion of the proposition.
Thus, we have shown that if Liee (0) 0, then both p5(y(x) = p* (yl xr) and (3(()=) = p (w/x) hold ((p xpx) a.s.)
-p.))a.s.)..
C. Proof of Theorem 4.1
Proof. First note that, for any f E θ we have that
Lreeg () = LBreg (0) L (0) (0) + Leveg(0) + Creeg (0) L Breg (0) + LBver(O) - CBreg
Lreg (0) + (CBreg (0) L (0) (Crreg (0) Ć C (0) + (C (0) L (0)
(7)
(8)
(4s)
(His
12

---

DDRO: A Statistically Consistent Approach to Aligning Large Language Models
We that (iii) for any 0 E θ because minimizes Breg (0) in the model. Next, bound the terms (v) and (ii).
Since both terms can be bounded as
(i), (td)
sup AEe
(Beg (0) Cace (0)
we just need to bound the right hand side. Define the functions 1 (p; (x,, u)) and / (p; (z, v)) as
p(y z) ply x))
ply x p
P(y z)
h+ (p: (z, v))
+
x)
p(y x))
plu x))
h- ( (x,
-
p(u x))
Then, have that
Ep_ sup AE
Lieeg()) LBegg (8)
ED
sup pEH
Σ
(p; - Ept [h+ (p)]
n+
EtED
Σ EED
h ( -Bp [1 (p)
S ED,
sup pEH
Σ h+ (+5 - -Ept [b+
n
ED+
+ EE
sup pEH
Σ 2, ED_
h (p;z)) - Ep- [h (p)
71
Here, by the standard symmetrization trick (Theorem 3. Mohri et al. (2012)) and the contraction inequality (Theorem 11.6 of Boucheron al. (2013) or Theorem 4.12 of Ledoux & Talagrand (1991)), we have
ED
sup PE
Σ h (p: Epp [m]
< 2 (h+ hH) s 2Lip())R,, (7)).
π
Finally, the left hand side of Eq. (7) can be lower bounded by
11rg p 1
< (0),
2(1-))2 7
from Lemma C.1. Finally, by taking infimum over 0 E θ in the right hand side of Eq. (7), we obtain the assertion.
C.1. Auxiliary lemma
Lemma C.1. Suppose that the same assumption Theorem 4.1 holds. For any p H. there exists μ 0 such that
2(1 1)2
|pp - p+ 111/2100t) S m233
L (g)
where m4 minleex ((ssup(pp++(()) p (y) x) 0.
Proof. Lemma in Kato & Teshima 2021 states that for any p E H. there exists such that
llg g" " lla x C
(8)
13

---

DDRO: A Statistically Consistent Approach to Aligning Large Language Models
We now demonstrate that when g and g" are close, p and p+ are also close. Let A' be an arbitrary prompt. For notational simplicity, omit the conditioning æ.
| - 9r g0
pe P
=
P p*
12((pt)
(1 t)ppt (p - (p-pt)
Pref
L2(pt)
Due to the finiteness of V. p has positive minimum value on its support:
m4 min p (y) > 0
yEy
Furthermore, by Assumption 3.2, on the support of p_, we have Pret 2 tp" 2 tm+ Consequently:
119 9" 2'112a00t
2
(1 t)ppt (p pp*)
Prer
LP(pt)
tm+
(1 Oppt - x²))
12(D!)
m
2 (1 1)2 ||l p² 112 112
By combining this with (8), we obtain that
m
2 10-p1122007 (p(tt) S CBreg (g)-
||p 1p- pz
t)2
Rearranging terms, we find that
2(1 -112
||p x* (126t)) S
1m22,11 L (g).
D. Training Details
This section provides details regarding the training procedures employed for each model in experiments.
Training Setup: All models were trained using the TRL. library (von Werra et al., 2020). The batch size was set to 64, and training was conducted for 1 epoch across all methods and datasets. The AdamW optimizer (Loshchilov & Hutter, 2017) was used for optimization.
Learning Rates:
The learning rates for each method were as follows: 5e-7 for DPO, 5e-6 for KTO, 1e-6 for BCO, and 5e-7 for DDRO. For existing methods, we utilized the learning rate values reported in their respective papers (Rafailov et al., 2024; Ethayarajh al., 2024; Jung et al., 2024). It is important to note that the DDRO learning rate may not be optimal for every choice of the set S. Since the scale of the loss function can vary depending on the selection of S, the learning rate might need to be adjusted accordingly.
Other Details: Following the approach in Ethayarajh et al. 2024, we did not include the KI, regularization term in the gradient calculation for training stability.
E. Impact of the Choice of Smoothing Function S(x) on Training Stability
To analyze in detail the impact of the choice of the smoothing function S(x),, which was introduced in the loss function of our proposed method DDRO (Fquation (6)) to encourage stable learning, we conducted an ablation study. Theoretically, S(x) is expected to smooth the Bregman Divergence term in the loss function and stabilize the optimization process. As described in Section 3.3, while S(x) should ideally satisfy monotonicity and convexity from a theoretical perspective, practically, it is not always necessary to adhere strictly to these constraints, allowing for the exploration of various functional forms.
14

---

DDRO: A Statistically Consistent Approach to Aligning Large Language Models
We compared four functional forms for S(x). As a baseline without smoothing, used the identity function (S(x:) .²). We also tested S(x) (x), log or (z), log (-x:)
gradient
icontly
nadosplona
2020
1500
Figure illustrates the transition of the gradient norm during training for each choice of ((x). The figure reveals that with the identity function (5(x) = x), gradient spikes oceur during training, suggest- ing unstable learning dynamics. In contrast, when using the other functions, the gradient spikes are suppressed, confirming more stable training.
1030
spoth
0.2 0.4 0.6
0.8
These results demonstrate the crucial role of smoothing with S(x) for the stable learning of DDRO. The gradient spikes observed with the identity function (5(z) = 2) indicate that without smoothing, the loss landscape may become complex, potentially leading to convergence suboptimal solutions or unstable learning. Conversely, we observed stabilization of learning with any choice of other than the identity function.
Figure 3. Gradient norm during training for different smoothing functions S(x) (identity: S(x) s.
logsig: S(x) log)(z)) neglogsigneg: S(x)
log (-x), sig: S(x) (x)).
F. Impact of the Choice of Convex Function f in Bregman divergence on Benchmark Performance
In proposed framework (5), there remains flexibility in the choice of the convex function used for the Bregman divergence. This section investigates how performance various benchmarks changes when f is altered.
For the experimental setup, we trained Pythia 1.4B (Biderman et al., 2023) on the UF-G dataset (see Section 5) with different convex functions f. The candidates for f. referencing (Sugiyama et al., 2012b), are as follows:
((1) = log L - (1 + 1) log(1 + (), 1 log - 1, (L (4-1)) 1-)
(12+a - l)/a (a 0.1, 0.5, 1.0, 2.0, 5.0)
The learning rate was individually tuned for each function f by selecting from 5e-7, 5e-8, or 5e-9 to accommodate the variation in loss scaling.
The results are presented in Table We note that f(t) = t log t (1 + log(1 t), listed in the first row the table, is the function used in the main experiments of this paper, and its corresponding results are identical to those shown in Figure I and Table As indicated Table 2, AlpacaEval was the benchmark where the choice of led the most remarkable differences in performance. For the other benchmarks, the performance variations were generally within a range of 1-2%. Notably, the function employed in the main experiments of this paper demonstrated superior performance over the other functions in AlpacaEval, which supports the validity of our choice.
Table Benehmark performance of Pythia 1.4B with varying convex functions ft) for Bregman divergence. Scores are reported for BBH, GSMSK, MMLU, TruthfulQA, and AlpacaEval, alongside the learning rate (Ir) used for cach function. The function [(1) logg (1 + log(1 achieved the highest AlpacaEvaal score (bolded), which adopted in our main experiment.
s())
Ir
BBH
GSM8K
MMLU
TruthfulQA
AlpacaEval
t flog
(1 + 1) log(1 + 1)
5e-7
0.2593
0.0167
0.2416
0.2313
0.5836
t llog
5e-9
0.2594
0.0227
0.2426
0.2264
0.4962
(t - 1)2/2
5e-9
0.2603
0.0235
0.2423
0.2264
0.5000
(a 0.1)
5e-9
0.2591
0.0227
0.2425
0.2264
0.4977
(a = 0.5)
5e-8
0.2593
0.0220
0.2428
0.2264
0.4872
(1ita - 11/ar
(a 1.0)
5e-8
0.2599
0.0174
0.2436
0.2301
0,4742
(a = 2.0)
5e-9
0.2589
0.0220
0.2425
0,2264
0.4955
(a 5.0)
5e-8
0.2597
0.0205
0.2426
0.2264
0.4820
15

---

DDRO: A Statistically Consistent Approach to Aligning Large Language Models
G. Benchmark Performance Details
This section provides detailed numerical results for the experiments presented Section We report performance metries on five standard benchmarks: BBH, GSM8K, MMLU, TruthfulQA, and AlpacaEval (LC Winrate). The evaluated methods include DPO, KTO, BCO, and our method DDRO and we applied them Pythia and LLaMA model families.
In Table 3 and Table 4, for each model and benchmark combination, the highest score among the compared methods is indicated in bold.
Table Comparison of KTO, BCO, and DDRO performance across benchmarks training models UF-G dataset (unpaired), DDRO comparable to or outperforms other methods on all evaluated benchmarks except for AlpacaEval.
Models
BBH
GSM8K
MMLU
TruthfulQA
AlpacaEval
Pythia 1.4B
+ KTO
0.2634
0.0190
0.2432
0,1958
0.7364
+ BCO
0.2503
0.0159
0.2422
0.2228
0.4284
+ DDRO
0.2593
0.0167
0.2416
0.2313
0,5836
Pythia 2.8B
+ KTO
0.2797
0.0235
0.2539
0,1946
0.7214
+ BCO
0.2877
0.0212
0,2504
0,2032
0.4933
+ DDRO
0.2854
0.0265
0,2532
0.2240
0.5740
Pythia 6.9B
KTO 0.2758
0.0212
0.2526
0,1885
0.7341
BCO
0.2755
0.0220
0.2590
0,2301
0.6469
DDRO 0.2794
0.0265
0.2531
0.2338
0.5986
LLaMA 7B
KTO
0.3213
0.0591
0.2877
0.1701
0.6636
BCO 0.3345
0.0819
0.3123
0.2166
0.5108
DDRO
0.3330
0.0993
0.3182
0.2191
0.5280
Table Comparison of DPO and DDRO performance across benchmarks training models UF-B dataset (paired). DDRO comparable to or outperforms DPO regardless of the limitation that it cannot directly leverage the paired structure of the data.
Models
BBH
GSMSK
MMLU
TruthfulQA
AlpacaEval
Pythia 1.4B
DPO
0.2648
0.0167
0.2423
0.2277
0.5785
+ DDRO
0.2715
0.0182
0.2396
0.2277
0.6609
Pythia 2.8B
DPO
0.2907
0.0235
0.2485
0.2191
0.5886
+ DDRO
0.2910
0.0205
0.2582
0.2326
0.6695
Pythia 6.9B
+ DPO
0.2766
0.0303
0.2582
0,2191
0.5804
+ DDRO 0.2892
0.0296
0.2542
0.2375
0.6405
LLaMA 7B
DPO 0.3281
0.1008
0.3332
0.2215
0.5564
+ DDRO
0.3350
0.1001
0.3220
0.2240
0.5973
16


Reviews:
[{"review_id": "34it2YN7n8", "reviewer": "Reviewer_oR36", "summary": "This paper focuses on LLM alignment, aiming to address two key limitations of existing direct preference learning methods such as DPO: (1) Existing work requires a preference model (e.g., the Bradley-Terry model), which may not generalize well to capture complex human preferences that do not fit these models; (2) Existing work lacks statistical consistency guarantees, indicating they do not ensure convergence to an optimal model. To handle these challenges, the authors propose Direct Density Ratio Optimization (DDRO), demonstrating that the ground-truth reward can be represented as the density ratio between positive and negative response distributions. They develop a learning approach based on Bregman divergence to estimate this ratio. Theoretically, they prove that DDRO guarantees statistical consistency, and empirically, they demonstrate that DDRO performs on par with or better than existing baselines.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and constructive comments on our paper. We appreciate the opportunity to address your concerns and clarify aspects of our work.\n\n\n1.  **Is the density ratio $g$ an implicit preference model?**\n\n    We thank the reviewer for this insightful question. The density ratio $g = p^- / p^+$ reflects inherent preferences derived directly from the true distributions ($p^+, p^-$) without assuming a specific parametric structure (like the Bradley-Terry model used implicitly by methods like DPO or KTO). Therefore, $g$ represents the data's preference information itself, not a constructed model, allowing DDRO to avoid explicit preference modeling.\n\n\n2.  **Performance comparison and statistical consistency:**\n     We appreciate your suggestion. However, our primary theoretical claim regarding statistical consistency focuses on the convergence target – ensuring the learned model converges to the true optimal distribution under certain conditions – rather than the rate of convergence. Our experiments aim to show DDRO achieves competitive or superior performance on various benchmarks at a fixed data size, implicitly supporting the benefit of a sound convergence target.\n\n\n\n3.  **Justification for avoiding explicit preference models:**\n    We note that the very fact that our method demonstrates advantages on the majority of benchmarks, despite underperforming on a single benchmark, suggests that preference-modeling-methods may have become overly specialized to the particular types of preferences required only by certain benchmarks.\n    Also, we believe that demonstrating the limitation of specific models (like the Bradley-Terry model assumed in DPO) does not necessarily require datasets with highly complex preferences. It is sufficient to show scenarios where preferences, even simple ones, deviate from the assumed model structure. \n    For example, Bradley-Terry model does not capture a non-transitive relatoinship such as in rock-paper-scissors.\n    We acknowledge that including demonstration of such cases would strengthen our argument and therefore plan to add a discussion or a simple illustrative example to the camera-ready version.\n\n     -  [1] Pan, Y., Tsang, I. W., Chen, W., Niu, G., \\& Sugiyama, M. (2022). Fast and robust rank aggregation against model misspecification. Journal of Machine Learning Research, 23(23), 1-35.\n\n4.  **Proof for Proposition 3.3:**\n    We apologize for the omission. We had intended the paragraph preceding Proposition 3.3 to serve as an justification. We will provide a formal proof for Proposition 3.3 in the appendix of the camera-ready version.\n\n5.  **Missing citations for related work:**\n    Thank you for bringing these relevant papers to our attention.     We have reviewed them carefully.\n    We acknowledge these papers implicitly involve density-ratio through regularization (like odds ratios) or KL constraints.\n    However, these are primarily for stabilization or performance enhancement, and they do not appear to directly estimate or optimize the density ratio itself as the main target. \n    In contrast, the central contribution of our paper is the introduction of a consistent theoretical framework built directly upon the perspective of direct density ratio matching. \n    Therefore, while valuable contributions, we believe their direct relevance to our paper's central novelty is limited.\n\n\n6.  **Typos and missing references:**\n    We will correct these errors and conduct a thorough proofreading using automated tools and careful manual checks to identify and fix any other potential issues in the final version.\n\n\n\n7.  **Generation of negative samples in unpaired Ultrafeedback:**\n    Unpaired datasets like Ultrafeedback typically consist of triplets: a prompt, a generated response, and a binary label (e.g., \"good\" or \"bad\") assigned by human annotators. Therefore, responses labeled as \"bad\" constitute the set $D^-$.\n\n8.  **DDRO vs. KTO on AlpacaEval LC Winrate:**\n    We interpret this result as potentially stemming from KTO's underlying preference model (based on spectrum theory) aligning particularly well with AlpacaEval benchmark. This focused alignment might come at the cost of performance on other, broader benchmarks. In contrast, DDRO, by directly learning from the preference signals without an intermediate model, avoids such model-induced biases, leading to more robust and generally strong performance across a wider range of benchmarks, as observed in our experiments (e.g., BBH, GSM8K, MMLU, TruthfulQA).\n\n\n\n9.  **Potential for zero probability mass in Eq. (6):**\n    As mentioned in the paragraph just before Eq. (6), this can indeed occur and potentially lead to gradient spikes. To mitigate this, we introduced the function $S(\\cdot)$ which shapes the loss landscape to be more benign.\n\n\nThank you again for your detailed feedback, which will help us improve the final version of our paper.", "claims_and_evidence": "This paper makes two key claims:\n\n1. Existing SFT methods lack statistical consistency guarantees, indicating that even with sufficient training data, they do not necessarily converge to the optimal model. The authors provide a theoretical proof in Sec. 4.1 and partially justify it empirically in Sec. 5, where DDRO achieves comparable (or slightly better) results.\n\n2. Current methods rely on predefined preference models, such as the Bradley-Terry or Plackett-Luce models, making them unsuitable for capturing more complex human preferences. The authors derive DDRO theoretically in Sec. 3, demonstrating that it does not require such an explicit preference model.\n\nHowever, while both claims are reasonable and well-motivated, they are not well-supported empirically.\n* For Claim 1, although DDRO does not require an explicit preference model, could the density ratio g = p- / p+ itself be regarded an implicit preference model? Besides, in Fig. 1, DDRO—which is theoretically statistically consistent—does not achieve a significant advantage over other unpaired baselines, such as KTO and BCO. According to line 27 (right part), statistical consistency is defined as \"as the amount of data increases, the learned model converges to the true optimal model.\" A straightforward way to validate this would be to train DDRO and baselines with varying dataset sizes and compare its convergence speed to KTO, demonstrating both that statistical inconsistency is an issue in practice and that the proposed method effectively addresses it.\n* For Claim 2, the authors state in line 24 (right part) that \"This assumption may not accurately capture the complexity of real human preferences,\" which is intuitively plausible but not well justified. Experimentally, preference-model-based methods like KTO and DPO do not perform significantly worse than DDRO. To support this claim, the authors should evaluate their method on datasets with more complex human preferences and empirically demonstrate the benefits of not relying on a predefined preference model.\n\nGenerally, I believe this paper makes sufficient theoretical contributions, and the use of density ratio is also interesting. I will raise my score if the authors can address the concerns above.", "methods_and_evaluation_criteria": "As discussed above, the two claims are well-motivated and intuitively reasonable through the proposed method, with the authors providing theoretical proofs for some aspects. However, since this work is not purely theoretical, additional experimental results are needed to empirically validate these claims. See [Claims and Evidence].", "theoretical_claims": "The authors present theoretical claims in Sec. 3, Proposition 3.3, and Theorem 4.1. However, I could not find the proof for Proposition 3.3. I reviewed the derivations in Sec. 3 and did not notice obvious issues. I also briefly examined the proof of Theorem 4.1 and did not identify clear errors. However, as I am not an expert in learning theory, I cannot guarantee the correctness of the proofs.", "experimental_designs_or_analyses": "The experimental design in this paper does not fully support its claims (see [Claims and Evidence]). Additionally, several existing density ratio-like LLM alignment methods have not been cited. The authors should cite these works and discuss how DDRO compares to them. Ideally, they should include the comparison with at least one of these related methods. For relevant papers, see [Essential References Not Discussed].", "relation_to_broader_scientific_literature": "As far as I know, the two claims made in this paper have not been explicitly discussed in other LLM alignment research. However, several papers have already explored unifying different DPO variants within a unified framework, making this work closely related to those efforts. Specifically:\n\n* Tang et al., Generalized Preference Optimization: A Unified Approach to Offline Alignment. 2024.\n* Han et al., f-PO: Generalizing Preference Optimization with $f$-divergence Minimization. 2024.", "essential_references_not_discussed": "There are already some ratio-like loss papers. The authors should cite and discuss them. Specifically:\n\n* Zeng et al., Token-level Direct Preference Optimization. 2024.\n* Hong et al., ORPO: Monolithic Preference Optimization without Reference Model. 2024.\n* Xu et al., Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation. 2024.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "1. Line 165, left part, he -> the\n\n2. Line 357, right part, reference missing in MMLU", "questions_for_authors": "1. In the unpaired ltrafeedback-gpt-3.5-turbo-helpfulness dataset, how are the unpaired negative samples used in Eq. (6) generated?\n\n2. In Fig. 1, DDRO outperforms KTO in AlpacaEval LC Winrate. How should this result be explained?\n\n3. In Eq. (6), is there a possibility that some positive samples have almost zero probability mass in the LLM’s space? If so, how can this issue be mitigated?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I8iOoyZ2Ck", "reviewer": "Reviewer_v1jw", "summary": "This paper introduces Direct Density Ratio Optimization (DDRO), a method for aligning LLMs by directly estimating the density ratio between preferred and unpreferred output distributions. DDRO minimizes a Bregman divergence-based loss, eliminating the need for explicit preference modeling while ensuring statistical consistency. Empirical results show that DDRO performs on par or outperforms existing alignment methods in two settings: (1) unpaired preference data, compared to KTO and BCO, and (2) paired preference data, compared to DPO.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and questions regarding our work on DDRO. We appreciate the careful reading and valuable feedback.\n\n\n\n1.  **Handling neutral data:**\n    The reviewer asked if DDRO can handle \"neutral\" labels, which might arise when annotators evaluate single responses without direct comparison. Currently, DDRO is formulated to utilize binary preference signals (preferred/unpreferred) and it cannot directly incorporate a distinct \"neutral\" category. However, we believe that binary feedback collection is a prevalent and practical scenario. Major LLM services like Gemini, Claude, and ChatGPT often employ simple thumbs-up/thumbs-down mechanisms to collect data, which correspond directly to the binary setting DDRO addresses. Therefore, we consider the setting DDRO operates in to be sufficiently realistic and widely applicable.\n\n\n\n2.  **Determining $p(+|x)$ and robustness:**\n    The reviewer inquired about determining the constant $p(+|x)$ and DDRO's robustness to its value. We assumed $p(+|x)$ to be constant (denoted as $t$) for simplicity in our analysis. This constant $t$ can be empirically estimated by sampling responses from the reference policy $p_{\\text{ref}}$ for a set of prompts $x$ and calculating the proportion of responses deemed \"preferred\". However, our empirical investigations (which we can add details of in the final version) showed that the training dynamics and final performance of DDRO were not highly sensitive to the specific value chosen for $t$. Using a standard default value like $t = 0.5$ generally works well in practice, suggesting that precise estimation of $p(+|x)$ is often unnecessary.\n\n\n\n3.  **Scenarios for $g_\\theta$ misspecification and mitigation:**\n    The reviewer asked for intuitive examples where the learned density ratio model $g_\\theta$ might be misspecified, even when initialized from the reference policy. Misspecification can occur if the true preferred distribution $p^+(y|x)$ is too complex to be accurately represented by the chosen model architecture, given its parameterization (e.g., the LLM has insufficient capacity). In that case, the model might not be able to capture the true function even with infinite data. The bias term in our analysis (Section 3.2) relates directly to this discrepancy between the best possible model within the chosen hypothesis space and the true function; it depends on the model class, not the data distribution. Strategies to mitigate this bias primarily involve using model architectures known for their expressive power and ensuring the model has sufficient capacity (e.g., using larger models or architectures proven effective). Therefore, processing data (including prompt selection strategies) do not reduce the model bias, although they influence the data distribution which affects the variance term.\n\n\n\n4.  **Inconsistency between Figure 1 and Section 5.2:**\n    Thank you for pointing out this discrepancy. As you mentioned, the \"GSM8K\" in the text describing Figure 1 should have referred to \"AlpacaEval LC Winrate\". We apologize for this mistake and will correct it in the camera-ready version.\n\n\n\n5.  **DDRO underperformance on AlpacaEval:**\n    Following the correction above, the reviewer asked for insights into why DDRO might specifically underperform KTO on AlpacaEval LC Winrate. We hypothesize that KTO's specific preference model assumptions might align particularly well with the nature of the AlpacaEval benchmark. This could lead to strong performance on this specific benchmark, potentially at the expense of broader generalization. DDRO, lacking such model-specific biases, results in strong results across a wider array of benchmarks.\n\n\n\n6.  **Error in Equation (6) and MMLU reference:**\n    Thank you for identifying the potential error in the KL term in Equation (6) and the broken MMLU reference in Section 5.1. We will carefully review Equation (6) to ensure correctness regarding the preferred/unpreferred KL terms and fix the MMLU citation. We will perform a thorough proofread to catch any similar issues.\n\n\nWe appreciate the reviewer's constructive feedback, which will help enhance the clarity and correctness of our paper.", "claims_and_evidence": "The claims made in Section 1 of the paper are well supported by the methodology design, theoretical analysis, and empirical results.", "methods_and_evaluation_criteria": "**Strengths**\n\n1. Direct density ratio estimation is well-motivated for LLM alignment, as it eliminates the need for an explicit preference model which may not accurately reflect human preferences. It also eliminates the reliance on paired preference data, which can be costly to annotate.\n\n2. The formulation of DDRO is well-structured, and its derivation is clear and easy to follow.\n\n3. DDRO models are trained using UltraFeedback, a widely used public preference dataset for LLM alignment. The aligned LLMs are evaluated on five diverse benchmarks covering language understanding and generation tasks, making the evaluation setup comprehensive.\n\n**Questions**\n\n1. In more realistic settings, when human labelers annotate single generations without direct comparison, many responses may be classified as neutral—neither strongly preferred nor unpreferred. Could DDRO handle such neutral data?\n\n2. DDRO assumes a constant $p(+ | x)$. What would be a good way to determine $p(+ | x)$? How robust is DDRO to policies with different $p(+ | x)$?", "theoretical_claims": "I briefly checked the proof of Theorem 4.1 in Appendix A without verifying all details. The theorem’s insights on the bias-variance trade-off are interesting. \n\nI have a question regarding the bias term—could you provide some intuitive examples of scenarios where the model $g_\\theta$ might be misspecified? Given that data are generated from the reference model and subsequently annotated by humans, and that $g_\\theta$ is initialized from the same reference policy, what are the cases where $g_\\theta$ might fail to capture the true preference distribution? Additionally, are there ways to mitigate such misspecification through strategies like prompt selection?", "experimental_designs_or_analyses": "**Strengths**\n\n1. The evaluation across five benchmarks covering different language tasks is comprehensive.\n2. The experiment results are generally strong and support the claim that DDRO performs on par with or outperforms baseline alignment methods on both unpaired and paired preference data.\n\n**Questions**\n\n1. In Section 5.2, Figure 1 suggests that DDRO outperforms KTO on GSM8K for most model sizes, which appears inconsistent with the description in Section 5.2. Additionally, DDRO underperforms KTO across all model sizes on AlpacaEval. Could you provide insights into why DDRO might specifically underperform on AlpacaEval?", "relation_to_broader_scientific_literature": "LLM alignment is an important research direction for ensuring the broad utility and safe deployment of LLMs. This paper contributes to this field by introducing a preference-modeling-free approach, which avoids making parametric assumptions about human preferences. By directly inferring the preference distribution from data, the proposed method alleviates the need for explicitly paired preference data, reducing annotation costs and improving scalability. This provides a more generalizable and flexible framework for LLM alignment for broader applications.", "essential_references_not_discussed": "The paper discusses and compares most key methods in LLM alignment, as summarized in Table 1. One widely adopted preference optimization method, ORPO, is not cited. While ORPO is a reference-model-free approach and may not be directly comparable to DDRO, discussing its relevance could potentially provide additional context.\n\nHong, J., Lee, N., & Thorne, J. (2024). Orpo: Monolithic preference optimization without reference model. arXiv preprint arXiv:2403.07691.", "other_strengths_and_weaknesses": "Please refer to previous comments.", "other_comments_or_suggestions": "1. In Equation 6, the KL term for the preferred examples appears to be mistakenly written as the KL term for the unpreferred ones.\n2. In Section 5.1, the reference to the MMLU benchmark is broken.", "questions_for_authors": "Please refer to previous comments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uqmXarjLkH", "reviewer": "Reviewer_VDcC", "summary": "This paper introduces Direct Density Ratio Optimization (DDRO), a novel alignment method for large language models (LLMs) that addresses the statistical inconsistency of existing approaches reliant on restrictive preference models (e.g., Bradley-Terry). By directly estimating the density ratio between preferred and unpreferred output distributions, DDRO bypasses explicit preference modeling, ensuring statistical consistency—guaranteed convergence to true human preferences as data scales, regardless of the underlying preference structure. Theoretical analysis validates this property, while experiments across some benchmarks demonstrate the effectiveness of DDRO.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper and providing valuable feedback. We address your points below.\n\n\n1.  **Limited improvement and average results:**\n    The reviewer observed that the empirical improvement of DDRO over KTO seems limited and suggested providing average results across datasets. Regarding the magnitude of improvement, we would like to place this in the context of one-iteration preference optimization. It is generally understood that achieving dramatically large performance gains from a single iteration of alignment can be challenging. This characteristic of incremental progress is commonly observed in the field, as evidenced by recent works like ORPO, CPO, and SPPO, which also often report performance improvements of a similar nature after one optimization step. Viewed within this context – where substantial leaps are not always expected from one iteration – we believe DDRO demonstrates notable effectiveness. We wish to highlight three key points:\n\n    Consistency: DDRO achieves the best performance on the majority of benchmarks and model sizes tested.\n\n    Significant Gains: On specific benchmarks like GSM8K with Llama-7b, DDRO shows substantial improvements over KTO (over 50\\% relative gain) and BCO (over 20\\% relative gain), which we believe are non-trivial, especially in the context where large gaps are not always expected after one iteration of alignment.\n\n    Paired Setting: DDRO achieves performance better than DPO in the paired setting, despite discarding the comparison information that DPO utilizes. This underscores the effectiveness of DDRO.\n\n    Regarding averaging results: While technically feasible, averaging scores across different metrics and scales can be misleading. Metrics with smaller variance or lower chance rates might be unjustly de-emphasized. We believe presenting the full suite of results, as in Figure 1, provides a more transparent and informative picture of DDRO's broad effectiveness, rather than potentially obscuring performance variations through averaging.\n\n\n\n2.  **Comparison with GSIL [1]:**\n    Thank you for your comment regarding GSIL. While both GSIL and our method use density ratios, their objectives and approaches are fundamentally different.\n    First, construction of the objective is different. GSIL aims to minimize $\\mathrm{KL}(\\pi_{\\text{data}}) \\| (\\pi_\\theta)$, effectively making the density ratio $(\\pi_{data} / \\pi_\\theta)$ approach 1. Our method, however, aims to match the policy density ratio $(p_\\theta / p_{ref})$ to the true density ratio $(p_+ / p_{ref})$. Thus, our core objectives differ significantly.\n    Second, GSIL requires multiple iterations to estimate $\\pi_{\\text{data}}$, and its theoretical convergence is not guaranteed. In contrast, our method runs a single iteration, offering advantages in efficiency and theoretical tractability.\n    We hope this clarifies the distinction.\n    \n    \n\n\n\n3.  **Comparison with state-of-the-art methods like SPPO:**\n    We appreciate the suggestion to compare against state-of-the-art methods. However, our work primarily focuses on the setting where offline, binary feedback labels are available (i.e., unpaired preference data). Methods like SPPO operate under a different, richer setting. SPPO typically requires access to a pre-trained reward model, allowing it to use continuous reward scores, and often involves online interaction or scoring during training. Comparing DDRO directly with methods designed for such richer information settings might not be entirely appropriate, as the available data and feedback mechanisms are fundamentally different. Our goal was to develop a theoretically sound and practically effective method specifically for the common scenario of offline binary feedback, where methods like SPPO might not be directly applicable or would require significant adaptation.\n\n\nThank you again for your comments, which encourage us to clarify the positioning and strengths of DDRO.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "This paper studies the problem of direct alignment algorithms for unpaired data which is practical and necessary.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "### Strength\n\n1. The paper is well written and easy to follow.\n\n2. This paper studies a necessary problem of direct alignment algorithms on unpaired data.\n\n3. The proposed DDRO is reasonable and provide a novel perspective.\n\n### Weakness\n\n1. The author compares their empirical results of DDRO with KTO. The improvement seems limited. Can authors provide average results on all datasets to better demonstrate the effectiveness of DDRO.\n\n2. [1] also discusses the density ratio estimation. Can authors discuss the main difference and advantage of their method?\n\n3. In the experiment section, it's better for the author to compare with the state-of-the-art method like SPPO to further verify the effectiness of DDRO.\n\n[1] How to Leverage Demonstration Data in Alignment for Large Language Model? A Self-Imitation Learning Perspective.", "other_comments_or_suggestions": "Please refer to Strengths and Weakness part.", "questions_for_authors": "Please refer to Strengths and Weakness part.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xy8CzhdWk3", "reviewer": "Reviewer_KCaR", "summary": "This paper introduces a new offline reinforcement learning approach for LLM alignment, motivated by the lack of statistical consistency in the traditional BT model, which underpins popular offline algorithms such as DPO. The author proposes directly optimizing the density ratio using a Bregman divergence loss and proves that the estimated parameters exhibit good statistical consistency under certain assumptions. Experiments conducted on Ultrafeedback datasets show marginal improvements over existing offline RL methods.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and feedback on our paper. We address your points below.\n\n1.  **Performance concerns:**\n    The reviewer expressed concern that DDRO does not demonstrate compelling advantages over existing models, potentially limiting its practical applicability. Regarding the magnitude of improvement, it's important to evaluate this within the context of single-iteration preference optimization. We believe that achieving dramatic performance gains from just one optimization step is challenging. In fact, this tendency for incremental progress is common in the field (see works such as ORPO[1], CPO[2], and SPPO[3], which often report similar levels of improvement after one optimization step).\n\n    Considering this background – where substantial leaps are not always expected from a single iteration – we believe DDRO's results are noteworthy. We highlight three key points demonstrating its effectiveness:\n\n    - Broad Consistency: DDRO achieves the best performance on the majority of benchmarks and model sizes tested. This result could indicate that explicitly modeling preferences causes a failure to model the preferences necessary for many benchmarks, limiting good performance only to those benchmarks where the model luckily matches the required preferences.\n\n    - Significant Gains on Specific Tasks: On specific benchmarks like GSM8K with Llama-7b, DDRO shows substantial improvements over KTO (over 50\\% relative gain) and BCO (over 20\\% relative gain), which we believe are non-trivial, especially in the context where large gaps are not always expected after one iteration of alignment.\n\n    - Effectiveness in Paired Settings: DDRO achieves performance better than DPO in the paired setting, despite discarding the comparison information that DPO utilizes. This underscores the effectiveness of DDRO.\n\n    References:\n\n    - [1] Hong, J., Lee, N., & Thorne, J. (2024). ORPO: Monolithic preference optimization without reference model. arXiv preprint arXiv:2403.07691.\n    - [2] Xu, H., Sharaf, A., Chen, Y., Tan, W., Shen, L., Van Durme, B., ... & Kim, Y. J. (2024). Contrastive preference optimization: Pushing the boundaries of llm performance in machine translation. arXiv preprint arXiv:2401.08417.\n    - [3] Wu, Y., Sun, Z., Yuan, H., Ji, K., Yang, Y., & Gu, Q. (2024). Self-play preference optimization for language model alignment. arXiv preprint arXiv:2405.00675.\n\n2.  **Proofreading errors:**\n    Thank you for pointing out the specific typo (\"he\" -> \"the\" in Line 165) and the broken MMLU citation (Line 358). We apologize for these oversights. We will meticulously proofread the manuscript, correcting these and any other errors, before submitting the camera-ready version.\n\n\nWe appreciate your feedback and hope these clarifications address your concerns about DDRO's performance and practical relevance.", "claims_and_evidence": "The primary claim is well supported by the theoretical analysis.", "methods_and_evaluation_criteria": "The proposed method is theoretically sound, and the empirical studies are well-designed. However, my concern lies in its performance. While the work establishes a solid theoretical foundation for offline RL methods, the proposed approach does not demonstrate compelling advantages over existing models, which significantly limits its practical applicability.", "theoretical_claims": "Yes, I checked the proof shown in Appendix.", "experimental_designs_or_analyses": "The experiments are well-designed, but the performance of method indicates that the work is incremental to the research of alignment.", "relation_to_broader_scientific_literature": "The proposed model could be a valuable addition to the family of offline RL methods. The authors clearly establish connections with existing approaches and provide solid theoretical analysis, which is often lacking in prior work。", "essential_references_not_discussed": "Most of the related papers are well cited.", "other_strengths_and_weaknesses": "see my comments above", "other_comments_or_suggestions": "The draft needs further proof-reading. \nFor example:\nLine 165: Here we consider a situation where he data...-> `he' must be `the'\nLine 358: the citation for MMLU is a '?'", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Reviewers broadly agree the paper is clear, with sound theory and empirical evaluations. 

I think the paper could have engaged more deeply with the large body of related work on optimizing using preference labels directly, and the impact would be improved by giving this some attention prior to publication.
Decision: Accept (poster)