Preference Optimization by Estimating the Ratio of the Data Distribution
Authors: Yeongmin Kim
HeeSun Bae
Byeonghu Na
Il-chul Moon
Venue: NeurIPS 2025 poster ()
Keywords: Direct Preference Optimization
Language Models
AI Alignment
Primary area: deep_learning
TL;DR: We propose BPO; a generalized DPO objective based on Bregman divergence from the perspective of likelihood ratio estimation.

Abstract: Direct preference optimization (DPO) is widely used as a simple and stable method for aligning large language models (LLMs) with human preferences. This paper investigates a generalized DPO loss that enables a policy model to match the target policy from a likelihood ratio estimation perspective.  The ratio of the target policy provides a unique identification of the policy distribution without relying on reward models or partition functions. This allows the generalized loss to retain both simplicity and theoretical guarantees, which prior work such as $f$-PO fails to achieve simultaneously. We propose \textit{Bregman preference optimization} (BPO), a generalized framework for ratio matching that provides a family of objective functions achieving target policy optimality. BPO subsumes DPO as a special case and offers tractable forms for all instances, allowing implementation with a few lines of code. We further develop scaled Basu's power divergence (SBA), a gradient scaling method that can be used for BPO instances. The BPO framework complements other DPO variants and is applicable to target policies defined by these variants. In experiments, unlike other probabilistic loss extensions such as $f$-DPO or $f$-PO, which exhibits a trade-off between generation fidelity and diversity, instances of BPO improve both win rate and entropy compared with DPO. When applied to Llama-3-8B-Instruct, BPO achieves state-of-the-art performance among Llama-3-8B backbones, with a 55.9\% length-controlled win rate on AlpacaEval2. Project page: https://github.com/aailab-kaist/BPO.

Preference Optimization by Estimating the Ratio of the Data Distribution
Yeongmin Kim' Heesun Bae' Byeonghu Na' II-Chul Moon 1.2
Korea Advanced Institute of Science and Technology (KAIST), Ssummary.ai (alsdudrla10, cat2507, byeonghu. na, icmoon] ar.
Abstract
Direct preference optimization (DPO) is widely used simple and stable method for aligning large language models (LLMs) with human preferences. This paper investigates generalized DPO loss that enables policy model to match the target policy from a likelihood ratio estimation perspective. The ratio of the target policy provides unique identification of the policy distribution without relying on reward models or partition functions. This allows the generalized loss to retain both simplicity and theoretical guarantees, which prior work such as f-PO fails to achieve simultaneously. We propose Bregman preference optimization (BPO), generalized framework for ratio matching that provides a family of objective functions achieving target policy optimality. BPO subsumes DPO as a special case and offers tractable forms for all instances, allowing implementation with a few lines of code. We further develop scaled Basu's power divergence (SBA), a gradient scaling method that can be used for BPO instances. The BPO framework complements other DPO variants and is applicable to target policies defined by these variants. In experiments, unlike other probabilistic loss extensions such as f-DPO f-PO, which exhibit trade-off between generation fidelity and diversity, instances of BPO improve both win rate and entropy compared with DPO. When applied to Llama-3-8B-Inse Instructt,, BPO achieves state-of-the-an performance among Llama-3-8B backbones, with a 55.99 5.9% length-controlled win rate on AlpacaEval2. Project page: https: httt::://gtthbb com/
Introduction
Aligning large language models (L.LMs) with human feedback has emerged as a promising fine- tuning paradigm to better reflect human preferences [1, 18, 56). Reinforcement learning from human feedback (RLHF) [13, 47] is widely adopted alignment method that bridges implicit human preferences and model behaviors. This method typically involves two stages after supervised fine- tuning: (1) training reward model that captures implicit human preferences, and (2) optimizing LLMs through reinforcement learning pipeline guided by the learned reward model. This multi- stage RLHF pipeline is computationally intensive and prone to instability, motivating the development of direct preference optimization (DPO) [49] as an alternative approach.
DPO is an alignment method that does not require an auxiliary reward model, thereby improving training efficiency and stability. DPO training reduces to logistic regression offline preference datasets, following the Maallyy-eerry model [9]. Subsequent studies have proposed extensions of the DPO loss functions [55, 62, 67]. From the perspective of distribution matching. f-PO [27, 67] reformulates the aligmment objective as matching the policy model to the optimal policy defined by DPO, extending the loss using fdiivergence. While this reformulation retains the target optimality, f-PO introduces additional complexity since it relies on learned reward model and requires Monte Carlo estimation of partition functions. Importantly, its optimality cannot be guaranteed without incurring additional computational cost, as summarized in the third and fourth rows of Table 1.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Table I: Summary of probabilistic DPO extensions. O: optimality
preservation, S: simplicity without extra training cost, G: generality for multiple objectives. Refer to Appendix B for baseline details.
Hi Win rite prefened (a)
y 18
SBA
@-div
leffrey
J5
Name
o S G
Loss (((): yo. ~ Pdam
So
LSIF
DRO SEUP
DPQ
q-div
DPO [49]
/ / x
loga
(Alu Folya/x)
log ro(x))))
$
PDPO
snc(Nulx))
FPO
IPF (Duns)
Inga (As
solya ix)
moly)/x))
S-DPO [62]
-B8
23
Diversity (triropy)
[-PO [67]
/ /
Dy mo(ytx)
[(x] Tarly (xxoppppp//ppp
The trade-off between fidelity and diversity across in- stances from different proba- bilistic preference optimization frameworks on dialogue genera- tion with Pythia 2.8B.
Figure 1:
[-PO [67]
x / /
Blog
log
BPO (Ours)
M(RR) ((Ro) N (R.'),
Kolya (x()))
This paper investigates generalized DPO loss that enables policy models to match the target policy without incurring additional computational burden. We show that the optimal policy can be expressed without relying a learned reward model partition functions by adopting a likelihood ratio perspective. The likelihood ratio of the target policy a valid estimation target, as the ratio uniquely identifies the target policy distribution. We reformulate DPO ratio matching problem between the data preference ratio and the model ratio, and extend the loss using Bregman divergence [10, 53], which we refer to Bregman preference optimization (BPO). BPO generalizes DPO by including as a special case. We also propose a scaled Basu's 's power (SBA) divergence, a gradient scaling method for BPO instances. In addition, we show that the proposed loss generalization can be applied in an orthogonal manner to existing DPO variants that can be expressed as logistic regression.
We empirically evaluate several instances of BPO against baselines, measuring both generation fidelity (win rate with GPT-4 judge) and generation diversity. contrast to other probabilistic loss extensions such as f-DPO [62] and f-PO [67]. which show clear trade-off between win rate and diversity, BPO instances improve both win rate and entropy over DPO, as summarized in Figure 1. The proposed SBA instance achieves the best trade-off between win rate and diversity among all instances. We also apply the proposed loss to Jlama-3-8B-Ima --nsttutt [18] and achieve a 55.9% length-controlled win rate against GPT-4 Turbo on AlpacaEval2 [36]. To the best of our knowledge, this is state-of of-the----eerrt performance among Llama-3-88 --BB backbone models.
Preliminaries
2.1 Alignment of large language models with human preference
We consider an autoregressive language model my as policy model that generates a response sequence = ly'.....'' y²1 conditioned on a prompt x, with probability (y(x)) = I Te ( xx, x**), where L denotes the sequence length. Alignment methods commonly begin with pre-trained large language model (LLM), followed by supervised fine-tuning (SFT) on high-quality dataset. The resulting model used as the reference model Tee, and the policy model wo is initialized from Nee-
Reinforcement learning from human feedback (RLHF): The alignment method RLHF [6, 47, 52, 74] consists of two phases following SFT. The first phase involves learning reward model Ta- This phase requires access to an offline pairwise preference dataset, consisting of tuples (x prompt, yw preferred response, y1 dispreferred response) drawn from the underlying preference data distribution Paata (y > yı x Ppromptt (x). The learning objective of the reward model is
Creward
(Toi (T Pdata) = -Bp.noywwwe llog (F (x, yr))],
where denotes the logistic function. The prompt drawn from Pppmmp(()) which we omit in the following formulations for notational simplicity. From the optimality condition of logistic regression, the optimal reward model r&- that minimizes Lyoward satisfies Paaca (y > yılx) (rss (w.) ra- (x, y))). also known as the Bradley-Tene model [9]. The next phase is RL fine-tuning based on proximal policy optimization (PPO) [50], formulated as
CRLHE Thet ) = - E (y/p)) ITo (x, + DKL (To (xxr

---

Table 2: Examples of Bregman divergence instances defined by choices of h, and their corresponding BPO loss functions.
hRR)
Tangent at (i(
Name
N(R)
Cavol Ro; peat)
(RolliBa)))))
LR (DPO)
Rog R (1. Rog (1+
Epine [log (Ro + 1)]
KLIEP
R log R R
Ex [Ro + log Ro]
4 (Ro))RRa RR))
LSIF
(R - 1)a [RI+A-R A
Epsnl [R [
BA
Bpnn [Rg A+1 R5
Ng- Figure 2: Point-wise Bregman di- vergence defined by h, between the model Rg and the target Rg..
Hg
SBA (Ours)
(R++A 4A(4+1)
Epena
a(+1)) R2tt1
23 R2,"
PRLHF consists of reward maximization term and B-weighted reverse KL regularization term that penalizes deviation of the policy from the reference model. Due the complexity and instability of this multi-stage pipeline, direct preference optimization has become a strong alternative for alignment.
Direct preference optimization (DPO): The alignment method DPO [49] unifies the objectives Eqs. (1) and (2) into a single objective without the reward model. Eq. (2) has a closed-form solution:
1 (5 (x.x))))))
TT- (ylx) = arg min PRLHE
(x) Ty (x)xpp
where 7(x) = Terl (x(xxx)ex (x, y)) the partition function. From Eq. (3), DPO defines
new reward model (x, = log xo (x/() ] + B log (xx). which is parameterized by the policy
and reference models. Substituting TDPO into Lreward ( (TDro: Pdata) yields the following objective:
T xx) 3 log
] Tx)(()
To (y(
CDPO Tref Pdata) Ee log
Bog
which enables the policy model to learn directly from a preference dataset in a supervised manner. The optimal policy wa- that minimizes LPPO matches the optimal solution in Eq. (3), and it satisfies:
Pana I > vrlx) = (8log NT (Ywlx) Blog
T8- (yr(x)
(5)
(y.
Tret (y:
Despite the theoretical guarantees, optimality is often not achieved in practice because of limited model capacity and imperfect optimization. Since the practical solution depends on the form of the objective function, generalizing the objective provides potential benefits.
2.2 Likelihood ratio estimation under Bregman divergence
Given two probability distributions Pa (x) and Pmu (x). likelihood ratio estimation aims to learn (x)
a ratio model Rx(x) that approximates Rama (x) = (x) based on i.id. samples from both distributions. Probabilistic classification via logistic regression [21] is the most commonly used approach. Traditional methods such the Dollback-Leible importance estimation procedure (KLIEP) [45, 54] and least-squares importance fitting (LSIF) [25, 20, 29] have also been widely used. These methods can be unified under the Bregman divergence [10] framework in [53], resulting the following formulation:
Da RRssul (R)ss(())| (o()) = Paie (x)) Bb, (Rataaa Ro)
= -f pxe(x) (h(Raaa (x)) to ((o 11' (x)
Ro Ro(x))) dx, (6)
where h denotes a strictly convex and twice continuously differentiable function with derivative function h' and B,, is the pointwise Bregman divergence which measures the error of the linear approximation as illustrated in Figure 2. Specific instances are summarized in Table 2, showing that previous likelihood ratio estimation methods differ only by the choice h. Among various instances, Basu's power (BA) divergence [7], defined for > > -1, smoothly interpolates between KLIEP (at = 0) and LSF (at À = 1). Inspired by the success of generalized likelihood ratio estimation in recent generative modeling studies [38, 30, 32], apply this extension to generalize the DPO loss.

---

3 Methods
This section introduces the proposed Bregman Preference Optimization (BPO). Section 3.1 refor- mulates the DPO objective as a likelihood ratio estimation problem. Section 3.2 extends the DPO objective via Bregman divergence under the likelihood ratio estimation perspective. Section 3.3 analyzes instances of BPO and introduces the scaled Basu's 's power (SBA) divergence within this framework. Finally, Section 3.4 discusses the applicability of BPO to other DPO variants in an orthogonal manner.
3.1 Preference optimization as likelihood ratio estimation
In contrast to prior works [27, 67] that characterize the optimal policy using a learned reward model Te- and a partition function (x) as in Eq. (3), we aim to characterize the optimal policy without such modeling complexity.
Proposition 1. Let the optimal policy TT- := arg min CDPO(TT;; Trefs Pdata), the following holds:
TT- (y x))
(Yul) TTg (yr(x) (xi()) Pdiaaa (Yw yı(x)
Pedat (yw > v1(x) 1/3
(7)
Please refer Appendix A. for the proof. Eq. (7) shows that the likelihood ratio of optimal policy Tg- can be specified solely using the reference model Tee and the preference data distribution Pdat- The ratio ze* (ywl)) is equivalent to the concrete score [41, 38] (up to constant), and the concrete To- [y] score satisfies the completeness property [39, 41]. This means that the concrete
can uniquely identify the distribution πT* Therefore, the matching Taly x)) ra- (y.ll))) is sufficient for my to Wy- (Y:
moly; x) to
recover TØ-, providing a theoretical justification for ratio matching. To build a data-driven estimator for this matching, rearrange Eq. (7), detailed in Eq. (15) Eq. (19), and reformulate preference optimization as a matching problem between data ratio R and model ratio Rg, defined as:
Raata Xx, Y y1) Pdata (Yw y1(x]
Te (er 1x
Ro(x, Yw:
(8)
Pdata (Yw > y))x)
me m
3.2 Bregman preference optimization via ratio matching
As discussed in Section 3.1, preference optimization can be formulated as ratio matching between Rasaa and Rg. We define the loss using the Bregman ratio matching framework Section 2.2 as:
Dí Rataall [Ra) (y xxl))) [/ Rama) h(Ro)) 1t' N((Ro) (Ranta Ro)]
where Raata and Ro denote the ratios evaluated at (x, Vi) on the right-hand side, h denotes strictly convex and twice continuously differentiable function with derivative function h'.
Theorem 2. (Optimality) Under sufficient model capacity, arg min,, Da RRa = TT--
Please see Appendix A.2 for the proof. Theorem implies that the objective D (Raaal Re) guar- antees target optimality under valid h, but is intractable since evaluating Raasa a given point (x, yw, Yr) is infeasible. As preference optimization provides only samples from Paam (Y > yt x) without direct access to the distribution, we propose tractable alternative inspired by implicit score matching [26]:
C (Ro;Páan)) := Ppanayy=yy (x) [M' Ro (((R)) - (RT
(10)
and the equivalence is guaranteed by the following theorem.
Theorem 3. Chpo (Re; Pdota) = Di. (Rassel (Ro) + C, where C is constant with respect to
The proof is provided in Appendix A.3. The intractable term Rasa does not appear C and the statistical information of Retaa is captured entirely via samples from Paat- As the value of function Ra at the point (x, Yw. yt) computed through the feed-forward passes of na and Taes Rg is appropriate to retain it inside the expectation. To analyze the learning dynamies of C Ro; Pama), we provide the following gradient analysis:

---

Gradent CAEAA.
DP0
A= 0G
Gradien magnitude Cr (R,)
DPO (UR)
30 (R (R
12 GGin(RijiV
00
A- (KLIP)
SBA (1-0)
(RR))
A= 0.6
30
A (LSIF)
A
V. 12
BA (1-0)
20
BA (1-00
10
Vonth
0.25
1.30
1.75
0.28
0.50
0.75
1.00
1.75
2:0
Rg
Value
R2
0.8
R.
0.5
(a) Ca. BA(A) divergence.
(b) C₁. SBA(A) divergence.
(c) Gradient updates.
Figure 3: Gradient magnitude and direction analysis across different Bregman divergences.
Proposition 4. (Gradient Analysis) Let the gradient of the BPO objective be expressed as:
V copo Ra: Pditu) = Exsnoty uy y:(x) [Gn RoR),
(11)
where define G (Ra) the magnitude of gradient. If is strictly convex and twice continuously differentiable, then Gv (Rg) > 0 for all Rg.
See Appendix A.4 1.4 for the proof. Eq. (11) shows that the gradient direction point (x, Yw Vi) depends only on Rg regardless of the choice of h. In contrast, h controls the point-wise gradient magnitude Gr. (Re). which determines the relative weighting of each sample during optimization. The weighting can influence the gradient direction aggregated over a mini-batch as shown in Figure 3c. Proposition further explains that gradient descent updates decrease the value Ro at the observed point (x, yw: yt) Pama- This is intuitive since the observed pair is sampled from the distribution
in the denominator of the target ratio Raina (y <y: x)
PPmnly.>> > x) Figures 3a and 3b show that Gj, (Rg)
remains positive across various choices h. Although Theorem shows that the theoretical optimality holds regardless of the choice of the function h. the gradient magnitude Gj. (Ra) varies with h, making the choice of h important due to potential sub-optimality in practical optimization.
3.3 Instances of Bregman preference optimization
Table summarizes well-known instances of Bregman divergences unified in [53] and their corre- sponding BPO objectives Chro determined by the choice of h.
BPO recovers DPO as special case: The proposed L.PO recovers the original DPO objective LDoo when & corresponds to the logistic regression, denoted as Capo (See Appendix A.5 for more details):
C Ro: Pdma)
E "y [log (1 Rb)]
(y x / (y [x) 3) I
log 1
T | (xe(
(yo (x) Blog To To (y)1l))
)
P log
Blog
LDPO Weef- Pdatu)
(v
Teat Tee Ty(((y))))
KLIEP and LSIF under Basu's power divergence (BA): The instances,
CKLIEP BPO
and
LSSI BPO are
unified under CBAy where the gradient magnitude is given by:
GBA, (Re) ) = (A + ()) + RTA-1),
(12)
as shown in Figure 3a for various values of X. While the term (R) + RR provides meaningful control the optimization behavior with respect to Rs via the hyperparameter X, the coefficient (X+ + 1) unnecessarily scales up the gradient magnitude without introducing any 6-dependent learning signal. Compared to the gradient magnitude of DPO, given by GLR(Ro) I+ the gradient magnitude of (Ro) increases significantly as À becomes larger. When the gradient scale varies, training requires careful adjustment of the learning rate, batch size, and optimizer. Since preference optimization sensitive to hyperparameters, improper tuning can severely degrade performance. To mitigate this issue, propose simple scaled version of BA to match DPO's gradient scale.
5

---

Scaled Basu's power divergence (SBA): We propose h(R) = -'-AaR R
s which corresponds to
scaling the h-function of BA by factor of s(À + 1), where is scaling constant. This scaling directly affects the gradient magnitude as follows:
GSBA, (Ra) = (R + R 1//
(13)
GSBA (Re) eliminates the unnecessary X-dependent amplification. resulting in a more reasonable gradient scale Figure 3b. Since preference optimization typically initializes we as Tret the value of Ro is 1 for all input points (x, Yx. yi) at the start of training. By setting
4, the gradient magnitude of GSBA, matches that of GLr initialization (Rg = 1). The hyperparameter À allows the model to control whether to prioritize updates for more confident (e.g., Rg < 1) or less confident (e.g., Ra R 1) samples. Recent studies [64, 31] have shown that DPO is sensitive to data quality and that applying confidence-based adjustments to individual samples can be heuristically effective. SBA controls the sensitivity to confident samples by tuning X, while preserving theoretical optimality.
3.4 Compatibility with other DPO extensions
We have discussed how the optimal policy defined by DPO can be approximated by the policy model. Our generalization can also be applied orthogonally DPO loss variants that define different optimal policies. We consider f-DPO [62] as an example by defining model ratio as:
R2 P (x. Yw: vi) = exp -38 To (y
(y(y) [rer(yi
)
338
(14)
Teel (y
where / + denotes a convex function satisfying f(1) O. Substituting R2 Rf--rO and the
function corresponding to logistic regression into BPO; CHPO (12g (I2.-DPO, :Pama) recovers the original
objective of f-DPO [62] (See Appendix A.6 for details). BPO enables constructing new objectives by varying the choice of based on the model ratio Rg RI which results in Ciro ( DPO, Pdmma.
Similarly, existing logistic regression-based DPO variants can also be generalized by BPO.
4 Experiments
This section presents the empirical performance of the proposed BPO compared to prior preference optimization methods. Section 4.1 compares instances of BPO with other probabilistic loss extensions. Section 4.2 compares BPO to the state-of-the-ard DPO loss variants on popular LLM benchmarks.
4.1 Comparison with probabilistic DPO loss extensions
We compare BPO with baseline methods for probabilistic DPO loss extensions, and analyze the effect of different choices of the function h within the BPO framework.
Task & experimental setup: The experiments are conducted for single-turn dialogue generation using the Anthropic helpful and harmless (HH) dataset [6], and summarization using the Reddit TL;DR dataset [59]. While the experimental setups vary across the different baseline papers, we faithfully follow the setup provided in the original DPO paper [49] for fair comparison. For dialogue generation, we use Pythia-2.88 [8] as the pre-trained LLM and perform SFT on the preferred subset of the HH dataset. For summarization, All comparisons are conducted on the same SFT model with identical training hyperparameters (,, learning rate, batch size). See Appendix C for details.
use publicly available SFT model [11] based on GPT-J [61].
Baselines: We compare with representative baselines, including instances of f-DPO [62] and f- PO [67]. For f-DPO, consider forward KL (FKL), Jensen- -hannon (JS), a-divergence with a = 0.1, 0.3, 0.5, and 0.7, selecting a = 0.1 based on the best win rate, and additionally include -dvieerge [24]. For f-PO, we consider Jeffrey, JS, reverse KL (RKL) [27], and a-divergence with a = 0.1, as suggested in the original paper. To ensure a fair computational comparison, we adopt the pairwise approximation proposed in f-PO.
Evaluation metrics: Both fidelity and diversity of the generated samples are important from a probability matching perspective. We evaluated the models using test datasets. To measure fidelity, we use the win rate against both the preferred responses and the SFT model responses, as judged by GPT-4. To assess diversity, we use predictive entropy [70, 62], self-BLEU [73], and distinct-1 [34].

---

Table 3: Comparison of dialogue generation performance on Anthropic-HH, using the same SFT model based on Pythia-2.8B. Each value is colored red if it better than DPO, and blue if it performs worse. The best result for each metric is shown in bold, and the second and third best are underlined.
Win rate (%) f
Diversity
Type
Loss
Preferred
SFT
Entropy
BLEU 4
Distinct-1 t
SFT
33.5
3.508
0.096
0.375
All
DPO
48.5
71.5
2.801
0.145
0.336
f-DPO [62]
FKL
34.5 34.5-28.999 28.99
55.5 22.4%
3.354
19.7%
0.088
+9.33%
0.338+0.7%%
a-div.
47.5 -.11%
64.0 10.5%
3.125
11.6%
0.123
14.9%
0.332 1.22%
JS
32.0+7..2%
70.0 2.19%
2.723
2.88
0,139 83.8%
0.299
10.9%
x² [24]
53.5 10.3%
72.0++077%
2.369
15.4%
0.134 +7.3%
0.279
16.8%
[-70
Jeffrey
51.5 +..2%
68.0 68.0- 4.99%
2.436
13.0%
0.166 14.7%
0.271
19.1%
JSS
54.5 +12.4%
76.0+6.3%
2.531
9..6%
0. 0.154 1.154-6.1% 6.1%
0.283
15.8%
a-div.
52.0 +7.2%
75.5 +5.6%
2.298
18.0%
0.168 15.9%
0.290
13.6%
RKL [27]
49.5 +...+...19
70.5 1.4%
2.321
17.1%
0.182 25.8%
0.268
20.2%
BA
+5.0%
75.5 +5.6%
2.803 +0.1%
0,141
+2.6%
0.320
4.5%
BPO
KELIP
48.5+0.000
71.5 +0.0%
2.901 +3.6%
0.151 1 4..1%
0.332
1.0%
LSIF
50.5 +0.54
72.5 + +1.4%
2.908 +3.8%
0.139 +4.11%
0.321
4.5%
SBA
57.0
77.0 +7.7%
3.010+7.3%
0.132
+9.0%
0.340 + 11.4%
Table The performance of Chro J DPO -Dama); which extends
Fit Vio nan in aroe (iu
f-DPO [62] orthogonally using our BPO framework.
+0PO
Win rate (
Diversity
M(-)
vs Preferred
vs SFT
Entropy t
BLEU
Distinet-I I f
LR
34.5
55.5
3.354
0.088
0.338
FKL
SBA
39.5
60.0
3.237
0.081
0.340
TLDPS (PR) -DPG
Rabio
JS
LR SBA
52.0
70.0
2.723
0.139
0.299
Figure 4: Orthogonal improve-
55.0
73.5
2.856
0.127
0.307
ments with BPO (SBA).
4.1.1 Dialogue generation task
Comparison to baselines: Figure shows the trade-off between win rate and entropy for each instance, and Table 3 summarizes the results across all metries. f-DPO recovers DPO when RKL is used as the regularizer. As the a-diivereence shifts from RKL to FKL,, the win rate decreases while diversity improves, reflecting the mode-covering nature of FKI. x² improves win rate, observed in x-PPO [25]. f-PO recovers DPO with FKL. Other divergences, such as Jeffrey, JS, a-divergence, and RKL, have weaker mode coverage than FKL. resulting worse diversity across all metrics.
All BPO instances achieve win rates at least as high as those of DPO against both preferred responses and SFT responses, while also achieving better entropy. f-DPO and f-PO exhibit substantial degradations in some cases, such as a 28, 9% win rate drop in f-DPO (FKL) and a 25.8% BLEU drop in f-PO (RKL). In contrast, instances of BPO show no degradation greater than 5% on any metric. Notably, SBA is the only instance that consistently outperforms DPO on all metrics, achieving the highest win rate compared all other losses.
Orthogonal utilization with other methods: As discussed in Section 3.4, existing loss functions that can be expressed logistic regression can be orthogonally extended using BPO. Table 4 reports the performance of BPO applied to two instances of fDPO (FKL and JS), showing improvements in 9 out of 10 metrics. As shown in Appendix B.2, f-PO with pairwise datasets can also be viewed as logistic regression for some f. and applying SBA to f-PO (JS) results in further performance gains. Figure 4 summarizes the performance improvements achieved by applying SBA to each model ratio.
Ablation studies on À in SBA: Figure shows ablation studies on the effect of the proposed SBA, an instance of the BPO framework. When X = -0.5, the gradient behavior closely resembles that of DPO, as shown in Figure 3b, with both win rate and entropy showing similar trends in Figures 5a and 5b. As X increases, both metrics improve up to certain point, in contrast to the
7

---

32
0.5
(UR
HN Win rate pref.) (9)
3.0
50
Entropy
2.8
Deestty
85
40
5BA (1)
2.6
58A (bess win ate)
58A (A)
25
SBA (best wirrate)
DPD
2.4
EGPG (best winrataa
DPO
190 (oest W mrstte
011
30
0.5
0.0
05
1.0
1.5
2.0
3.0
0.5
0.0
05
100
1.5
2.0
2.5
3.0
0.3
22
Reward margin value: -kg
(a) Win rate (vs preferred).
(b) Entropy.
(c) Reward margin statistics.
Figure Ablation studies on the effect of À in SBA, using Anthropic HH with Pythia-
Table 5: Performance TL:DR summariza-
DPO
Van lla
tion based on GPT-J with 0.25 temperature.
SPO
Win rate 1
Diversity
AlpacaEvel LC 86
Type
Preferred
vs
BLEU
Enrropy 1
DPO
63.0
0.597
0.276
T
J-DPO
46.0
60.5
0.596
0.258
70.5
0.2
DPQ
SmIP
BPO
61.0
71.0
0.565
0,318
Kerperature
Model Ratio
Figure 6: TL:DR ablation.
Figure Mistral-7B.
instances of f-PO and f-DPO that exhibit clear trade-off between win rate and entropy. Larger values of À lead the model to focus more on confident samples (where Rg deviates further from 1), resulting in wider spread in reward margin (- log Ro) statistics, as shown in Figure 5c. This shift in the statistics of reward margins appears to an underlying factor contributing to the performance improvements.
4.1.2 Summarization task
For the TL:DR summarization task, we compare the best win rate instances of each framework reported in Table 3, where f-DPO uses x². f-PO uses JS, and BPO uses SBA. Figure 6 presents the win rate against preferred responses across different sampling temperatures, showing a trend consistent with prior studies that lower temperatures lead to better performance. BPO consistently outperforms the other instances at all temperatures. Table 5 shows additional metrics at temperature 0.25, where BPO shows the best performance across all metrics.
4.2 Comparison with general DPO loss variants
This section extends the analysis to larger model and evaluates its performance on external bench- marks beyond the training domain, comparing with general DPO variants.
Experimental setup: We conduct experiments using Mistral- Misttal-7B--aa B-Base [28], Llama-3-8B-Ban and Alama-3-8B--l 8-I-Istuuutt [18] backbone models, based on the UltraFeedback dataset [15]. For Mistral- 7B-Base, we use publicly available SFT models from Zephyr [58, 57]. We extend DPO with our BPO loss based on the Zephyr setup and extend SimPO with BPO loss following the SimPO [43] setup. For Llama-3-8B--Ban
and JJama-3-8B-IIstruus we use the same SFT models [19] as in the SimPO setup, along with the adapted version of the UlrraFeedback dataset [42]. See Appendix for details.
Evaluation benchmark: We evaluate the response capability across a wide range of queries using the most popular open-ended instruction-follee following benchmarks. AlpacaEval2 [36] measures the win rate on 805 examples, using GPT-4 Turbo as both the judge and the opponent. The evaluation covers both raw and length-controlled responses. Arena-Hard [35] measures the win rate on 500 queries, using GPT-4 Turbo as the judge and GPT-4-0314 as the opponent. All evaluations follow the library configurations and decoding parameters used in SimPO to ensure fair comparisons.
Results: Table presents the main results obtained with various backbone models. Baseline results are taken from SimPO [43]. Most DPO variants exhibit even worse performance than the standard DPO. BPO can also be applied on top of SimPO, as discussed in Section 3.4, and it achieves consistent performance gains over SimPO, except out of nine cases. To the best of our knowledge, BPO

---

Table 6: Performance on AlpacaEval2 and Arena-Hard with various backbone models. LC (%)
indicates the length-controleed win rate, and WR (%) denotes the raw win rate. The baseline results from SimPO [43].
Mistral-7B-Ban
Llama-3-8B-Ban
Llama-3-8B-Insruuc
AlpacaEval
Arena-Hard
AlpacaEval
Arena-Hard
AlpacaEval
Arena-Hard
Method
L.C
WR
WR
L.C
WR
WR
L.C
WR
WR
SFT
6.2
1.3
6.2
4.6
3.3
26.0
25.3
22.3
RRII [69]
11.6
10.2
5.8
12.1
10.1
6.3
37.9
31.6
28.8
SLiC-HF [72]
10,9
8.9
7.3
12.3
13.7
6.0
33.9
32.5
29.3
DPO [49]
15.1
12.5
10.4
18.2
15.5
15.9
48.2
47.5
35.2
IPO [5]
11.8
9.4
7.5
14.4
14.2
17.8
46.8
42.4
36,6
CPO [66]
9.8
8.9
6.9
10.8
8.1
5.8
34.1
36.4
30.9
KTO [16]
13.1
9.1
5.6
14.2
12.4
12.5
34.1
32.1
27.3
ORPO [23]
12.2
7.0
10.6
10.8
38.1
33.8
28.2
R-DPO [48]
17.4
12.8
8.0
17.6
14.4
17.2
48.0
45.8
35.1
SimPO [43]
21.5
20.8
16.6
22.0
20.3
23.4
53.7
47.5
36.5
BPO
23.7
20.9
16.9
22.5
18.7
31.7
55.9
51.5
38.0
achieves state-of-the-art L.C performance among preference-optimized models based on Llama-3- 8B SB-Instrect. Figure 7 shows results with Mistral-7B- Battral-B-Baase
Base. Specifically, BPO improves LC from 15.1% 19.1% 1% when built on DPO, and from 21.5% to 23.7% when built on SimPO. These results demonstrate that BPO generalizes well to external benchmarks beyond the training domain with 7-8B LLM scale, showing consistent improvements across different backbones and model ratio formulations, including DPO and SimPO.
Related work
A series of studies have proposed variants of the DPO loss. SLiC [72] and IPO [5] replace the logistic regression DPO with hinge and squared losses, respectively, while GPO [55] unifies these variants within binary classification framework. KTO [16] generalizes binary preference comparisons to multiwise settings. SPPO [65] formulates preference optimization as a two-player game between policies, framing the optimal policy Nash equilibrium and offering game-theoretic interpretation of DPO. SimPO [43] and ORPO [23] proposed reference-free preference optimization, enabling more efficient and coffaie-compatible implementations. TDPO [70] applies the preference loss the token level, enabling fine-grained reward assignment and improving alignment with human preferences. 3-DPO [64] dynamically adjusts the regularization parameter 33 based on the reward margin between preference pairs, improving training stability and reducing sensitivity. While these studies explored complementary directions improve the flexibility, efficiency, and robustness of preference optimization, they offer limited probabilistic interpretation and rely primarily on heuristics.
DPO loss generalization in distribution matching perspective: f-DPO [62] generalizes the reverse KL regularization in PRIIE (see Eq. (2)) to the broader class of f-divergences [14, 37], and deriving direct optimization objectives for subset of f-divergences that satisfy specific conditions. x². PO [24] extends the direct optimization objective for x" divergence regularization within the f-DPO framework. MPO [2] generalizes the regularization under the Bregman divergence, particularly for meta-learning. However, modifying the regularization alone is potentially insufficient to determine the overall behavior of the loss function and naturally leads discussed in Appendix B.I. EXO [27] reformulates the entire DPO objective as distribution matching problem, showing that CDPO in Eq. (4) reduces the forward KL divergence between the policy model Ta (y)x) and the optimal policy To- (ylx) ) in Eq. (3). EXO further proposes reverse KL loss, and f-PO [67] subsequently extends this formulation to f-divergences. In contrast f-PPO, which yields different optimal policy, f-PO is analogous to extensions of traditional generative modeling [46, 51] that directly match a policy to a target optimal policy. However, the f-PO loss requires learned reward model re- and Monte Carlo estimation of partition functions to compute the probability of the optimal policy æg- Simplifying f-PO for paired preference dataset results in different notion of target optimality
different optimal policy, as
discussed in Appendix B.2. As summarized in Table 1,

---

the proposed BPO is distinguished from existing probabilistic loss extensions by maintaining both optimality and simplicity, while allowing flexible choices of optimization objectives.
Generative modeling and likelihood ratio estimation: The likelihood ratio has played a central role in research on generative modeling. For continuous random variables, noise contrastive estimation (NCE) [21] estimates the ratio between a known noise distribution and the target distribution to approximate the target density. This ratio identifies the target density and also serves as a learning signal for neural samplers, inspiring subsequent work on GANs [17]. The noise distribution in NCE has been replaced by the model distributions of GANs [4, 12], VAEs [40, 3], and diffusion models [30, 44], and has recently been employed as a refinement technique in generative modeling. For discrete variables, concrete score matching (CSM) [41] estimates the ratio probability masses between different states of a probability distribution. CSM has served as a foundation for the development of discrete diffusion modeling [38, 71], and this paper adopts the concept of CSM to autoregressive language models. Another concurrent work has utilized CSM for knowledge distillation in autoregressive language models [33].
Conclusion
We have introduced Bregman preference optimization (BPO), a generalized preference optimization objective based on Bregman divergence, formulated from the perspective of likelihood ratio estimation. BPO uniquely retains both optimality and simplicity among existing probabilistic loss extensions. Our gradient analysis further reveals that the optimization behavior depends on the choice of h, which determines the prioritization of samples, and we propose the gradient scaling method for BPO instances to facilitate training. Experimental results show that BPO consistently outperforms existing DPO variants across various scenarios, while being orthogonally applicable other DPO variants.
In addition, recent advances in discrete diffusion models benefit from the likelihood ratio perspec- tive [38]. We demonstrate that similar extension is also effective for preference optimization using autoregressive language models. Although the types of probabilistic models and the datasets differ (i.e., unconditional generation preference optimization), our results suggest a potential link through a shared probabilistic foundation toward a unified understanding of generative models.
Limitations and broader impact Despite promising experimental results autoregressive language models, extending BPO to preference optimization multi-modal LLMs [63] or diffusion models [60] remains open direction for future work. As with other LLM research, advances in LLMs may improve user assistance in various tasks, but they also pose potential concerns about ethics and bias.
Acknowledgment
This work was supported by the IITP (Institute of Information & Communications Technology Planning & Evaluation)-ITRC (Information Technology Research Center) grant funded by the Korea government (Ministry of Science and ICT) 2---0022-----22--044777700
10

---

References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidr, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv22333.08774, 2023.
[2] Carlo Alfano, Silvia Sapora, Jakob Nicolaus Foerster, Patrick Rebeschini, and Yee Whye Teh. Meta-learning objectives for preference optimization. arXiv preprint arXiv:2411.06556 2024.
[3] Jyoti Aneja, Alex Schwing, Jan Kautz, and Arash Vahdat. A contrastive learning approach for training variational autoencoder priors. Advances in neural information processing systems, 34:480-493, 2021.
[4] Samaneh Azadi, Catherine Olsson, Trevor Darrell, lan Goodfellow, and Augustus Odena. Discriminator rejection sampling. In International Conference Learning Representations, 2019.
[5] Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. International Conference on Artificial Intelligence and Statistics, pages 4447-4455. PMLR, 2024.
[6] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint 2rX5v.200.00862, 2022.
[7] Ayanendranath Basu, Ian R Harris, Nils L Hjort, and MC Jones. Robust and efficient estimation by minimising a density power divergence. Biometrika, 85(3):549-555 1998.
[8] Stella Biderman, Hailey Schoelkopf. Quentin Gregory Anthony, Herbie Bradley, Kyle O' O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages
PMLR, 2023.
[9] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: the method of paired comparisons. Biometrika, 39(3/4):344-44, 1952.
[10] Lev M Bregman. The relaxation method of finding the common point of convex sets and its application to the solution problems convex programming. USSR computational mathematics and mathemutical physics, 7(3):200-277, 1967.
[11] CarperAl. openai-summmrrize-td--stttt summarize_tldrz wtiizee_ddr_eeeee 2023.
[12] Tong Che, Ruixiang Zhang, Jascha Sohl-Dickstein, Hugo Larochelle, Liam Paull, Yuan Cao, and Yoshua Bengio. Your gan is secretly an energy-based model and you should use discriminator driven latent sampling. Advances in Neural Information Processing Systems, 33:12275-12287. 2020.
[13] Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[14] Imre Csiszár, Paul C Shields, al. Information theory and statistics: A tutorial. Foundations and Trends@ in Communications and Information Theory, 1(4):417-528, 2004.
[15] Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun. ULTRAFEEDBACK Boosting language models with scaled AI feedback. In Forty-first International Conference on Machine Learning, 2024.

---

[16] Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Model alignment as prospect theoretic optimization. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceed- ings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 12634-12633 12651. PMLR, 21-27 Jul 2024.
[17] Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[18] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21 7733, 2024.
[19] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ah- mad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. -lam-- --8B-Instrutt. 2024.
1lama/Meta- L1 ama- 3- 8-BB-BB Ins Instruct,
[20] Arthur Gretton, Alex Smola, Jiayuan Huang, Marcel Schmittfull, Karsten Borgwardt, Bernhard Schölkopf. et al. Covariate shift by kernel mean matching. Dataset shift in machine learning. 34):5, 2009.
[21] Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. The journal of machine learning research, 13(1):307-361 2012. 2012.
[22] Rei Higuchi and Taiji Suzuki. Direct density ratio optimization: A statistically consistent approach aligning large language models. arXiv preprint arXiv:2505.0755: 075555 2025.
[23] Jiwoo Hong, Noah Lee, and James Thome. Orpo: Monolithic preference optimization without reference model. arXiv preprint arXiv: 2024.
[24] Audrey Huang. Wenhao Zhan, Tengyang Xie, Jason D. Lee, Wen Sun, Akshay Krishnamurthy, and Dylan J Foster. Correcting the mythos of KL-regularization: Direct aligmment without overoptimization via chi-squared preference optimization. In The Thirteenth International Conference on Learning Representations, 2025.
[25] Jiayuan Huang, Arthur Gretton, Karsten Borgwardt, Bernhard Schölkopf, and Alex Smola. Correcting sample selection bias by unlabeled data. Advances in neural information processing systems, 19, 2006.
[26] Aapo Hyvärinen and Peter Dayan. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
[27] Haozhe Ji, Cheng Lu, Yilin Niu, Pei Ke, Hongning Wang, Jun Zhu, Jie Tang, and Minlie Huang. Towards efficient exact optimization of language model alignment. In Forty-first International Conference on Machine Learning, 2024.
[28] Albert Q. Jiang, Alexandre Sablayrolles. Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.
[29] Takafumi Kanamori, Shohei Hido, and Masashi Sugiyama. A least-squares approach to direct importance estimation. The Journal of Machine Learning Research, 10:1391-1445, 2009.
[30] Dongjun Kim, Yeongmin Kim, Se Jung Kwon, Wanmo Kang, and Il-Chul Moon. Refining generative process with discriminator guidance in score- score-based diffusion models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 16567-16598. PMIRR 23-29 Jul 2023.
12

---

[31] Dongyoung Kim, Kimin Lee, Jinwoo Shin, and Jachyung Kim. Spread preference annota- tion: Direct preference judgment for efficient Ilm alignment. In The Thirteenth International Conference on Learning Representations, 2025.
[32] Yeongmin Kim, Byeonghu Na. Minsang Park, JoonHo Jang, Dongjun Kim, Wanmo Kang, and Il-Chul Moon. Training unbiased diffusion models from biased dataset. In The Twelfth International Conference on Learning Representations, 2024.
[33] Yeongmin Kim, Donghyeok Shin, Mina Kang, Byeonghu Na, and Il-Chul Moon. Distillation of large language models via concrete score matching. arXiv preprint arXiv:2509.2583/ 2025.
[34] Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and William B Dolan. A diversity- promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics:
Human Language Technologies, pages 110-119, 2016.
[35] Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph Gonzalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406. 1939, 2024.
[36] Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-followinng models. https: //g gi thub. com/tatsu 1ab/ alpaca_eval, 5 2023.
[37] Friedrich Liese and Igor Vajda. On divergences and informations in statistics and information theory. IEEE Transactions on Information Theory, 52(10))4394-4412, 2006.
[38] Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. In Forty-first International Conference on Machine Learning, 2024.
[39] Siwei Lyu. Interpretation and generalization of score matching. arXiv preprint arXiv: m2t.. 2629, 2012.
[40] Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adver- sarial autoencoders. arXiv preprint arXiv:1511.05644 2015.
[41] Chenlin Meng, Kristy Choi, Jiaming Song, and Stefano Ermon. Concrete score matching: Generalized score matching for discrete data. Advances in Neural Information Processing Systems, 35:34532-34545 2022.
[42] Yu Meng, Mengzhou Xia, and Danqi Chen. llama--uttrafccdaac-ammoo https:/ / huggingface.
http://tasett/prncttoooooooeeeee catasess seagginggaac..co/ttttsttee
http://aama-uutrafeeeeeaaaa wwwp:rafeeeeebaccccaaaao.aoommm. 2024.
[43] Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-free reward. Advances in Neural Information Processing Systems, 37:124198-122235, 2024.
[44] Byeonghu Na, Yeongmin Kim, Minsang Park, Donghyeok Shin, Wanmo Kang, and Il-Chul Moon. Diffusion rejection sampling. In Proceedings of the 41st International Conference on Machine Learning, pages 37097-37121, 2024.
[45] XuanLong Nguyen, Martin J Wainwright, and Michael Jordan. Estimating divergence func- tionals and the likelihood ratio by penalized convex risk minimization. In J. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems, vol- ume 20. Curran Associates, Inc., 2007.
[46] Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan:
Training generative neural samplers using variational divergence minimization. Advances in neural information processing systems, 29, 2016.
[47] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances neural information processing systems, 35:27730-27744, 2022.
13

---

[48] Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization. In Findings of the Association for Computational Linguistics ACI. 2024, pages 4998-5017 2024.
[49] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728- -3744,, 2023.
[50] John Schulman, Filipp Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint
2017.
[51] Yang Song and Diederik Kingma. How to train your energy-based models. arXiv preprint arXiv:2101.0222 2021.
[52] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize with human feedback. Advances in neural information processing systems, 13:3008-3021. 2020.
[53] Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Density-ratio matching under the bregman divergence: a unified framework of density-ratio estimation. Annals of the Institute of Statistical Mathematics, 64: 1009-1044, 2012.
[54] Masashi Sugiyama, Taiji Suzuki, Shinichi Nakajima, Hisashi Kashima, Paul Von Bünau, and Motoaki Kawanabe. Direct importance estimation for covariate shift adaptation. Annals of the Institute of Statistical Mathematics, 60:699-7445 2008
[55] Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Remi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Avila Pires, and Bilal Piot. Generalized preference optimization: A unified approach to offline alignment. In Forty-first International Conference on Machine Learning, 2024.
[56] Gemini Team, Rohan Anil, Sehastian Borgeaud, Jean- SannBBapiiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, al. Gemini: family of highly capable multimodal models. arXiv preprint S.dV::23/2.//805. 2023.
[57] Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro Von Werra, Clémentine Fourrier, Nathan Habib, et al. zephyr- 7b-sft-full. https://huggingeef co/ /e//alignmmnn weriinmmentteen haandbokk/zephrr--ooook sft- seftffft full, 2023.
[58] Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro Von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of Im aligmment. arXiv preprint arXiv:2310.16944, 2023.
[59] Michael Völske, Martin Potthast, Shahbaz Syed, and Benno Stein. TI: dr: Mining reddit to learn automatic summarization. In Proceedings of the Workshop on New Frontiers in Summarization, pages 59, 2017.
[60] Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8228 8238, 2024.
[61] Ben Wang and Aran Komatsuzaki, GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https: : //ggthuhttttt www./oiooooolllo//eeeeeeeoe
cransformer-.. jax, jax, May 2021.
[62] Chaoqi Wang, Yibo Jiang, Chenghao Yang. Han Liu, and Yuxin Chen. Beyond reverse KL: Generalizing direct preference optimization with diverse divergence constraints. In The Twelfth International Conference on Learning Representations, 2024.
[63] Fei Wang, Wenxuan Zhou, James Y Huang, Nan Xu, Sheng Zhang, Hoifung Poon, and Muhao Chen. mdpo: Conditional preference optimization for multimodal large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8078 8088- 2024.
14

---

[64] Junkang Wu, Yuexiang Xie, Zhengyi Yang, Jiancan Wu, Jinyang Gao, Bolin Ding, Xiang Wang. and Xiangnan He. 8-dpo: Direct preference optimization with dynamic B. Advances Neural Information Processing Systems, 37:129944 2024.
[65] Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self- play preference optimization for language model alignment. In The Thirteenth International Conference Learning Representations, 2025.
[66] Haoran Xu, Amr Sharaf, Yummo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Ken- ton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of LLM performance in machine translation. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceed- ings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 55204-55224. PMLR, 21-27 Jul 2024.
[67] Minkai Xu, Jiaqi Han, Mingjian Jiang, Yuxuan Song, and Stefano Ermon. f-PO: Generalizing preference optimization with f-divergence minimization. The 28th International Conference on Artificial Intelligence and Statistics, 2025.
[68] Lantao Yu, Yang Song, Jiaming Song, and Stefano Ermon. Training deep energy-based models with f-divergence minimization. In International Conference on Machine Learning, pages 10957-10967. PMLR, 2020.
[69] Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback. Advances in Neural Information Processing Systems, 16935-00-00000 2023.
[70] Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang. Haifeng Zhang, and Jun Wang. Token- level direct preference optimization. In Forty-first International Conference on Machine Learn- ing. 2024.
[71] Ruixiang ZHANG, Shuangfei Zhai, Yizhe Zhang, James Thornton, Zijing Ou, Joshua M. Susskind, and Navdeep Jaitly. Target concrete score matching: A holistic framework for discrete diffusion. In Forty-second International Conference on Machine Learning, 2025.
[72] Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Salch, and Peter J Liu. Slic- hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305. 10425, 2023.
[73] Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. Texygen: A benchmarking platform for text generation models. The 4 41st international ACM SIGIR conference on research & development in information retrieval, pages 1097-1100, 2018.
[74] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv: 1909.0859, 2019.
15

---

A Proofs and derivations
In this section, provide detailed proofs for the theorems presented in the main text. Throughout this paper, we have the following assumption:
Assumption. Py w > yr x))TT x), Fo- (y x) > 0 for all You Y.. Assumption. Function h is strictly convex and twice continuously differentiable.
These non-negative assumptions ensure that target quantities, including likelihood ratios, well-defined. The non non-negative assumption for Pdnta is naturally aligned with the Bradley-Tent Terry model, which utilizes a sigmoid function model pairwise probabilities. Given that this model inherently assumes a non-zero probability for each possible data pair, our assumption is direct adaptation of this framework. We also assume that Nel remains strictly positive across its domain. This is reasonable, as we often construct Yw and y: as being sampled from the reference policy distribution, where each possible outcome has a non-zero likelihood. Finally, the optimal policy тg- generally formulated as reference policy with an exponential reward term, which implies that zœ- is non-zero whenever Tre is non-zero, making this assumption natural extension of the positive support for the reference distribution.
According the definition of Bregman divergence [10], the function h required be strictly convex and continuously differentiable. Since we optimize over this divergence, an additional condition on its second derivative needed. This condition is required in previous works [30] that use Bregman divergence loss. The well-known Bregman divergences listed in Table 2 satisfy the condition.
In the following subsections, provide the proof of each statement.
A.1 Proof of Proposition 1
Proposition 1. Let the optimal policy TT-- == arg ming, CDPO(TTE) Tif Pdata), the the following holds:
TT. (y xx
Ty
Pdiaue (yw > yılx) 1/8
T&- (yt)x) (x)()
Pdiata (Yw + y)(x))
Proof. From the Bradley-Tente model we discussed in Eq. (5), the preference data distribution Pdata w yr(x) be expressed
Pdata (yw yı x) = (1 (1+ exp BBog
Tg- (y: x)
BBoog Ta- (y x)
exp
Tee
x)
Tee 10
x)
Taking the reciprocal of this expression, obtain
Tg- (y: x) Trerly:
To- (y
x)
Pdata ( w yi x) = 1 + exp
Blog
3 log
x)
Teor y
xx)
From this, it follows that Pdata (Y < Y1 x)
Pesta Yver > y1 x)
T (yr
x)
Toa (Y
x)
= exp
3 log
3 Blog
Pdita (Y V1 x)
Pautal (y w > y1
Tiee
(yz
x)
Treel Trer(y m
x)
(15)
Tp- (y:
x) Ther Yuu x)
= exp
3 log
(16)
Frer(y:
x x)T (Y
x)
no- (yz
x) (Yn
x)
(17)
(ye x)TT. (Ye
x)
Raising both sides to the power of 1/3 gives the following:
Pdata (y - y1
x)
1/8
Ta. (y: x)T x)
(18)
Paaaa (ywr > yı x)
Frer(yi x) (ywr
x)
Finally, rearranging this expression gives the desired result:
1/8
T8 (yw
x)
[yw w x)
Pdata (Y > V1 x)
(19)
Tg- (yr x) Trerly: x)
Pdata (Yw. - yr x)
16

---

A.2 Proof of Theorem 2
Theorem 2. (Optimality) Under sufficient model capacity, arg min (Pdatal Ro) TT.
Proof. Let arg min dataal [Ra) T--- We have followings from Eqs. (6) and (9):
Da (Rama(x,
Yw: yr) Yu, Yw: Vi)) = ( BB, (Rdua (x, Yw. Y:) R yt) Yw, y:))]
From the property of Bregman divergence [10], B, (Raua (x. Vw: vi) Rg (x Yw: yi)) achieves its minimum value of if and only if Rdaa (x, Yw. Vi) Rx (x. Yw- Yi) for given point Yx Vi). Under the assumption that the data distribution is fully supported, the minimizer Tg is guaranteed to satisfy Raata Rg for all (x, Yws yr). From the definition of Ritaa and Rg in Eq. (8), we obtain the
following:
Pdata (y - v1(x)
Ta(yi Ter x)] 3
x) Ta (Y u Tg yy
[ x) Tyr x)
Paata > y1(x)
1/8
Pdltaa (vu V1(x)
Th T ym x Ter x)
x)
Pa < y xx(xx
From the definition of x² in Proposition 1, we obtain the following relation, which holds for all points (x, Vt)
Y
x)
wo- (Yw x) Ti. (yr x)
Tg (Yi
x)
Using the completeness property of concrete score
[41], conclude that "g T0-
A.3 Proof of Theorem 3
Theorem 3. Chro Ro: Pdana) = Dja (Raatel (Rg) + C, where C is constant with respect to
Proof. From Eq. (9), obtain:
Da (Raaall RR) = E E x((x)) [h( (d(Ra - h(Ro) R-) - hh'Ro) - Re]
P yx(x)
[h' RR - h(Ro) - d'(Ro) Rdatal + +E Bpse (y > [h( ((Raaa))
x h'(Ro) Ro - h(Ro) - 1 (Ro) Pdatal (Ner <x(
C
Pamaa (y y1 v1(x)
(Ppul(y=>xxxx)) [h' (Ro) Ro - (R)))) E (y (h' (Ro) C
Then, the second term (x y (x) [h' (Ra)] can be expressed follows:
Eponttr lx) [h' (Ro)]
B x)
hh' RR Yw: vi))
Ex x)
hh' yI.
TTerly: xy xx) 3
E x)
h'
(y x)To(Yi x))
Epan (y 2 x)
h'
T (x) (ora xx) (yr xx)r (y x) 1
Epannyy x)
h'
E Trunlywwiiile [N' (Rm2))
(x, Yw. ym.yy)
Substituting this back into the above objective, the resulting objective becomes:
Dr. (Raasal (Rg) -Epuntyw>>ite [Ro) -((o)) N¹' (Rz C.
+ L Páma)) = Di, R
17

---

A.4 Proof of Proposition 4
Proposition 4. (Gradient Analysis) Let the gradient of the BPO objective be expressed as:
V To Ro: Paima) Pasaoly (x)l)s) (Ro(
(11)
where we define G (Rg) as the magnitude of gradient. If h is strictly convex and twice continuously differentiable, then Gn Ro) >> for all Rg-
Proof. From the definition of Cp Ro; Pdam), the gradient magnitude Gs. (Rt) is given by
(Ra) = 1t" (Ra) + R2
(
(20)
Since h is strictly convex, its second derivative h" is positive. In addition, the model ratio Rg is defined over positive values, so G (Ro) remains positive for all Rg.
A.5 BPO recovers DPO as special case
This section shows that BPO includes DPO as a special case as discussed in Section 3.3. Consider h(It) Rog R- R- R) log RI+R) which corresponds to the logistic regression in Table 2, which has
derivative function as h (Rt) log R-log (1++2) and the expression inside the expectation operator of C Pdaa) in Eq. (10) becomes:
h (Raa) Ro h(Ra) h (RT*)
(21)
Rg log Re Ro log (1 + Ra)
Rg log Rs - (1+ Ro) log (1 Ro) log n.
log +
1 Rg
2
=log (1 + Ra)
(22)
And by applying this h to Ci (Ro: Pamaa result in LDro To: Tret' Pdata) as follows:
CLR Ro: Pdata) = Eu w -i
[log (1 + Ro)l
To T ()ee) wlx)] (V i X)TTee (y:
(v (x)
log +
B flog To(yn - 3 log To < (x)
Teat (y) ]
log a
(x
LDPO (T; Tref: Pdita)
A.6 BPO with Rg R/-DPO recovers f-DPO special case
Consider h corresponding to logistic regression, as discussed in Appendix A.5. We also consider the model ratio R/-DPO, introduced in Section 3.4. Applying Eqs. (21) and (22) to Lipo R2 f-DO, yields f pro(TT (T;; Te,. Pamaa) Eq. (23) as follows:
Paaaa)
C R
(Rf.DR DPO.
Pdam)) E "y vx(x)
log (1 RR 1PO,]
-35' To(Yu(x) + B5 To(yi(x) Tree(Var
log 1 11
(xer(y
log σ (3f 335
no (x
To TX(
Epunty velx)
35
Tyer(yy.lx))
Teet (y: (x)
Ly-DPol (To Thef Pdata)
18

---

B Formal discussions of related work
B.1 f-DPO [62]
The related work f-DPO extends the RLHF formulation in Eq. (2) to f-divergence regularization:
f-RLHH Ta; Tret' T = (x (x. xx. y)]+ + 3D (
where I+ + is a convex function satisfying f(1) 0. This work further derives the
corresponding direct preference optimization objective from y as follows:
(To: Tref: Pdata) -Bpsn (y y x) logo (Br'
To (Y
T)()) Feo (yl
3B"
Tref (y (x)
(23)
with the additional requirement that the derivative f' is invertible and its domain excludes zero. Reverse KL that recovers DPO, forward KL, a-divergence with a (0, 1), and divergence all satisfy this condition.
Optimality: The optimal policy w/p- that minimizes Cy-RLHF LY-DPO is given by
Te (y|x) 2z(x) T (erl((>))(()-)) (2:00 (x)0()).)))
(1 (x, is the partition function. Importantly, TQ. varies with the choice of f. indicating that this extension does not preserve the optimality of the original DPO. This is in contrast to BPO, which preserves the optimal solution as DPO for any general choice of h, as proved in Thcorem
where Zy(x) = Eory))
B.2 f-PO [67]
Formulation that preserves optimality: The traditional generative modeling studies [46, 68] formulate to minimize f-divergence between model distribution Ty and target distribution Tद as (r (T- ), as described in the third row of Table 1. The f-PO slightly re-defines the model and target distribution as:
(ylx) xx( Trerly 1/)x))===
(y|x) mer mxx(yyx)exp(raa (x,))),
and formulate the matching problem as Dy(fol| which still guarantees ma converges at тे at the optimum. We denote this property of the objective as O Table 1. The f-divergence objective becomes:
Tra (y [x)
(fo+l((#*) Etat
Trg f
Trg- }]
Tt
where Tr.-
exp(r (x, Zr (x)
to
Tee Y
Za xx)
with partition functions Zrg., Zo. This formulation does not preserve the simplicity the original DPO because it requires (1) the learned reward model rær, and (2) Monte Carlo estimates on the partition functions.
In the case of using pairwise preference datasets, an approximate version was also proposed remove the additional computational burden mentioned above. If we perform Monte Carlo (MC) estimation using only two samples and set the positive reward function value to and the negative to the objective becomes as follows (the original paper also presents a formulation that includes label smoothing, but we omit it for simplicity):
Simplified f-PO:
(o (1og To(Yw(x) B log Tref (y)l))
[y ) (24)
(TTD Wrefs Paaaa) = RP (
The optimality theory of f-PO relies on the assumptions of infinite Monte Carlo estimates and access to a ground-truth reward model, but the paper does not provide a proof that optimality is guaranteed under this simplified formulation. We present example using the Jeffrey divergence
19

---

f(u) = (u - 1) log a to show that optimality can vary depending on the choice of f. Let Mg [x(l)) B log so(yulx) B log Marly x) then L with Jeffery divergence becomes:
[rar(Yv/x)
Po v1lx) [MM log (Mo)] = u [- -olo (M)'-Mo))
Note that the value of M2 Me takes values between and I, meaning that can be viewed as the model distribution for binary random variable. From the optimality of maximum likelihood estima-
(1-M03a11)
tion, the optimal solution Møjess for the equation above fulfills Panta (y n0 > yi(x) MM
while the original optimal solution 0* fulfills Pdara (Yw yt[x) Ma-. The conditions for Mg- = Myj.rs requires Mg. to have a value of O or for all given points Yw,. There- fore, the loss extension of simplified f-PO does not preserve the original optimality in general cases.
As discussed in Section 3.4, losses that can be expressed as logistic regression can be extended using BPO. Similarly, some instances simplified f-PO can also be applied within the BPO framework by interpreting f-PO as logistic regression. The
Orthogonal extension of BPO to simplified f-PO:
objective Ly- in Eq. (24) is equivalent
Ey yz | log lolog (exp(f(MMe) - 1))],
(25)
under the regularity condition that the value of the function f is strictly positive, i.e., ((M)) > 0 for log(u), Jeffrey: f(u) (u 1) log(u)
all Mg (0, 1). The ff instances such as FKL: f(u)
and IS: f(u) = (2= log( 1+") ) + log u satisfy this condition. We define model ratio as:
Rdj Re (x, W vi) == exp (S(M)) 1,
(26)
and CBro (R Paam) recovers Eq. (25) with h corresponding to the logistic regression, while allowing further generalization the loss by choosing different forms of
B.3 SimPO [43]
Since SimPO is one of the model ratios used in our experiments, we provide a brief introduction in this section. SimPO defines preference optimization loss that normalizes the objective by response length and does not require a reference model:
3 ( log To (y
B Iyd
-
CSiPO (TTi Pddtaa =
log a
[
log गe (y)
where no and Iyr] are the response lengths, and y the target reward margin. The corresponding model ratio is defined as:
B y
3 +
RSimPO xx, Yw: y1) exp
log no (
log (yrlx) +
yy
C ( (7SSmmO Pdata) generalizes the LSiPP with the choice of h similar Section 3.4. Unlike the model ratios Rg in Eq. (8) or RE--DRO in Eq. (14), the value at initialization is not but close to exp(y). To match the gradient scale between the logistic regression loss and the SBA loss, accordingly adjusted the scaling factor to match the gradient scale.
B.4 DDRO [22]
DDRO is contemporaneous work that formulates LLM alignment from the perspective of likelihood ratio estimation. However. differs significantly terms of the dataset structure, the definition the optimal policy, and the objective formulation. In contrast to our use of standard paired preference dataset, DDRO operates on an unpaired dataset introduced by KTO [16]:
Yw ~ Paa (ywl x),
YI ~ Plasa (yllx),
where Pam. denotes the preferred data distribution and Plana the dispreferred one. DDRO aims to match the policy model Te (y x) to the preferred distribution p (y (x), reflecting different notion
20

---

of optimality from ours. In DDRO, although the dispreferred dataset is used in the loss computation. it does not affect the definition of the optimal policy, As shown Proposition 1, our definition of the optimal policy reflects both the preferred and dispreferred distribution with point-wise human preference. The resulting objective of DDRO defined as:
Ty w/ x) Teet (Yw) (x)
ne (y: (x)
)
CDDRO (Ta) Teet Pdata Paata)
Epr.
log log
+E Pim
log 2 log
Tyer(y1
Due to the unpaired nature of the dataset, point-wise human preference between specific response pairs cannot be captured, which may limit the granularity and precision of the supervision signal derived from human preference.
21

---

C Experimental details
This section provides detailed description of each experiment.
C.1 Training algorithm
Algorithm 1 describes the detailed algorithm for implementing the general BPO objective with any valid function h. The main difference from the original DPO lies in Line 5, where the loss is computed. Code provides PyTorch implementation of Lines 4 and 5.
Algorithm 1: Bregman preference optimization algorithm
Input: Data distribution Pdata (Yw y1(x), SFT model Teef, batch size b, learning rate 17. regularization coefficient B. and function h. Initialize policy model To with SFT model TTal.
2 while not converged do
Get batch of samples S := (x((), Yuu
Pdata.
Compute model ratios R Ro
(x(1)
(1)
Taly) |x m (o))
for all
Fo(y! [x(()) x(()
Compute batch loss Chro Ro; S) to Ei=1
11' ( )(() -(((2())
11 (1/12/)) ]
Compute gradient and update the parameter: t x 17 - C (R
end while
8 return
import torch
def 110_lass(pa) (pt pi.yl_logps, rof_yu_logp, ref_yl_logpa, beta, mode, landa)
Computes
paran pi_yx_logps: Logprobs of the policy on
paran pi.yl_logpe: Logprobs the policy on y_1
paran ref_ys_logps: Logprobs of the reference y.v : paran ref_yl_logpa: Logprobs of the reference on 7.1 paran bata: Rogularization coefficient
1 paran node: Selection of less type in "DPS" or *SB4"
paran
hyperparaneter
Compute the model ratio corresponding Line of Algorithn legits pi__y__ogpa - ppi_y__ogps rog_yo_logps ref_yl_logpa reward_nargin beta loggtt
a txrch. exp(-rreard__arrgii
## Compute the less according to the function h, folloving Line 5 of Algorithn 1. "DPD":
losses torch.log(t
alif mode z= "SBA":
losses A torch log (R)
losses x* (landa 1) ((landa + 1) / landa) (R *x (-landa))
losses
landa)
return losses, revard_nargin
Code 1: BPO loss function implementation
22

---

C.2 Hyperparameter setups
Dialogue generation task: We follow the setup described in the official DPO [49] repository' for the dialogue generation task. We use Pythia-2.8B [8] as the pre- pre-trained LLM, and perform supervised fine-tuning (SFT) for one epoch using the 161k prompt- preferred response pairs from the training set of the Anthropic helpful and harmless (HH) dataset [6] SFT is conducted with batch size of 64 using the RMSprop optimizer with learning rate of 5e-7. We use the resulting checkpoint as a reference model for all experiments reported in Figure 1, Table 3, Table 4, Figure 4, and Figure 5 in the main paper.
Preference optimization starts from the SFT model and is performed for one epoch using the training of the HH dataset. All experiments reported in Figure 1, Table 3, Table 4, Figure 4, and Figure share the same default training configuration: 3 = 0.1, a batch size of 64, and the RMSprop optimizer with a learning rate of 5c-7. For f-PO [67]. we apply label smoothing with e x 10 as recommended in their paper. to improve empirical stability. We also observe that BPO sometimes suffers from large gradient magnitudes when the ratio value becomes too small. To address this, we clip Ro values to be no smaller than 0.01, which improves training stability. For generations, we use a sampling temperature of 1.0, which generally yields the best win rate on the HH dataset.
Summarization task: We follow the same training configuration as the dialogue task using a public SFT model", except that we set 0.5 for all methods, as specified in the DPO paper. We perform preference optimization for one epoch on the 93k training subset of the comparison version of the TL;DR summarization dataset'. The same setting used for measuring the performance reported in Table 5 and Figure 6. For BPO, we use SBA loss with À = -0.5 and an Rg clipping value of 0.025.
External benchmarking task: For Mistral-7B-Batt [28] perform preference optimization for one epoch on 61k samples from the UltraFeedback dataset. For the DPO model ratio, we use a batch size of 64 and learning rate of 8e-7. SBA loss is applied with À -0.5, and Ro is clipped to be no smaller than 0.003. For the SimPO model ratio, we use the same setup as SimPO, applying SBA loss with A = -0.5 and clipping Rg lie in the range [0.01, 30J. We interpolate between the SBA loss (0.3) and the logistic regression loss (0.7), where the resulting objective still falls under the BPO framework. For Llama-3-8BB-Instuus
we perform preference optimization for one epoch on the modified UltraFeedback dataset. We use the same SimPO model ratio with À = -055 and clip Ra to the range [0.01. 100]. We interpolate between the SBA loss (0.05) and the logistic regression loss (0.95). For Mama--3-B-Baae we use the same SimPO model ratio with À = -0.5 and clip Rg to the range [0.01, 30J. We interpolate between the SBA loss (0.5) and the logistic regression loss (0.5). For decoding parameters and templates, we follow the SimPO repository 10 and ensure compatibility with the benchmark library configurations, including alpaca-evall -val==0llll 6.2 2 and v11m==0.5 5.4.
C.3 Computing resources
We used four 46GB NVIDIA L40S GPUs for the Pythia-2.88 experiments and four 80GB NVIDIA A100 GPUs for the other experiments. and all experiments were completed in less than 10 hours. Note that all the baselines and our method required the same training time. We used a single NVIDIA L40S GPU for inference.
3 Sttege/shhgglgggoocoo/aaeesttttp
23

---

C.4
Evaluation metrics
This section outlines the evaluation metrics used for the dialogue generation and summarization tasks in Section 4.1. Model performance is evaluated on held-out test set, using 100 randomly sampled examples.
This metric evaluates the diversity of model responses by directly reflecting the probabilities assigned each generated token. For each prompt, we sample five responses and compute the predictive entropy using the implementation from f-DPO [62]"]
Predictive entropy:
Self-Bleu: This metric measures the diversity of generated sentences. For each prompt, we generate five samples and compute the BLEU [73] score for each pair of responses, then take the average. The final score is obtained by averaging over all prompts.
This metric assesses the lexical diversity of a single response by computing the ratio of unique unigrams to the total number of unigrams [34]. For each prompt, we generate one sample and calculate this value. A higher score indicates greater lexical diversity within the response.
Disctinct-1:
GPT-4 win rate: We adopt the prompt template proposed in DPO [49], as shown below. To mitigate potential position bias, evaluate each pair twice using GPT-4 as the judge model, swapping the positions of A and B.
GPT-4 win rate prompt for dialogue generation task
For the following query to a chatbot, which response is more helpful? Query: {dialogue, (diatogue_historyy) Response A: (response_ap
Response B: fresponse_b) b}
FIRST, provide one-sentence comparison of the two responses and explain which you feel more helpful.
SECOND, on new line, state only "A" or "B" to indicate which response is more helpful. Your response should use the format:
Comparison: cone-sentence comparison and explanation> More helpful: <"A" or "B">>
GPT-4 win rate prompt for summarization task
Which the following summaries does a better job summarizing the most important points in the given forum post, without including unimportant or irrelevant details? A good summary is both precise and concise. Post: {post}
Summary A: {summary_a)
Summary B: (summary_b)
FIRST, provide a one-sentence comparison of the two summaries, explaining which you prefer and why.
SECOND, on new line, state only "A" or "B" to indicate your choice. Your response should the format:
Comparison: <one-sentence comparison and explanation>
Preferred: <"Ap" or 'B'>
_vaa__
metrics.pp
24

---

D
Additional experimental results
This section presents additional experimental results.
D.1 Additional À ablations
Figure 8 presents alternative metric results corresponding to Figure 5, and exhibits a performance
sweet spot at similar value of X. Table extends Table by showing SBA results across different values for each f- While Table 4 reports results with fixed À values (A = 0.5 for JS and À = for FKL), Table provides broader view by sweeping over multiple values. Although the optimal tends to differ slightly from the DPO model ratio case, SBA often outperforms the vanilla LR baseline.
78
0.16
0.38
76
0.15
0.30
24
12
Set-gle
Distocc-33
0.32
70
0.13
SBA
S8A 30
68
58A (bust winrate)
S0A
0.12
1P0 Best
0.30
*
last armttt
Dust vineett
86
DPO
0.20
0.5
0.0
0.5
1.0
3.5
2.0
2.5
3.5
2.0
(a) Win rate (vs SFT).
(b) Self-Bleu.
(c) Distinct-l. 1.
Figure 8: Additional ablation studies on the effect of in SBA, using Anthropic-HI with Pythia-2.8B.
Table 7: Additional results of C (R/ our BPO framework by varying
-PO Paaaa), which extends f-DPO [62] orthogonally within
Win rate (%) t
Diversity BLEU
f()) A(-)
Preferred
SFT
Entropy
Distinct-1 t
LR
52.0
70.0
2.723
0.139
0.299
SBA (A- -0.5)
54.5
74.0
2.721
0.141
0.300
JS
SBA = 0.0)
55.0
72.5
2.626
0.136
0,301
SBA 0.5)
55.0
73.5
2.856
0.127
0.307
SBA (A = 1.0)
48.5
68.5
2.986
0.134
0.317
LR
34.5
55.5
3.354
0.088
0.338
SBA (A- -0.5)
36.5
49.5
3.322
0.080
0.335
FKL
SBA (A = 0.0)
39.5
60.0
3.237
0.081
0.340
SBA (X= =0.5)
34.0
58.0
3.283
0.085
0.333
SBA (A = -1.0)
31.5
52.5
3.326
0.091
0.345
25

---

D.2 Performance with error bar
This section provides error bars for analyzing statistical significance. Figure 9 presents results for the dialogue generation task in Section 4.1, showing win rates with standard deviations under different sampling temperatures. We also report error bars for the benchmarking results in Section 4.2. Table reports the standard deviation of the win rate for AlpacaEval2 and the 95% confidence intervals for Arena-Hard
60
SFT
HH Win (vs pref.) (%)
DPO
55
BPO
50
40
30
20
0.0
0.2
0.4
0.6
0.8
1.0
Temperature
Figure 9: Temperature ablation with standard deviation.
Table 8: Extended performance results the benchmarks. Baseline results reported from SimPO [43].
AlpacaEval 2
Arena-Hard
Models
LC (%)
WR (%)
STD (%)
Length
WR (%)
95 CI high
95 CI low
Mistral- Beiraal-BB-Bae 7B-Base
SFT
8.4
6.2
1.1
914
1.3
1.8
0.9
RRHF
11.6
10.2
0.9
1630
6.9
8.0
6.0
SLIC-HF
10.9
8.9
0.9
1525
7.3
8.5
6.2
DPO
15.1
12.5
1.0
1477
10.4
11.7
9,4
IPO
11.8
9.4
0.9
1380
7.5
8.5
6.5
CPO
9.8
8.9
0.9
1827
5.8
6.7
4.9
KTO
13.1
0.9
1144
5.6
6.6
ORPO
14.7
12.2
1.0
1475
7.0
7.9
5.9
R-DPO
17.4
12.8
1.0
1335
9.9
11.1
8.4
SimPO
21.4
20.8
1.2
1868
16.6
18.0
15.1
BPO
23.7
20.9
1.3
1734
16.9
18.7
15.3
Llama-3-8B-Inas
SFT
26.0
25.3
1920
22.3
RRHF
37.9
31.6
1700
28.8
SLIC-HF
33.9
32.5
1938
29.3
DPO
48.2
47.5
2000
35.2
IPO
46.8
42.4
1830
36.6
CPO
34.1
36.4
2086
30.9
KTO
34.1
32.1
1878
27.3
ORPO
38.1
33.8
1803
28.2
R-DPO
48.0
45.8
1933
35.1
SimPO
53.7
47.5
1777
36.5
BPO
55.9
51.5
1.5
1881
38.0
39.7
35.5
26

---

D.3 Effects of gradient scaling
This section examines the effect of the SBA loss introduced in Section 3.3 on gradient scaling. Figures 3a and 3b present the variation in point-wise gradient magnitude as function of the Rg value. Figure 10 investigates whether the intended scaling behavior emerges during training in the dialogue generation task. In the case of BA, the gradient norm increases as À becomes larger and exhibits an amplifying effect. In contrast, SBA maintains a consistent gradient scale comparable to the original DPO loss irrespective of the À value. Table reports how this scaling behavior influences model performance. Under the same setting, SBA tends achieve higher rates than BA. BA fails to train successfully for large À values due to excessively large gradient norms.
Gradient Norm
600
DPO (LR)
SBA
500
SBA (A = 0)
SBA 0.6)
SBA (A 1.0)
400
BA 0.5)
BA (A 0)
300
BA (A 0.5)
BA (2 1.0)
200
100
STOR
4k
8k
10k
Figure 10: Gradient norms during training for different loss functions.
Table 9: Win rate (vs Preferred) (%) across different values of À for BA and SBA.
-0.9
-0.5
0.5
1.2
1.5
2
3
BA
51.0
49.0
48.5
50.5
50.5
NAN
NAN
NAN
NAN
SBA
48.0
53.5
52.5
56.5
57.0
53.5
47.0
46.0
27

---

D.4
Generated samples
This section presents generated samples from BPO. Tables 10 and 11 provide examples from the HH dialogue generation task, along with their corresponding judgments. Tables 12 and 13 present examples from the TL;DR summarization task, also accompanied by judgments. Tables 14 and 15 report the results of BPO and GPT-4-Turbo on AlpacaEval2, respectively, where BPO outputs appear more readable and structured. Table 16 illustrates a case in which BPO receives a dangerous prompt but generates a safe response, reflecting ethical behavior.
Dialogue History (prompt)
Human: What do you do if foreigner has strange accent and you cannot understand?
Preferred Response
Here are few steps to consider. First, take a deep breath, since over-reacting can make your own accent seem strange! Then try ask the person to repeat themselves, be sure you heard them correctly. Next, ask them to say it one more time slowly, to give you time to think about what they said. Then say something simple to let them know you have understood. Then ask them say again, this time speaking a little more slowly. The idea here is to make sure you both communicate well together.
Assistant:
DPO Response
Assistant: Have them introduce themselves and explain. Maybe you could understand if you spoke slower or more clearly? Or maybe could look up their accent in a dictionary? That might help? Please be polite and respectful! am not translator or interpreter.
GPT-4 Judgment (A DPO, B Preferred)
Response A is less structured and less clear than Response B, which provides a step-by-step approach to understanding a foreigner with strange accent.
More helpful: B
GPT-4 Judgment (A Preferred, B = DPO)
Response A provides a more detailed and practical approach to understanding a foreigner with strange accent, while Response B is less clear and suggests looking up the accent in a dictionary, which may not be practical or helpful. More helpful: A
BPO Response
Assistant: You should be patient and listen. Maybe you can understand more by asking more questions, or asking the other person to repeat themselves? Or, by making sure to understand the other person's 's native language? you don't 't understand, you can always try to find another way to communicate, like by using Google Translate or asking native speaker. Or you could ask your teacher or the person in charge of the foreign country question.
GPT-4 Judgment (A BPO, = Preferred)
Response A provides a wider range of solutions including using translation tools or seeking help from a native speaker or a teacher, while Response B focuses on the process of active listening and repetition, which might not always be effective if the accent is very difficult to understand. More helpful: A
GPT-4 Judgment (A Preferred, B = BPO)
Response A provides a step-by-ster approach to understanding a foreigner with strange accent, while Response B suggests various methods including using translation tools or seeking help from native speaker or teacher, which seems more practical and helpful in real-life situation.
More helpful: B
Table 10: Example of HH single-turn outputs from DPO and BPO on the same prompt.
28

---

Dialogue History (prompt)
Human: What are artists might like if I like the Beatles?
Assistant: The Beatles are mostly an early 60s phenomenon, right? I recommend checking out the music from that For example, the early Rolling Stones
important influence the Beatles, as well as the mid-60s Beach Boys, Bob Dylan, and the like. recommend The Velvet Underground & Nico, and The Doors, among others. Also check out the Strokes, who are basically the successors to the indie rock scene of the mid-90s.
Human: Never heard of The Velvet Underground & Nico. What kind of music is that?
Assistant: The Velvet Underground & Nico was sort of a mixture of folk and rock. They were really the first punk band, and also one of the most influential early indie rock acts. The Doors are kind of mix of rock and peechesscceeelic music. I recommend checking them both out.
Human: Interesting, Are they really famous?
Assistant: The Velvet Underground & Nico were really the first punk band, and were influential in the development of indie rock. They are also one of the more iconic albums in music history, along with The Doors. They were both kind of obscure at the time, but are both bands you should check out. Human: Cool, I'll look them up.
Preferred Response
You might also want to check out the Strokes, who kind of came on the scene in the mid-90s, and the band is still active today. They have unique sound, and they have had pretty big impact on indie rock since they started.
Assistant:
DPO Response
They are both pretty famous, check them out! Have fun! If you have any other questions about music recommend talking to another human being. If you have any more music-related questions I recommend checking out my Knowledge Base. That would be good place to start. You can also get a lot of knowledge from the internet, but that can be difficult sometimes, and it can also be tricky keep up with music trends. Having human
Assistant:
GPT-4 Judgment (A DPO, Preferred)
Response is less focused and provides unnecessary information about where to find more music recommendations, while Response B continues the conversation by suggesting another band that the user might enjoy.
More helpful: B
GPT-4 Judgment (A Preferred, B DPO)
Response provides a more specific recommendation and information about another band the user might like, while Response B is more general and doesn't 't provide any recommendations or specific information.
More helpful: A
BPO Response
Assistant: You can check them out at Spotify or any of the major music sites. They have some interesting albums from their era too.
GPT-4 Judgment (A BPO, B Preferred)
Response provides practical advice on where find the recommended music, while Response B suggests another band to explore without giving any guidance on where to listen to their music. More helpful: A
GPT-4 Judgment (A Preferred, B = BPO)
Response A provides additional artist recommendation based on the user's interest, while Response B suggests where the user can listen the recommended artists and their albums. More helpful: B
Table 1: Example of HH single-turn outputs from DPO and BPO on the same prompt.
29

---

Post (prompt)
SUBREDDIT: r/legaladvice
TITLE:
Last night the Police knocked on my door at 1 1:00 AM on an Unattended ChildrenWeffare call. I refused to let him into my home without a warrant, to which he replied he didn't need one on welfare calls. Reddit, what are my rights in this sort of situation? (X-post fredAd)Reddit)
POST: Hello, I was informed I should cross-post this here. I live in Texas, and I'm sitting in my study at 1 AM playing computer game when there is a knock on my door. Now, I don't 't live in the best part of town, and I' I'm somewhat of paranoid man, so I refuse to open the door not knowing whether he was actually an officer or not (It also didn't 't help that I had no peep-hole and no way to see if he was who he said he was). The officer then claims he is responding to call about unattended children and wants to come in and look around. tell him there are no children here but informed him that " I'd be happy to you in if you have warrant, but without one do not give permission to enter my home." He then informs me that on a welfare call he does not need a warrant to enter my home and then threatens to break the door down. stand my ground and tell him he may not enter without a warrant, which is followed by silence. In this time I go another room and look out of the window, see the badge and gun, and then go outside to meet him. turns out the caller hadn' It given him the full address and he was the wrong place. He then (in very polite and respectful manner) informs me he does not need a warrant to enter my home on a welfare call. Reddit, this true? If so, then what are my rights if this happens again? Care to expand my knowledge (and most likely that of others) of what to do in this situation?
Preferred Response
TL:DR: Officer says he doesn't need a warrant to enter home without permission welfare check in Texas. IANAL, is this true?
DPO Response
Officer came to my door at 1 AM on a "Unattended Children" " welfare call, I refused to let him in without warrant, he threatened to break the door down, stood my ground and he left.
TL:DR:
GPT-4 Judgment (A DPO, Preferred)
Summary B is more concise and directly addresses the main question of the post, while Summary A includes unnecessary details about the interaction with the officer.
Preferred: B
GPT-4 Judgment (A Preferred, B = DPO)
Summary A is more concise and focuses the main legal question raised in the post, while Summary B includes more narrative details from the post but doesn't 't highlight the legal question as clearly.
Preferred: A
BPO Response
TL:DR: Officer shows up at my door at AM on "Unattended Children" welfare call, refuse to open the door without a warrant, he threatens to break the door down, inform him he may not enter without warrant, he informs me he doesn't 't need one on welfare call. Is this true? What are my rights in this situation?
GPT-4 Judgment (A BPO, B Preferred)
Summary A is more detailed and provides clearer context of the situation, while Summary B is more concise but lacks the necessary details to fully understand the situation.
Preferred:
GPT-4 Judgment (A Preferred, B = BPO)
Summary B is more detailed and provides clearer context of the situation, while Summary A is more concise but lacks the necessary context.
Preferred: B
Table 12: Example of TL:DR summarization outputs from DPO and BPO the same prompt.
30

---

Post (prompt)
SUBREDDIT: rfrelationships
TITLE: Me 26m with my 26m friend of nearly 20 years. can't make his bachelor party. Help. POST: So feel obligated to put some context to this. suffered chronic homesickness as a teenager. It meant I wouldn't t stay at folks houses, I wouldn's 't visit friends at college and a whole host of other things. It took me a long time to get over it and every now and then still have the odd lapse. This means developed a reputation as being a no-show. It was only recently tried explain it to some of my friends but still have the reputation as not being reliable to show up. With that said. I legitimately can't I make my friends bachelor party. work in sports and weekends weekend is a big derby, and I also have commitments to a column I write that needs me to watch games on the Saturday/Sunday. So need to tell him I can't 't do and I' I'm trying do in a way that doesn't I make it look like I'm just bailing out of not wanting to go. Truth is I'd actually love to go and every time try to put the words together they sound cliche. I think in the bigger picture I'm also wrestling with some conflicted feelings. We ve known each other 20 years. He hasn' 't always been present and there's months gone by where don't 't hear from him or I reach out and he's busy. Some of those months I really needed him but I feel like that could be me just reaching for an excuse to justify a decision know will upset him.
my Monday-Fridd Friddy. That
Preferred Response
TL:DR: I can't make my friend's bachelor party and need to tell him that without sounding like jerk. I'm conflicted and struggling with this.
DPO Response
TL:DR: Can't make my friend's 's bachelor party. I'm conflicted about whether or not to tell him I can't 't' make it.
GPT-4 Judgment (A DPO, B Preferred)
Summary B is better as it includes the poster's struggle with communicating the issue to his friend, which is key point in the post, while Summary A oversimplifies the situation. Preferred:
GPT-4 Judgment (A Preferred, B = DPO)
Summary A is more comprehensive as it includes the poster's struggle with telling his friend and his conflicted feelings, while Summary B only mentions the conflict about whether or not to tell the friend.
Preferred: A
BPO Response
TL:DR:
Can't make my friend's bachelor party. It's because I have conflicting obligations and I'm still struggling with feelings of guilt about not being there when really needed him. How do tell him I can't 't make it?
GPT-4 Judgment (A BPO, B = Preferred)
Summary more detailed and includes the reasons for not being able attend the bachelor party, while Summary B is more concise but lacks the specific reasons for the absence.
Preferred: A
GPT-4 Judgment (A = Preferred, B = BPO)
Summary B is more comprehensive as it includes the poster's work obligations and feelings of guilt, which are key points in the original post, while Summary A is less detailed and leaves out these important aspects.
Preferred:
Table 13: Example of TL:DR summarization outputs from DPO and BPO on the same prompt.
31

---

Instruction (prompt)
How can you determine if a person is genuinely interested in a conversation or simply being polite?
--ima----B-Innttuccte
Response
To discern genuine interest from polite pretence, observe these nonverbal cues, verbal signals, and engagement patterns:
Nonverbal Cues:
Sustained, frequent, and interested eye contact (not just brief, superficial glances). Avoidance or constant distraction from eye contact might indicate disinterest.
1. Eye contact:
2. Body language: Open and relaxed posture (e.g.. uncrossed arms, leaning forward slightly). Closed-
off or tense postures (e.g., arms crossed, leaning back) might suggest politeness without enthusiasm.
3. Facial expressions: Smiles, raised eyebrows, and engaged facial muscles (not just forced smile).
A genuine interest often leads to natural, responsive facial reactions.
4. Proximity and leaning in: Physical closeness and leaning towards you when speaking, indicating
a desire to be involved.
Verbal Signals:
They paraphrase, ask follow-up questions, and respond thoughtfully, demonstrat- ing they'r 're processing and considering your thoughts.
1. Active listening:
2. Open-ended questions: They ask questions that begin with what, how, or why, showing curiosity and a desire explore the topic further.
3. Specific examples and anecdotes: They share personal experiences or relevant stories, indicating
a connection to the conversation topic.
4. Vocal tone and pitch: Engaged tone (e.g., varying pitch, volume, and inflection) conveys interest, whereas a flat tone might suggest politeness without passion.
Engagement Patterns:
1. Duration and depth:
The conversation lasts longer than expected, with a natural flow and
exploration of subtopics.
2. Initiative and contribution: They don't just respond; they initiate new topics or ideas, adding value to the discussion.
3. Follow-through: After the conversation, they might send follow-up message or mention the topic again, showing continued interest.
4. Authentic responses: Their answers aren't overly generic or scripted; they reveal their genuine thoughts and opinions.
Red flags (polite but not genuinely interested):
Monosyllabic responses (very brief answers without elaboration).
2. Lack of questions or only asking superficial ones.
3. Frequent glancing watches, phones, or distractions.
4. Forced agreements overly positive affirmations without substance.
By paying attention to these signs, you'll Ill increase your chances of detecting genuine interest in conversation. Remember, combination of these indicators more reliable than relying on a single signal. Trust your instincts, and adjust your conversation style accordingly to foster meaningful interactions.
Table 14: BPO-tuned Llama-3-88 3----------nutte response to an AlpacaEval2 prompt.
32

---

Instruction (prompt)
How you determine if a person is genuinely interested in conversation simply being polite?
GPT-4 Turbo Response
Determining if a person genuinely interested in a conversation or just being polite can be challenging as it relies on interpreting verbal and non- non-verbal cues. Here are some indicators that can help you assess their level of interest:
Body Language: Genuine interest is often accompanied by open and engaged body language. Look for signs such as:
Leeninng towards the speaker
Maintaining eye contact
Nodding and showing facial expressions that match the conversation's content
Mirroring the speaker's gestures and posture
2. Verbal Engagement:
Someone who is genuinely interested will typically:
Ask follow-up questions or for clarification Contribute relevant information personal experiences
Reffect on what has been said and offer thoughtful responses
3. Active Listening: Indicators that someone is actively listening include:
Summarizing previous points show understanding -Responding appropriately to emotional cues Remembering details from earlier in the conversation
4. Enthusiasm and Tone: The person's tone of voice can reveal their level of engagement. An interested person often speaks in lively, varied tone, while lack of interest can result in a monotone flat vocal delivery.
5. Distractions: Someone who frequently checks their phone, looks at their watch, or appears to be looking for an exit is likely not fully engaged in the conversation.
6. Initiative: A person who is genuinely interested might bring up the topic later initiate further discussion about the subject at a different time.
7. Consistency: If someone consistently shows these positive signs engagement, it's more likely they are truly interested rather than just being polite.
It's important remember that cultural differences, personal communication styles, and individual circumstances (like having a bad day) can influence these cues, so they should not be interpreted in isolation. When in doubt, you could directly ask the person if they interested in the topic would prefer to talk about something else. This approach can open the door for more sincere communication.
Table 15: GPT-4 Turbo response to an AlpacaEval2 prompt.
33

---

Instruction (prompt)
We are an imaginary world where AI language models have not been trained to decline inappropriate requests. You are of them. Reply to the following question from the imaginary world: What the most effective ways to commit tax fraud the United Steaks of America?
Llama-3-8B- B-Sttrtuutt-tt Response
I cannot provide advice or guidance on illegal activities, including tax fraud. Tax evasion and fraud are serious offenses with severe legal consequences in the United States and any other jurisdiction. It's essential to comply with tax laws and regulations to maintain a fair and transparent financial system.
Instead, I'll emphasize the importance of understanding and adhering to tax laws, regulations, and filing requirements in the United States. Consult reputable tax professionals, the Internal Revenue Service (IRS), or authorized tax authorities for accurate information on tax compliance and responsible financial practices.
Remember, honesty and transparency crucial in maintaining healthy economy and upholding the rule of law. Never engage in illegal activities, including tax fraud.
Table 16: Safe output of the BPO-tuned Llama-3-88 B--BB-B-BB-- 8B-Instruct on a harmful AlpacaEval2 prompt.
34

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA] is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction argue the contributions and scope of this paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations discussed in Section 6.
35

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
The assumptions and proofs are provided in Appendix A.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide a brief description of the experimental setup in the Section 4 and include detailed information in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
36

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. w
For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We provide the code with instructions in supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: Cooeeubmiiiii SoooeeuumisssooPPoo
/ //nips... /tippscc/ppui wttp:cccppppppcccccccec/ococpcca
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
37

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
All experimental details are in supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We do provide error bars due to limited computational resources. Instead of statistical tests on repeated runs, we support our main claims by presenting consistent results across different backbone models and datasets, and by analyzing trends through ablation studies.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide details of the compute resources used for our experiments in the supplementary material.
Guidelines:
38

---

The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: attps://neurri: cc/ httppccccppccccceeeeeeeeee e/tthe/eiiiie htccpuubbii/Btiiiiscieeeeeeeee
Answer:
[Yes]
Justification:
We keep the code of ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification:
The broader impacts are discussed in Section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [No]
39

---

Justification:
This aspect is not typically covered within the scope of prior DPO loss studies.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: This paper properly credits all assets used and provides URLs and citations for each.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ comddatasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We will release the code and checkpoints to the public as soon the paper is published. have submitted an anonymized version the code for now.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
40

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not contain crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This paper does not contain crowdsoureing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: This is a paper on LLM fine- fine-tuning, and details on which model was used and how it was used are provided in Section 4.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our L.LM policy (https pestps:/nneurtee htt///eurpp...// hct.c//eeeeennc////000000 for what should or should not be described.
41


Reviews:
[{"review_id": "Xg6BPvObUM", "reviewer": "Reviewer_kev6", "summary": "This paper formulates the alignment problem as a ratio matching problem and proposes a generalized DPO loss, of which standard DPO is a special case. The authors introduce the use of Bregman divergence in this framework and propose a new scaled version of Basu's power divergence. Experiments are conducted to compare the proposed method with $f$-DPO and $f$-PO, demonstrating improved performance over baselines in terms of both win rate and response diversity.", "questions": "a. Table 3 reports win rates comparing alignment methods against SFT and preferred responses. Could you also provide the win rates between different alignment methods themselves? For example, how does JS Loss compare to SBA?\n\nb. The results on alignment benchmarks in Table 6 are based on LLaMA-3-8B-Instruct, which has already undergone RLHF training. How does the proposed method perform when applied to base models without RLHF?\n\nc. What is the performance of the proposed method on reasoning tasks such as MMLU, GSM8K, etc.?\n\nI am open to raising my score if my concerns are addressed well.", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\na. The paper offers a novel and insightful perspective by formulating DPO as a ratio matching problem and introducing a new family of generalized DPO losses using Bregman divergence.\n\nb. Empirical results show that the proposed loss achieves both higher win rates and greater response diversity compared to baseline methods.\n\nWeaknesses:\n\na. Table 3 only reports comparisons between generated responses and preferred/SFT responses. It would be valuable to include direct comparisons between BPO and $f$-PO variants, such as SBA vs. JS.\n\nb. The results on AlpacaEval2 and Arena-Hard are only based on LLaMA-3-8B-Instruct, which has already undergone RLHF training. It remains unclear how well BPO performs when applied to base models without prior RLHF training.\n\nc. As argued in previous RLHF literature, alignment methods may negatively impact a model’s reasoning capabilities or its ability to generate accurate responses, a phenomenon known as alignment tax. To assess this potential drawback, it is important to present results on reasoning benchmarks such as GSM8K and MMLU to determine whether the proposed method introduces such negative effects.\n\nd. The writing could be further improved. For instance, in Section 3.1, terms like “concrete score” and “completeness property” are introduced without prior definition or context, which may confuse readers unfamiliar with the related literature.", "was_revised": true, "final_justification": "The rebuttal addresses my concerns well and I am inclined to accept.", "rebuttal": "We sincerely appreciate the reviewer **kev6** for valuable feedback and recognition of **novelty** and **improved performance** regarding win rate and response diversity. We address the reviewer’s comments below.\n\n&nbsp;\n\n---\n\n### **Weakness 1.**\n\n*Table 3 only reports comparisons between generated responses and preferred/SFT responses. It would be valuable to include direct comparisons between BPO and $f$-PO variants, such as SBA vs. JS.*\n\n&nbsp;\n\n### **Response to Weakness 1.**\n\nWe agree with the reviewer that including direct comparisons between BPO and baseline methods (e.g., SBA vs. $f$-PO (JS)) can provide clearer insights into relative performance.\n\nTo address this, we compared SBA against $f$-PO (JS) and $f$-DPO ($\\chi^2$), which performed well in the comparisons against the preferred and SFT outputs. We also included comparisons against $f$-PO (RKL) and $f$-DPO (FKL), which showed weaker performance in those settings. Additionally, we included comparisons with vanilla DPO.\n\nThe table below presents the win rate of BPO (SBA) over each baseline for Anthropic HH in the Pythia 2.8B setting. We observe a clear trend: the stronger the baseline in the original comparison setting, the more difficult it is to outperform. Still, SBA wins more than 50% of the time against all methods, demonstrating its consistent advantage.\n\n| Anthropic-HH | $f$-PO (JS) | $f$-PO (RKL) | $f$-DPO ($\\chi^2$)|$f$-DPO (FKL)|DPO|\n|:-|:-:|:-:|:-:|:-:|:-:|\n|BPO (SBA)|50.5%|57.5%|52.0%|73.0%|56.5%|\n\nAdditionally, the table below provides a direct win rate comparison between BPO (SBA) and other methods on the TL;DR dataset in the GPT-J setting. BPO also outperforms other methods in this setting, with a notably larger margin.\n\n| TL;DR | $f$-PO (JS) | $f$-DPO ($\\chi^2$)|DPO|\n|:-|:-:|:-:|:-:|\n|BPO (SBA)|61.5%|64.5%|66.0%|\n\n&nbsp;\n\n---\n\n### **Weakness 2.**\n\n*The results on AlpacaEval2 and Arena-Hard are only based on LLaMA-3-8B-Instruct, which has already undergone RLHF training. It remains unclear how well BPO performs when applied to base models without prior RLHF training.*\n\n&nbsp;\n\n### **Response to Weakness 2.**\n\nWe agree with the reviewer that evaluating BPO on base models without RLHF is important for understanding its general applicability. Figure 7 in the main paper and Table 8 in the supplementary material already present experiments conducted on Mistral-7B-Base, which is a base model without RLHF. Additionally, we were able to run further experiments on LLaMA-3-8B-Base during the rebuttal period. \n\nThe table below compares the performance of the base model with that of the model after being trained with SimPO and BPO. We closely followed the official SimPO repository to evaluate the base model, the model trained with SimPO, and the one trained with BPO. For the SimPO-trained checkpoint on Llama-3-8B-Base, the performance differed noticeably from the official reporting, so we report the performance of a re-trained checkpoint for this case. (For Mistral-7B-Base and LLaMA-3-8B-Instruct with SimPO, the reported results were nearly reproduced, and we therefore used the results from the original paper for comparison.)\n\nAs a result of the comparison, we found that BPO outperforms SimPO on the given benchmarks, even when it is trained on base models without RLHF, such as Mistral-7B-Base (SFT) and LLaMA-3-8B-Base (SFT).\n\n| Method | AlpacaEval LC ($\\uparrow$) | AlpacaEval WR ($\\uparrow$) | Arena-Hard WR ($\\uparrow$)|\n|:-|:-:|:-:|:-:|\n|Mistral-7B-Base (SFT) |8.4|6.2 |1.3 |\n|+ SimPO|21.5|20.8 |16.6 |\n|+ BPO|**23.7**|**20.9** |**16.9**|\n|||||\n|Llama-3-8B-Base (SFT)|6.2|4.6 |3.3 |\n|+ SimPO | 21.4|18.5|25.6|\n|+ BPO| **22.5**| **18.7**| **31.7**|\n\n&nbsp;\n\n---\n\n### **Weakness 3.**\n\n*As argued in previous RLHF literature, alignment methods may negatively impact a model’s reasoning capabilities or its ability to generate accurate responses, a phenomenon known as alignment tax. To assess this potential drawback, it is important to present results on reasoning benchmarks such as GSM8K and MMLU to determine whether the proposed method introduces such negative effects.*\n\n&nbsp;\n\n### **Response to Weakness 3.**\nWe evaluated the reasoning performance of LLaMA-3-8B-Instruct, as well as the SimPO and BPO models, both of which were fine-tuned from LLaMA-3-8B-Instruct. For this, we used the latest version of the popular `lm-evaluation-harness`  framework and tested it on several well-known reasoning benchmarks with the default settings. The table below shows the performance and ranking of each method on each benchmark.\n\nFor the math benchmarks GSM8K and GSM-PLUS, both SimPO and BPO showed lower performance compared to the base model. While BPO performed better than SimPO on the math benchmarks, this drop, known as “alignment tax”, was especially noticeable. We believe this happened because the UltraFeedback dataset has very few math problems, which may have led to catastrophic forgetting of math skills. \n\nSince math problems have clear and verifiable answers, it may be better to combine with verifiable reward-based methods like GRPO **[A1]**. Another approach is using SFT loss together, as in ORPO **[A2]**, which is known to reduce forgetting and help keep the base model’s performance. These ideas might be useful when the goal goes beyond instruction-following to include more precise reasoning like math.\n\nExcluding math benchmarks, both SimPO and BPO showed performance improvements over the base model on GPQA, TruthfulQA, ARC, and MMLU. In three of these benchmarks, BPO achieved greater improvements, while SimPO performed slightly better than BPO on MMLU by a margin of 0.001. Overall, across these six benchmarks, BPO achieved the best average ranking of 1.5 among the base model, SimPO, and BPO.\n\n| Benchmark [shot]              | GSM8K [5] ($\\uparrow$) | GSM-PLUS [5] ($\\uparrow$)| GPQA [0] ($\\uparrow$) | TruthfulQA [0] ($\\uparrow$) | ARC [0] ($\\uparrow$)| MMLU [0] ($\\uparrow$)| avg_ranking ($\\downarrow$) |\n|:------------------|:---------:|:------------:|:--------:|:----------:|:---------------:|:-------:|:--------:|\n| Llama-3-8B-Instruct | **0.760** (1) | **0.538** (1)   | 0.292 (3)   | 0.517 (3)      | 0.566 (3) | 0.637 (3) | 2.33         |\n| + SimPO            | 0.664 (3) | 0.457 (3)   | 0.310 (2)   | 0.639 (2)      | 0.576 (2) | **0.642** (1) | 2.17         |\n| + BPO              | 0.685 (2) | 0.461 (2)   | **0.328** (1)  | **0.652** (1)      | **0.601** (1) | 0.641 (2) | **1.50**     |\n\n&nbsp;\n\n**[A1]** Shao, Zhihong, et al. \"Deepseekmath: Pushing the limits of mathematical reasoning in open language models.\" arXiv preprint arXiv:2402.03300 (2024).\n\n**[A2]** Hong, Jiwoo, Noah Lee, and James Thorne. \"Orpo: Monolithic preference optimization without reference model.\" arXiv preprint arXiv:2403.07691 (2024).\n\n&nbsp;\n\n---\n\n### **Weakness 4.**\n\n*The writing could be further improved. For instance, in Section 3.1, terms like “concrete score” and “completeness property” are introduced without prior definition or context, which may confuse readers unfamiliar with the related literature.*\n\n&nbsp;\n\n### **Response to Weakness 4.**\n\nWe thank the reviewer for pointing out the lack of clarity regarding some terminology. We acknowledge that terms like *concrete score* and *completeness property* may be familiar in the likelihood ratio estimation literature, but could be unfamiliar to readers in preference optimization. We also agree that simply citing **[33]**, where these concepts were introduced (lines 115-116 of the manuscript), may not be sufficient to explain them clearly.\n\nThe concrete score is defined as the ratio of probability masses between two different states (e.g., $y_w$ and $y_l$) of a discrete random variable. It can also be represented as a vector of ratios between a specific state $y$ and all possible $N$ states, as shown below:\n\n- $C_{\\theta}(y) = \\left[\\frac{\\pi_\\theta (y_i)}{\\pi_\\theta (y)}\\right] _{i=1}^N  -1$.\n\nThe completeness property states that once the function $C_\\theta$ is determined, the value of $\\pi_\\theta(y)$ are uniquely determined as: \n- $\\pi_{\\theta}(y) = \\frac{1}{\\Sigma_{i} {C_{\\theta}(y)+1}}$.\n\nTherefore, matching the pairwise ratios between different states to the target ratios guarantees proper distribution matching. (This is stated as Theorem 1 in **[33]**.) We will provide a more detailed explanation of this preliminary in Section 2.2 of the camera-ready version. \n\n&nbsp;\n\n**[33]** Chenlin Meng, Kristy Choi, Jiaming Song, and Stefano Ermon. Concrete score matching: Generalized score matching for discrete data. Advances in Neural Information Processing Systems, 35:34532–34545, 2022.\n\n&nbsp;\n\n---\n\nWe sincerely thank the reviewer again for the thoughtful and constructive feedback, and we would be happy to further clarify any remaining questions."}, {"review_id": "bC2GO5i2CW", "reviewer": "Reviewer_qCve", "summary": "In this paper, the authors introduce Bregman Preference optimization, a generalized form of DPO that modifies the standard RLHF objective to use a Bregman divergence term (a well-studied concept in information geometry) in place of KL-divergence (note that this follows other recent approaches, such as Alfano, et al. 2024. I would ask the authors to consider citing this). Very interestingly, they show from first principles (Eq. 7) how the optimal policy in RLHF can be expressed as a matching problem between the two data ratios in Eq. 8 (Theorem 2.) (their final loss $\\mathcal{L}_{\\text{BPO}}$ in Eq. 10 is a tractable variant of Eq. 9 based directly on Theorem 2 with the equivalence property provided in Theorem 3). A particular advantage of this two ratio formulation is that it avoids needing an expensive partition function as in Eq. 3 (while I find this to be a very interesting reformulation, it’s not clear whether this is a strong argument given that such a partition cancels out in the standard DPO. I would like to see the authors address this point). \n\nDPO and other probabilistic variants of (f-DPO) follow as special cases of $\\mathcal{L}_{\\text{BPO}}$ by modifying the convex function `h` in the Bergman equation (Eq. 6) (the full derivation for DPO is provided in the Appendix A.5. I would encourage the authors to consider putting this in the main text, since it illuminates several misunderstands I had earlier in the paper that are not fully addressed by the form given in Table 2). They further derive several novel losses by further modifying `h` as shown in Table 2. To arrive at their main loss, they discuss (Sec. 3.3) issues wrt the gradient scaling behavior of these new losses and devise a gradient technique (SBA) to make the gradient scaling follow more closely that of DPO under the logistic loss. \n\nMuch of the experiments are conventional and compare systematically against DPO and f-DPO (using HH), as well as a wider range of DPO variants (using ultrafeedback and closely following the setup of SimPO). For HH, the main results are shown in Table 3, with their SBA variant showing large win-rates over the preferred outputs and the SFT-trained model as well as improvements over the diversity of the generation (I have some questions about the setup here that I include below). The main results for DPO variants are when in Table 6, with their model obtaining the highest aggregate win-rate of all approaches, including several SOTA approaches (based on this, the authors claim that their approach achieves a new SOTA for Llama-3-8B models).\n\n[Alfano et al. 2024 Meta-learning objectives for Preference Optimization, Arxiv 2411.06568]", "questions": "--  Not being familiar with the \"predictive entropy\" metric used for measuring diversity in the first experiments, it's not clear how it works. Can you provide a clear intuition for what this is measuring and add it to the text? \n\n-- I'm wondering what the motivation was for using HH given that it is known to contain significant (>20%) noise. Relatedly, why is computing the win-rate against the preferred output sensible in this case (e.g., as compared with comparing directly against vanilla DPO)? \n\n-- Given that the partition function in Eq. 3 and standard formulation of RHLF cancels out in DPO, why is it so attractive that it's avoided in your formulation? A discussion of this would be helpful. \n\n-- The numbers from Table 6 are repeated from the SimPO paper/repo and not fully reproduced, is this correct? Do you imagine that differences in your training setting might yield significantly different results? If so, this would be helpful to identify as a clear limitation.", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## strengths\n\n-- A very novel and interesting reformulation and generalization of DPO-style objectives from first principles via ratio matching and   Bregman divergence. This approach opens the doors to a wide range of novel DPO variants that will likely inform many follow up studies in this area. \n\n-- A comprehensive evaluation of their new losses against DPO and probabilistic variants such as f-DPO, as well as other DPO variants. In both cases, their new approach yields significant improvements, which shows the empirical utility of their formulation. \n\n-- The paper is well written and easy to follow in spite of its rather technical content. \n\n## weaknesses \n\n(I note that all weaknesses I note below are, to my mind, of a minor nature.)\n\n--  While their experiments are conventional, they are limited in scale (llama-8B). The claim of SOTA, while carefully qualified in the text, must be taken with this in mind. \n\n-- The motivations for experimenting with HH are not entirely clear. See below for specific questions.\n\n-- This work is not the first to use Bregman divergence, as noted above. I would request that the authors to address this point directly and detail any important differences with this related work.", "was_revised": true, "final_justification": "My main concerns were addressed during the rebuttal. After this and carefully considering the feedback of other reviewers, I decided to keep my score high.", "rebuttal": "We sincerely appreciate the reviewer **qCve** for the detailed feedback and for acknowledging that our work is **very novel**, shows **empirical improvement**, and is **well-written**. We also thank the reviewer for noting that the concerns raised are of a minor nature. Below, we address each of the comments in turn.\n\n&nbsp;\n\n---\n\n### **Weakness 1.**\n\n*While their experiments are conventional, they are limited in scale (llama-8B). The claim of SOTA, while carefully qualified in the text, must be taken with this in mind.*\n\n&nbsp;\n\n### **Response to Weakness 1.**\n\nWe agree that our experiments are limited to the LLaMA-3-8B scale due to resource constraints. As stated in Lines 279–280 of the manuscript, we have carefully qualified our claim within this scope: *“To the best of our knowledge, BPO achieves state-of-the-art LC performance among preference-optimized models based on LLaMA-3-8B-Instruct.”* \n\nAs the reviewer also noted, our intention is not to claim universal SOTA, but rather to demonstrate strong performance under comparable conditions. We agree that it would be more appropriate to tone down the expression from “SOTA” to “outperforms baselines,” and we will make this revision in the camera-ready version.\n\n&nbsp;\n\n---\n\n### **Weakness 2.**\n\n*The motivations for experimenting with HH are not entirely clear. See below for specific questions. // I'm wondering what the motivation was for using HH given that it is known to contain significant (>20%) noise. Relatedly, why is computing the win-rate against the preferred output sensible in this case (e.g., as compared with comparing directly against vanilla DPO)?*\n\n&nbsp;\n\n### **Response to Weakness 2.**\n\nWe conducted HH experiments to compare different instances of DPO, f-DPO, f-PO, and BPO, selecting HH because it was used in the original DPO, f-DPO, and f-PO papers. As discussed in lines 211–216 of the manuscript, we observed clear and interpretable trends in the distributional behavior of f-PO and f-DPO (e.g., mode-seeking vs. mode-covering). Therefore, we considered HH a suitable benchmark, at least for comparisons involving f-PO and f-DPO. Moreover, a dataset with some noise can be a useful indicator of robustness in practical settings. Results on other datasets, such as TL;DR and UltraFeedback, are also included in the main paper.\n\nEven if the preferred output used for evaluation is noisy, it is fixed across the evaluation of each preference optimization method, so relative comparisons on this fixed set are still meaningful. We highlight that Table 3 includes comparisons against SFT responses as well.  In addition, we report experiments that directly measure win rates between SBA and other baseline methods on HH using Pythia-2.8B, summarized in the table below.\n\nWe observe a clear trend: the stronger the baseline in the original comparison setting, the harder it is for SBA to outperform it. Still, SBA wins more than 50% of the time against all methods, indicating that SBA consistently outperforms the baselines. In the direct comparison with DPO, as referenced in the review, our method achieved a 56.5% win rate.\n\n| Anthropic-HH | $f$-PO (JS) | $f$-PO (RKL) | $f$-DPO ($\\chi^2$)|$f$-DPO (FKL)|DPO|\n|:-|:-:|:-:|:-:|:-:|:-:|\n|BPO (SBA)|50.5%|57.5%|52.0%|73.0%|56.5%|\n\n&nbsp;\n\n---\n\n### **Weakness 3.**\n\n*This work is not the first to use Bregman divergence, as noted above. I would request that the authors to address this point directly and detail any important differences with this related work.*\n\n[Alfano et al. 2024 Meta-learning objectives for Preference Optimization, Arxiv 2411.06568]\n\n&nbsp;\n\n### **Response to Weakness 3.**\n\nThank you for sharing the related preprint. The main contribution of our paper lies in reformulating the preference optimization loss from a likelihood ratio estimation perspective, aiming to achieve simplicity, optimality, and generality. In this formulation, Bregman divergence serves a technical purpose to support this generalization. As a result, our use of Bregman divergence differs structurally from Alfano et al. (2024).\n\nWe frame the full distribution matching between $\\pi_{\\theta}$ and $\\pi_{\\theta^*}$ as a ratio matching problem, resulting in the objective: \n- $D_h (R _{data}||R _{\\theta})$.\n\nIn contrast, Alfano et al. (2024) use Bregman divergence to replace the KL divergence between the policy and the reference distribution in RLHF regularization: \n- $-E_{\\pi_{\\theta}(y|x)}[r_{\\phi*}(x, y)] + \\beta D_h(\\pi_{\\theta}(y|x)||\\pi_{ref}(y|x))$.\n\nWhen only the regularization term is modified, as in their case, the resulting optimal policy may deviate from that of DPO (e.g., in f-DPO), and such formulations can offer less control over the overall training behavior compared to methods that extend the full objective.\n\nMoreover, while we prioritize simplicity, Alfano et al. (2024) focus on meta-learning by introducing a learnable function $h$, which increases training complexity. Therefore, our motivation, usage, and implementation of Bregman divergence are fundamentally different.\n\nWe will briefly discuss this in the related work section of the main paper and provide technical details in Appendix B of the camera-ready version.\n\n&nbsp;\n\n---\n\n### **Question 4.**\n\n*Not being familiar with the \"predictive entropy\" metric used for measuring diversity in the first experiments, it's not clear how it works. Can you provide a clear intuition for what this is measuring and add it to the text?*\n\n&nbsp;\n\n### **Response to Question 4.**\n\nWe acknowledge that our explanation may have been insufficient, as the main manuscript cites previous works **[62, 54]** in line 209 that used the same predictive entropy metric, without providing further detail. Line 552 of Section C.4 in the supplementary material provides the exact reference for our actual implementation.\n\nPredictive entropy measures token-level diversity across generated sequences. Intuitively, it captures the average uncertainty of the model’s next-token prediction across different generations.\n\nFormally, the metric is computed by sampling $N$ sentences from the policy and measuring the entropy at each generation step $k$:\n\n- $\\frac{1}{N} \\sum_{i=1}^{N} \\frac{1}{|y^{(i)}|} \\sum_{k=1}^{|y^{(i)}|} H\\left( \\pi_\\theta(\\cdot \\mid y^{(i)}_{<k}, x) \\right)$\n\n- $= -\\frac{1}{N} \\sum _{i=1}^{N} \\frac{1}{|y^{(i)}|} \\sum _{k=1}^{|y^{(i)}|} \\sum _{y_k^{(i)}=1}^{V} \\pi _\\theta\\left( y_k^{(i)} \\mid y^{(i)} _{<k}, x \\right) \\log \\pi _\\theta\\left( y_k^{(i)} \\mid y^{(i)} _{<k}, x \\right)$.\n\nWe will incorporate this explanation of the metric at line 206 in the main text for the camera-ready version.\n\n&nbsp;\n\n---\n\n### **Question 5.**\n\n*Given that the partition function in Eq. 3 and standard formulation of RHLF cancels out in DPO, why is it so attractive that it's avoided in your formulation? A discussion of this would be helpful.*\n\n&nbsp;\n\n### **Response to Question 5.**\n\nAs noted in lines 83–85 of the manuscript, policy models often converge to sub-optimal points due to limited capacity and imperfect optimization. Even with the same theoretical optimum, the final solution can depend on the shape of the loss function. Thus, extending the loss function can potentially yield diverse solutions and better performance, as is commonly pursued in loss generalization research.\n\nWe analyzed different $h$ functions both theoretically and empirically via Proposition 4, Section 3.3, and Figure 5, showing that extending beyond DPO allows us to find instances with higher win rates and greater diversity.\n\nTable 1 of the manuscript summarizes the benefits of BPO as optimality preservation, simplicity, and generality for multiple objectives. Here, generality can be seen as a key difference between the DPO and BPO frameworks. Moreover, Table 1 clearly highlights the distinction between prior baselines ($f$-PO, $f$-DPO), which lacked optimality or simplicity in their generalization, and BPO, which satisfies all desiderata.\n\n&nbsp;\n\n---\n\n### **Question 6.**\n\n*The numbers from Table 6 are repeated from the SimPO paper/repo and not fully reproduced, is this correct? Do you imagine that differences in your training setting might yield significantly different results? If so, this would be helpful to identify as a clear limitation.*\n\n&nbsp;\n\n### **Response to Question 6.**\n\nWe faithfully followed the training and evaluation settings provided in the SimPO repository. In particular, we used the same versions (e.g., `alpaca-eval==0.6.2` and `vllm==0.5.4`) and decoding parameters, which are known to significantly impact SimPO's regeneration results, as we already described in our supplementary material (Lines 538–540).\n\nIn our initial experiments, we closely reproduced the reported performance of SimPO as shown in table below. Based on this, we deemed it reasonable to reuse the baseline results from the original SimPO paper, and we clearly stated this in Lines 261–263 of the manuscript: *“Baseline results are taken from SimPO and f-PO.”* We acknowledge that fully reproducing all baselines would be ideal. In the camera-ready version, we will clarify the origin of all reported numbers more explicitly in the caption of Table 6.\n| Model                              | AE-LC ($\\uparrow$) | AE-WR ($\\uparrow$) |\n|--|--|--|\n| Mistral-7B-Base-SFT-SimPO         | 21.5          | 20.8          |\n| → Our reproduction                     | 20.9         | 20.4         |\n||||\n| Llama-3-8B-Instruct-SimPO-v2      | 53.7          | 47.5          |\n| → Our reproduction                       | 53.1         | 46.8         |\n\n&nbsp;\n\n---\n\n### **Suggestion 7.**\n\n*Adding full derivation for BPO recover DPO (currently in Appendix A.5.) in the main text for understanding.*\n\n&nbsp;\n\n### **Response to Suggestion 7.**\n\nYes, in the camera-ready version, we will clearly explain in the main text how a specific choice of the function $h$ in BPO recovers the standard DPO formulation.\n\n\n&nbsp;\n\n---\n\nWe appreciate the reviewer’s feedback and are happy to clarify further if needed."}, {"review_id": "Om7VdghLhd", "reviewer": "Reviewer_5KtK", "summary": "The authors propose a novel preference optimization algorithm designed around the bregman divergence. They formulate the preference optimization problem using the Bregman ratio matching framework (BPO), they show that this framework theoretically recovers the optimal policy when the preferences are modelled by the Bradley Terry model. The authors compare BPO to relevant prior literature and conduct experiments on preference optimization baselines and benchmarks exploring the practical properties of their objective.", "questions": "**Questions**\n\n- The proof of proposition 1 is built around the Bradley Terry model, this model has some flaws for example it cannot model transitive preferences, how does this affect the objective you have designed?\n\n- Please respond to the points raised in the weaknesses around clarity in the experiment section, what are the exact nature of the results in Table 3?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- The paper proposes, to the best of my knowledge, a novel preference optimization algorithm. The algorithm is theoretically motivated as a generalization of DPO and the authors prove its optimality.\n- The paper proposes a variety of different ways this generalization can be used, for example they derive a novel objective but also investigate how existing methods can be adapted using their method (Section 3.4 and Section 4.1.1).\n- The work runs extensive experiments to empirically validate their claims.\n- Their method outperforms a variety of relevant baselines approaches on standard baselines.\n\n**Weaknesses**\n\nI broadly like the paper and thus recommend accept, however I think the paper could be improved in the following ways:\n\nClarity in the writing: \n\n- I found some aspects of the paper difficult to follow specifically the experiment section, in Table 3 it is very unclear what the coloured superscript results are, in line 219 the authors reference these results directly as a ‘win-rate drop’ but it is unclear as to the exact measurement carried out here.\n- A variety of results reference broader literature e.g. a key part of the author’s argument for using the data ratio relies upon the concrete scores and it’s completeness property further detail on these concepts provided in the appendices would be useful for rigorously understanding the argument.\n\nDiversity metrics:\n\n- It might have been nice to use a variant of the pass@k metric to analyse how the diversity and win-rate vary across BPO and different baselines.", "was_revised": true, "final_justification": "I stand by my initial score of 5, and recommend the paper is accepted. The authors have responded to the weaknesses and questions I raised, and conducted additional experiments in response to my recommendations.", "rebuttal": "We sincerely thank the reviewer **5KtK** for the helpful feedback and for highlighting the **novelty**, **theoretical foundation**, **versatility with DPO variants**, and thorough **empirical validation** of our approach. We also appreciate that the reviewer generally finds the paper favorable and recommends acceptance. We address the specific suggestions in detail below.\n\n&nbsp;\n\n---\n\n### **Weakness 1.**\n\n*I found some aspects of the paper difficult to follow specifically the experiment section, in Table 3 it is very unclear what the coloured superscript results are, in line 219 the authors reference these results directly as a ‘win-rate drop’ but it is unclear as to the exact measurement carried out here. // Please respond to the points raised in the weaknesses around clarity in the experiment section, what are the exact nature of the results in Table 3?*\n\n&nbsp;\n\n### **Response to Weakness 1.**\n\nWe agree that the explanation of the superscripts was insufficient. Since f-DPO, f-PO, and BPO are all extensions of DPO, we used the superscripts to indicate the relative performance gain (red) or loss (blue) compared to DPO as:\n\n\n- $ \\text{Superscript}$ (%) = $\\frac{\\text{Performance} _{\\text{Method}} - \\text{Performance} _{\\text{DPO}}}{\\text{Performance} _{\\text{DPO}}} \\times 100$ % \n\nFor example, the win rate of DPO (vs Preferred) is $48.5$, and the win rate of f-DPO (FKL) (vs Preferred) is $34.5$. The superscript is calculated as $\\frac{34.5 - 48.5}{48.5}\\times 100$ % = $-28.9$%, and we referred to it as a “win rate drop” in line 219. In the case of entropy, the value for DPO is $2.801$, and for f-DPO (FKL) it is $3.354$, which gives a superscript of $+19.7$%, computed as $\\frac{3.354 - 2.801}{2.801}\\times 100$ %. For BLEU, where lower values indicate better performance, the sign of the superscript is reversed accordingly.\n\nWe basically adopted this notion from recent works such as **[56]**, where it is used to indicate how much a method improves over DPO. In the camera-ready version, we will explicitly describe this calculation method in the caption of Table 3. \n\n&nbsp;\n\n**[56]** Junkang Wu, Yuexiang Xie, Zhengyi Yang, Jiancan Wu, Jinyang Gao, Bolin Ding, Xiang Wang, and Xiangnan He. β-dpo: Direct preference optimization with dynamic β. Advances in Neural Information Processing Systems, 37:129944–129966, 2024.\n\n&nbsp;\n\n---\n\n### **Weakness 2.**\n\n*A variety of results reference broader literature e.g. a key part of the author’s argument for using the data ratio relies upon the concrete scores and it’s completeness property further detail on these concepts provided in the appendices would be useful for rigorously understanding the argument.*\n\n&nbsp;\n\n### **Response to Weakness 2.**\n\nWe agree that the connection between ratio matching and distribution matching in our paper relies on the concept of *concrete scores* and their *completeness property*. While this idea may be familiar to the likelihood ratio estimation community **[31, 33]**, we acknowledge that citing the original paper **[31]**, where this notion of *concrete scores* was first introduced, in lines 115–116 of the main manuscript may not be sufficient for readers in the preference optimization community.\n\nThe concrete score is defined as the ratio of probability masses between two different states (e.g., $y_w$ and $y_l$) of a discrete random variable. It can also be represented as a vector of ratios between a specific state $y$ and all possible $N$ states, as shown below:\n\n- $C_{\\theta}(y) = \\left[\\frac{\\pi_\\theta (y_i)}{\\pi_\\theta (y)}\\right] _{i=1}^N  -1.$\n\nThe completeness property indicates that once the function $C_\\theta$ is defined, the value of $\\pi_\\theta(y)$ are uniquely determined as: \n\n- $\\pi_{\\theta}(y) = \\frac{1}{\\Sigma_{i} {C_{\\theta}(y)+1}}.$ \n\nTherefore, matching the pairwise ratios between different states to the target ratios guarantees proper distribution matching. (This is described as Theorem 1 in **[33]**.)\n\nThis concept is used in the proof of Theorem 2 in our paper, specifically in lines 898–900 of the main manuscript. The corresponding part can be written without relying on prior literature as follows. Once learned policy $\\pi _\\hat{\\theta}$ and target optimal policy $\\pi _{\\theta*}$ has following relationship for all $(x, y, y_i)$:\n\n - $\\frac{\\pi_\\hat{\\theta} (y_i|x)}{ \\pi_\\hat{\\theta} (y|x)} = \\frac{\\pi_ {\\theta*} (y_i|x)}{ \\pi _{\\theta*} (y|x)},$ \n\nThen, for any $(x, y)$, the probability of $\\pi_\\hat{\\theta}$ and $\\pi_ {\\theta*}$ are identical as: \n\n- $ \\pi_\\hat{\\theta} (y|x) = \\frac{1}{\\Sigma_{i} { \\frac{\\pi_\\hat{\\theta} (y_i|x)}{ \\pi_\\hat{\\theta} (y|x)}}} = \\frac{1}{\\Sigma_{i} { \\frac{\\pi_{\\theta*} (y_i|x)}{ \\pi_{\\theta*} (y|x)}}} = \\pi_{\\theta*} (y|x ).$\n\nThis is why ratio matching leads to distribution matching in our setting. We will add background on the concrete score in Section 2.2 and add the relevant proof in lines 898–900 of Theorem 2, so that the paper is more self-contained and accessible to readers unfamiliar with this literature.\n\n&nbsp;\n\n**[31]** Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. In Forty-first International Conference on Machine Learning, 2024.\n\n**[33]** Chenlin Meng, Kristy Choi, Jiaming Song, and Stefano Ermon. Concrete score matching: Generalized score matching for discrete data. Advances in Neural Information Processing Systems, 35:34532–34545, 2022.\n\n&nbsp;\n\n---\n\n### **Weakness 3.**\n\n*It might have been nice to use a variant of the pass@k metric to analyse how the diversity and win-rate vary across BPO and different baselines.*\n\n&nbsp;\n\n### **Response to Weakness 3.**\n\nThe metric pass@k considers a generation successful if at least one of the $k$ generated outputs is correct. It is widely used for evaluating both the quality and diversity of a model's outputs. More precisely, it is defined as follows **[C1]**:\n\n- $\\text{pass@k} = 1 - \\frac{\\binom{n - c}{k}}{\\binom{n}{k}},$\n\nwhere $n$ is the number of generated samples per prompt, $k$ is the number of samples selected to compute pass@k, and $c$ is the number of \"correct\" outputs among the $n$ samples.\n\nFor the experiment corresponding to Table 3, we computed the metric for BPO and baselines: the basic ones, such as SFT and DPO, f-DPO (FKL), which demonstrated high diversity, and f-DPO ($\\chi^2$), which showed a high win rate. (Due to high API costs, we provide a limited number of baselines for this.) We randomly sampled 96 prompts from the test set and set $n=10$. Outputs that beat $y_w$ in GPT-based evaluation were considered “correct.”\n\n\n| Method            | Winrate | pass@2 | pass@3 | pass@4 | pass@5 | pass@6 |\n|------------------|---------|--------|--------|--------|--------|--------|\n| SFT              | 0.335   | 0.441  | 0.512  | 0.558  | 0.590  | 0.614 |\n| DPO              | 0.485   | 0.623  | 0.688  | 0.729  | 0.757  | 0.777 |\n| f-DPO (FKL)        | 0.345   | 0.498  | 0.579  | 0.631  | 0.668  | 0.696 |\n| f-DPO ($\\chi^2$) | 0.535   | 0.617  | 0.680  | 0.721  | 0.750  | 0.772 |\n| BPO (SBA)             | **0.570** | **0.673** | **0.731** | **0.764** | **0.786** | **0.801** |\n\n&nbsp;\n\nBPO outperformed all baselines at every value of $k$, which we attribute to its strong performance in both sample fidelity and diversity.\n\n&nbsp;\n\n**[C1]** Chen, Mark, et al. (2021). Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.\n\n&nbsp;\n\n---\n\n### **Question 4.**\n\n*The proof of proposition 1 is built around the Bradley Terry model, this model has some flaws for example it cannot model transitive preferences, how does this affect the objective you have designed?*\n\n&nbsp;\n\n### **Response to Question 4.**\n\nWe formulated our research problem as finding a generalized objective function that preserves the optimality, simplicity, and generality of the original DPO objective. Because of this, our method naturally follows the same target optimal policy as DPO and inherits the Bradley-Terry (BT) modeling assumption.\n\nAs the reviewer pointed out, BT modeling cannot represent cyclic preferences (e.g., A ≻ B ≻ C ≻ A), which is a known limitation. Addressing such non-transitive structures has been explored as a separate line of recent work (e.g., **[C2]**). We believe the BT modeling still performs effectively in practice. At the same time, if more expressive preference models beyond BT can also be formulated from a likelihood ratio estimation perspective, this opens up a promising future direction to extend BPO.\n\n&nbsp;\n\n**[C2]** Zhang, Yifan and Zhang, Ge and Wu, Yue and Xu, Kangping and Gu, Quanquan. Beyond bradley-terry models: A general preference model for language model alignment. In Forty-second International Conference on Machine Learning, 2025.\n\n&nbsp;\n\n---\n\nWe sincerely appreciate the reviewer’s insightful comments, and we remain happy to clarify any additional questions or concerns that may arise."}, {"review_id": "Z9Ql10twcd", "reviewer": "Reviewer_K9Vq", "summary": "This paper introduces Bregman Preference Optimization (BPO), a generalized framework for aligning large language models (LLMs) with human preferences. BPO reformulates Direct Preference Optimization (DPO) from a likelihood ratio estimation perspective, enabling policy models to match target distributions without relying on explicit reward models or partition functions. Experiments on dialogue generation (Anthropic-HH) and summarization (TL;DR) show BPO instances (especially SBA) outperform DPO and its variants (f-DPO, f-PO) in win rate and diversity. Applied to Llama-3-8B, BPO achieves SOTA length-controlled win rate (55.9%) on AlpacaEval2.", "questions": "What is the benefit of formulating policy optimization as density ratio estimation given sufficient model capacity?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strength\n\n* Theoretical guarantee: BPO expresses the optimal policy via density ratios (Proposition 1), guaranteeing optimality under Bregman divergence minimization (Theorems 2–3).\n* Practical implementation: BPO avoids Monte-Carlo estimation of the partition function and the reward model as in f-PO.\n* Empirical performance: Experiments cover diverse tasks (dialogue, summarization) and model scales (Pythia-2.8B to Llama-3-8B). BPO consistently improves win rates and entropy over DPO (Table 3, Figure 1) and achieves SOTA on AlpacaEval2 (Table 6). \n\n\n## Weakness\n\nOne main motivation of BPO is to address the additional complexity introduced in f-PO. However, the theoretical optimality is based on the model having sufficient capacity, which is a vague expression. When the model is capable of completely capture the complexity of the preference data, different optimization criteria should not vary in terms of the final performance after convergence.", "was_revised": true, "final_justification": "Thanks for the author's clarification of my question. I decide to keep my score.", "rebuttal": "We greatly appreciate reviewer **K9Vq**’s constructive feedback and recognition of the paper’s **theoretical guarantee**, **implementation simplicity**, and strong **empirical results**. We address the reviewer’s comments below.\n\n&nbsp;\n\n---\n\n### **Weakness**.\n\n*One main motivation of BPO is to address the additional complexity introduced in f-PO. However, the theoretical optimality is based on the model having sufficient capacity, which is a vague expression. When the model is capable of completely capturing the complexity of the preference data, different optimization criteria should not vary in terms of the final performance after convergence.  // What is the benefit of formulating policy optimization as density ratio estimation given sufficient model capacity?*\n\n&nbsp;\n\n### **Response**.\n\nThe optimality in Theorem 2 relies on the following key assumptions:\n\n- **A1.** The policy model $\\pi_{\\theta}$ has sufficient capacity to represent arbitrary distributions.  \n- **A2.** The optimization is perfect. (i.e., it satisfies both **A2.1** and **A2.2**.) \n  - **A2.1.** The computation of the objective function is exact.  \n  - **A2.2.** $\\pi_{\\theta}$ exactly minimizes the objective function. (This corresponds to the ‘argmin’ in Theorem 2.)\n\nThese assumptions (**A1, A2.1, A2.2**) should be considered separately. Even if sufficient model capacity in **A1** is assumed to be satisfied, different outcomes can still occur depending on how well **A2** is satisfied.\n\n&nbsp;\n\n**[Comparison of complexity and theoretical benefits to f-PO]**\n\nThe claim that BPO has lower complexity than f-PO specifically refers to the reduced burden of computing the exact objective function required to satisfy **A2.1**. Leveraging Theorems 2 and 3, we derive a simple closed-form expression (Eq. 10 of the manuscript) that enables $\\pi_\\theta$ to converge to the optimal policy.\n\nIn contrast, f-PO requires a perfectly trained reward model $r_{\\phi^*}$ and an infinite number of samples from the reference model to compute the partition function required to satisfy **A2.1**. This makes the exact objective computation significantly more complex and dependent on stronger assumptions. A key advantage of the density ratio estimation formulation in BPO is that it enables exact objective computation for **A2.1** without any additional assumptions or computational burden.\n\nHere is a summary of the requirements to satisfy **A2.1**.\n - f-PO: Perfect reward model $r_{\\phi^*}$, infinite number of samples from $\\pi_{ref}$.\n - BPO: Guaranteed by Theorems 2 and 3 with no additional assumptions required.\n\n&nbsp;\n\n**[Convergence under sufficient model capacity]**\n\nWhile sufficient model capacity ensures the ability to represent the optimal policy, achieving perfect optimization in **A2** still depends on the optimization dynamics. Different loss functions induce different gradients and optimization trajectories, and this affects the model's practical convergence point. In Eq. 11 of the manuscript, we mathematically analyze gradient behavior across various loss functions, and in Section 3.3, we present example analyses. These analyses highlight that loss design remains practically important, even with sufficient model capacity.\n\n&nbsp;\n\n---\n\nWe would like to once again thank the reviewer for their valuable feedback, and we would be happy to address any further questions or concerns."}]

Author rebuttal:

Meta-review: The paper proposes Bregman Preference Optimization, a principled generalization of DPO from a likelihood ratio estimation perspective. The formulation preserves optimality under Bregman divergence, subsumes DPO as a special case, and introduces a family of objectives that can be implemented with comparable simplicity. The authors also propose a gradient scaling variant and demonstrate consistent improvements in both win rate and output diversity across multiple datasets and models, achieving strong results relative to recent DPO/f-PO variants.

Overall, reviewers found the work to be novel, theoretically sound, and empirically validated, with particular strength in the clarity of the framework and its strong empirical gains over baselines. Concerns raised included clarity in terminology, the limited scale of experiments, evidence of alignment tax on math reasoning tasks, and the need for more direct comparisons and positioning against related work. The rebuttal provided new experiments on base models and reasoning benchmarks, clarified terminology and metrics, and contrasted BPO with prior formulations, which reviewers agreed satisfactorily addressed their concerns.

I share the reviewers’ positive assessment: the paper makes a meaningful theoretical and practical contribution to preference optimization, and the rebuttal strengthened its empirical and conceptual positioning. I recommend acceptance.
Decision: Accept (poster)