Energy-Based Preference Model Offers Better Offline Alignment than the Bradley-Terry Preference Model
Authors: Yuzhong Hong
Hanshan Zhang
Junwei Bao
Hongfei Jiang
yang song
Venue: ICML 2025 poster (2025)
Keywords: preference model
offline alignment
Primary area: deep_learning->large_language_models
TL;DR: a better alternative to the Bradley-Terry preference model

Abstract: Since the debut of DPO, it has been shown that aligning a target LLM with human preferences via the KL-constrained RLHF loss is mathematically equivalent to a special kind of reward modeling task. Concretely, the task requires: 1) using the target LLM to parameterize the reward model, and 2) tuning the reward model so that it has a 1:1 linear relationship with the true reward. However, we identify a significant issue: the DPO loss might have multiple minimizers, of which only one satisfies the required linearity condition. The problem arises from a well-known issue of the underlying Bradley-Terry preference model: it does not always have a unique maximum likelihood estimator (MLE). Consequently, the minimizer of the RLHF loss might be unattainable because it is merely one among many minimizers of the DPO loss. As a better alternative, we propose an energy-based preference model (EBM) that always has a unique MLE, inherently satisfying the linearity requirement. To showcase the practical utility of replacing BTM with our EBM in the context of offline alignment, we adapt a simple yet scalable objective function from the recent literature on fitting EBM and name it as Energy Preference Alignment (EPA). Empirically, we demonstrate that EPA consistently delivers better performance on open benchmarks compared to DPO, thereby validating the theoretical superiority of our EBM.

Energy-Based Preference Model Offers Better Offline Alignment than the Bradley-Terry Preference Model
Yuzhong Hong"
Hanshan Zhang" 22
Junwei Bao
Hongfei Jiang'
Yang Song'
Abstract
Table 1. The intrinsic evaluation based the average Pearson coefficient E [-1, 1J and the average slope-1 linear regression shows that EPA renders closer approximation the slope-1 linearity than DPO. This is consistent with the extrinsic evaluation based the Alpaca Eval 2.0 benchmark.
Since the debut of DPO, it has been shown that aligning a target L.M with human preferences via the KL-constrained RLHF loss is mathematically equivalent to special kind of reward modeling task. Concretely, the task requires: 1) using the target LLM to parameterize the reward model, and 2) tuning the reward model so that it has 1:1 linear relationship with the true reward. How- ever, we identify significant issue: the DPO loss might have multiple minimizers, of which only one satisfies the required linearity condition. The problem arises from well-known issue of the un- derlying Bradley-Terry preference model: it does not always have unique maximum likelihood es- timator (MLE). Consequently, the minimizer of the RLHF loss might be unattainable because it is merely one among many minimizers of the DPO loss. As a better alternative, we pro- pose an energy-based preference model (EBM) that always has a unique MLE, inherently satis- fying the linearity requirement. To showcase the practical utility of replacing BTM with our EBM in the context of offline alignment, we adapt a simple yet scalable objective function from the recent literature on fitting EBM and name it as Energy Preference Alignment (EPA). Empirically, we demonstrate that EPA consistently delivers bet- performance open benchmarks compared to DPO, thereby validating the theoretical superi- ority of our EBM.
PEARSON (1)
2
МЕTHOD
SLOPE- NA SLOPE-1: I: V
AE2.0 (%, 1)
LINEARITY: v LINEARITY:
DPO
0.4693
5.78
17.43 15.24
EPA (1:1:2)
0.5808
5.26
19.20 / 19.26
EPA (1:3:2)
0.5754
5.01
21.31 20.13
1. Introduction
Reinforcement Learning with Human Feedback (RLHF) (Christiano et al., 2017) has been widely used to align a large language model (L.LM) with human preference. The canonical RLHF objective (Ziegler al., 2019; Stiennon et al., 2020; Ouyang al., 2022; Perez al., 2022) is defined follows (given æ):
CRLH E [Timee (I, v)1 + 3KL (T<( mo(M]
where xx) is the target LLM (i.e., the policy) to tune, Teet (y (r) a frozen LLM initialized identically as the target LIM and Firee (2. y) reward maximize.
The PRLHF as defined above is not differentiable w.r.t 0 (Ziegler et al., 2019; Rafailov et al., 2023), hence not SGD- friendly. Luckily, it has been shown that the unique mini- mizer of CRLI can be analytically expressed (Korbak et al., 2022a). Then, Rafailov al. (2023) further reformulate the analytical minimizer as the unique solution the following of equations:
'Equal contribution 'Zuoyebang Education Technology (Bei- jing). Co., Ltd 'StepFun Technology Co., Ltd.. Correspon- dence Yuzhong Hong weaeenee..gttggmi.... shan Zhang <u5975228@mn wnw.eaan.uuuuu........ Junwei Bao <baojun- wei001 @ com...com>
To (o())>)
Han-
ro(2, ()= 8 log
Ree((y(z)
(2)
ro Time (x. v) C(x)
(2) defines ry as the log ratio reward and E3. states that there holds a slope-1 linearity between the log ratio reward and the true reward. This formulation implies that as
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Energy- -Based Preference Model Offers Better Offline Alignment
Preference Models
Vanilla Losses
RLHF Optimization
Loss Modification Tricks
(Note: ro v) 3 log Fa (t)2)) (s/lr)))
RIHF Loss
Net Differentiable
(Energy- Basedd IPM
ply any other y (2)x)
log(+)
Mexmmm Estination
exp |ntrue (a:, w)]
zoft toottmax9
Energy Discrpancy (dde B5X)
Introctable
exp |nceuo (2)
Perturbatinn (Thma3 2, 3.31 My any other [(]
EPA
Brarley-Terry
Mosmum Estinutio (Iteal bPG)
plu >~ m1(zz)
log( )
exp |Trrue Bw
5of tnax r
Σ
exp [Fteua (22, (yy
DPO
y (ouua))
Maee & plus 20/1z)
Figure 1. An illustration of the contributions of the paper. Our core argument is that an Energy-Based model (EBM) is a better alternative the Bradley- Brarley-Terry model (BTM) due to its guaranteed unique existence of maximum likelihood estimator (MLE) (which is identical to the minimizer of the RLHF loss). The advantage of our EBM comes from its intrinsic considcration the infinity in the size of the space of y|æ, whereas BTM ignores issues caused by the pair sampling distribution (p(yw, 31 (z)) in such infinite space. Henee we name EBM the Infinite Preference Model. Although approximating the MLE with our proposed EPA loss introduces inevitable error in practice, we find that is still empirically better performing than its counterpart DPO, with or without loss modification techniques presented in previous offiine alignment literatures.
long as we can find a differentiable objective function C(ro) to achieve (3) and ro is parameterized by Eq,(2), we can convert the RLHF problem into an offline supervised task. This is the approach of interest in this paper, a drastically different one from classical online RL methods such as PPO.
with DPO. The reason is that the unique existence of BTM's MLB (i.e., the minimizer of Looo) is not guaranteed without some non-trivial constraints on the structure of p(y yn (r), a well-known issue of BTM given an infinite candidate space (i.e., that of y/z)) in the literature on learning to rank (Ford, 1957; Simons & Yao, 1999; Han et al., 2020; Hendriekx et al., 2020; Bong & Rinaldo, 2022; Wu et al., 2022). It is also related to the theoretical issues around dataset coverage in the RL literature (Kakade & Langford, 2002; Munos & Szepesvári, 2008; Zhan et al., 2022). Moreover, Tang et al. (2024) have shown that when offline data are drawn from Ter (a usual practice), any pair-wiss loss will cease to correlate with PRLH when Ta deviates enough from Jet due reward maximization.
The poster child example of this offline approach is DPO (Rafailov et al., 2023):
rideal (ro)
E p(t (els)) p(u mwlx)
E
[loga(ro(z) - ro(2,
where yoe and y are two responses and yo > y means you is prefered yr given the prompt x. The ideal' DPO loss
is essentially the maximum likelihood estimation loss of the Bradley-Terry model (BTM) who posits a sigmoidal relationship between p(u > yı|z) and Time (x, Bw) (x, 31). If the maximum likelihood estimator (MLE) uniquely exists, Rafailov et al. (2023) show that the MLE will make the 1 linearity hold.
These theoretical facts motivate us to find an alternative model that always has a unique MLE that guarantees the slope-1 linearity. As illustrated in Figure and showcased by the results of a proof-of-concept experiment in Table 1, we argue that an Energy- the Infinite Preference Model (IPM) is such an alternative. Also, even though its MLE can only be estimated far from perfectly, we observe a closer approximation to the slope-1 I linearity and better overall alignment with human preference than DPO. In summary, our contributions are as follows:
model (EBM) called
However, alluded to by our framing, we argue that it is false to conclude that the slope-1 linearity (i.e., the minimizer of the RLHF loss) is guaranteed to be reached
"ideal" in the sense that the expectations in its loss function are accurately computed. In practice, they can only he approximated, which can introduce an additional error.
theoretically, we show BTM does not and IPM does
2

---

Energy-Based Preference Model Offers Better Offline Alignment
have guaranteed unique existence of its MLE, equíva- lent to the minimizer of the RLHF loss;
of theoretical justification on how they are related to the minimizer of the RLHF loss.
empirically, by only loosely estimating the MLE using contrastive loss called EPA, we achieve new state of the art of offline alignment on open benchmarks in various settings.
2.2. Fitting Discrete EBMs
To provide a theoretical background for our proposal, we give a concise review of the most related work on fitting discrete EBMs
Energy-based models (EBM) (LeCun et al., 2006) are gen- erative models that posit a Boltzmann distribution of data, i.e., p(z) x exp (-E()) where Exx) (called the energy function) is a real-valued function to lear. An EBM is called discrete when the data point æ is defined on discrete space. To fit p(z) with maximum likelihood estimation re- quires the computation of the normalizer Σ exp (-E(x) )) (called the partition function), which is intractable. There- fore, EBMs are usually learned with a tractable approxima- tion.
2. Related Work
2.1. DPO and its Recent Improvements
The first approach to avoid DPO's theoretical issue is to use non-BTMs to model data distributions. Rafailov et al. (2023) suggest the DPO's counterpart for the Plackett-Luce Model (we refer to it as DPO-PL), which is a generalized version of BTM for K-wise comparison. IPO (Azar et al., 2023) uses a different pair-wise preference model than BTM. The loss derived from that model can be interpreted as: the difference of log ratio rewards of the yw and yI regresses to a constant. However, Tang et al. (2024) show that IPO is still incapable of optimizing LRIIF. similar to DPO. Ethayarajh et al. (2024) (KTO) point out some limitations of modeling human preference with a pair-wise model. Instead, they independently model data distribution for desirable sam- ples and another one for undesirable samples. However, such data distributions do not reflect how most benchmark datasets are sampled. This could be the reason why some empirically driven studies find that KTO underperforms DPO on these benchmarks (Meng et al., 2024; Zhou et al., 2024).
The classical approach to approximate the gradient of maximum likelihood estimation by online sampling from parameterized po(x)) with MCMC (Song & Kingma, 2021). A Although they are ideally effective, it is usually difficult or expensive to do such sampling, which harms practical results. Therefore, there are also many MCMC-free meth- ods (Meng et al., 2022; Hyvärinen, 2007; Dai et al., 2020; Lazaro-Gredille et al., 2021; Eikema et al., 2022). Recently, Schröder al. (2023) have introduced the notion of energy discrepancy, whose unique global minimizer is identical to the MLF of the EMB in question. Hence, to find the MLE, one can simply minimize the energy discrepancy, which is feasible with SGD on offline data. For its simplicity, we derive EPA based on their theoretical framework.
The second approach is to tweak the DPO loss. Some loss- tweaking tricks can be effective on their own. For example, Park al. (2024) (R-DPO) introduce length penalty on the log ratio reward to make DPO less prone to the verbosity bias. Amini et al. (2024) (ODPO) add a dynamic margin between You and yt based on the intuition that some pairs have stronger or weaker desirability gaps than others. The most effective one discovered so far is on-policy weighting (WPO) (Zhou et al., 2024). Its idea is to approximate the on-policy training scenario by assigning larger weights to the loss of samples closer the current policy at each step and smaller weights to that of less closer ones. Other tricks come in combinations, For example, CPO (Xu et al., 2024) removes the reference model in the log ratio reward and add an SFT loss at the same time. ORPO (Hong et al., 2024) an improvement over CPO by adding yet another set of tricks: normalizing the policy to the token level (length normalization) and then contrasting the policy distribution with one minus itself. To separate the wheat from the chaff, Meng et al. (2024) find the most simple and effective recipe: removing the reference model, adding a margin and apply- ing length normalization, which gives rise to SimPO. The problem with applying tricks is that there is usually a lack
2.3. EBMs for RLHF
EBMs are not rare in the RLHF research. One of the re- search directions is to formulate RLHF as a Distribution Matching problem. The typical example for this approach is Distributional Policy Gradients (DPG) (Parshakova et al., 2019; Khalifa et al., 2021). However, would like to point out that our EBM is different and used for a different pur- pose. The EBM in DPG is a non-paramttric one predefined as the learning signal. Our EBM is parametric one to fit the distribution of data. The only connection between the two EBMs is that they are used to find the same optimal policy (Korbak et al., 2022a). Deng et al. (2020) uses an EBM for language modeling. Their work essentially solves the self- piay-like RLHF problem (Chen et al., 2024b). Although their EBM is also parametric, it fits the optimal policy dis- tribution. Our EBM instead fits the preference distribution. Chen et al. (2024a) proposes two methods - infoNCA and NCA, based on the same EBM as that of Deng et al. (2020), The NCA loss follows the same derivation of the loss pro- posed by Deng al. (2020) except that they parameterize
3

---

Energy-Baser Preference Model Offers Better Offline Alignment
their energy function differently. The infoNCA loss exhibits similarity our loss. However, we will show that infoNCA is just worse-performing ablation version of EPA.
Therefore, as long as we can find the MLE of the IPM pa- rameterized as so, we are guaranteed to reach the minimizer of LrliF since it is the unique solution to (2) and (3),
On a side note, the IPM has been previously introduced by other studies on RLHF for a different purpose: to theoreti- cally equate the maximization of RRLL to the variational inference of the optimal policy with पe as the prior (Korbak et al., 2022b; Yang et al., 2024). However, to the best of our knowledge, we are the first one introduce IPM not just as theoretical toy, but as tool (when parameterized by the log ratio reward) to do proper RL-free RLHF
3. IPM: Our EBM for Preference Modelling
In the first subsection, we show that an energy-based model (EBM) is guaranteed to have unique MLE which is equiv- alent to the minimizer of the RLHF objective. In the second subsection, based on a framework by Schröder et al. (2023), we describe a general strategy to approximate the MILE us- ing offline data. Using it will provide the theoretical account for our proposal in section 4.
3.2. Offline Approximation of MLE
3.1. Theoretical Guarantee
Despite the powerfulness of IPM, finding its MLE is non- trivial task. Directly finding it with the minimization of the negative log likelihood - log go (o(y)z)) is intractable hecause of the infinity in the denominator.
Given any x, it is obvious that the space of y|æ is infinitely large because can be any token sequence of unlimited length no matter how likely or unlikely it is a response to J. This infinity is problematic for BTM. For example, if there is single y that is never sampled, it is easy to refute the unique existence of BTM's MLE (see Proposition B.5). However, EBM can naturally take the infinity into account to avoid the issue. Specifically, we model a onc-to-infinite preference (v.s. BTM and the more general Plackett-Luce model only model a one-to- number preference) as follows:
There are good tractable approximations but usually with complex online training algorithms. For simplicity and scalability purposes, we choose follow Schröder al. (2023), who provide a general strategy that finds the optimal EBM by simple SGD with offline training data. The strategy is based on two theorems formally adapted for our purpose as follows.
Theorem 3.2. For any random variable Z with the condi- tional variance Var(Y [6] being positive, the global unique minimizer r (z. y) of the functional Energy Discrepancy (ED) defined as follows the optimal IPM (i.e., p(y] & exp Jr" (x1(x,, v)).
p(y|=)= = p(vy # y: y > lx)
(5)
Namely, )( is the probability that candidate is pre- ferred over all other candidates. Under mild assumptions (Assumptions B.I and B.2) that make an EBM applicable, we define the Infinite Preference Model (IPM) to be the one that posits that p(y| (x) is Boltzmann distribution induced by the corresponding true reward (i.e., using- -Time (2r, v) as the energy function):
ED EDe.swoleehaael E
[log Eyp(()l) exp (r(2, (') r(x, w)ll (8) p(al))
Theorem 3.3. For any random vuriable Z whose back- ward and forward transition probabilities, from Y solve the equation Σyp(2ly)f = Σyp(ylz) f f() for an arbitrary I. the estimation error of the following statistic estimate of ED, ((.p()))))p( [r] vanishes almost surely when N + 0o and M + 20.
exp[rime (I, w)] p(yl=)= E2 exp(rime (I. y')
(6)
IPM is better alternative to BTM because of the following theorem (see Appendix B for proof).
Theorem 3.1. when we parameterize the IPM as follows, the unique existence of the IPM's MLE is guaranteed and will be reached if and only if the slope-1 linearity (i.e., (3)) holds between the log ratio reward and the true reward.
[[0[z] =
1
Mi log(1 + 2i5h exp (ro(x, ve(x,v')) (9)
N
Mog((M)
qo ge(v)=)= = exp|ra(x, y)1
where { are samples from p(y]2), fotigar from p(y)zo), and zg a single sample from p(zly').
2ng exp(re(x,
where re is defined as in (2).
A one-sentence interpretation of the above theorems is: if we have a particular kind of negative sampling strategy by perturbing observed preferred samples, we will learn the optimal IPM by minimizing a contrastive loss between the
"One should not confuse p(ylz) with x (y]z) although both of them are distributions over y given p(y]z)) is how likely humans would rate a y as the hest whereas (y)az) measures how likely y is to generated.
4

---

Seergy-Based Preference Model Offers Better Offline Alignment
negatives and observed positives. Therefore, when the IPM is parameterized by the log ratio reward, we will find the exact minimizer of PRLI with this loss function.
draw the negatives from perturbation source that is both informative of the positives and of high variance at the same time. For the informativeness, we consider strong negatives yt because they are semantically close to ye- For the high variance, we consider weak nega- tives such as mismatched responses. We will show the effectiveness of such choice with ablation experiments in section 5.
Note that the property of the negative sampling source Z as described in Theorem 3.3 is just sufficient condition as opposed to a necessary one. This leaves room for empirical discovery of better negative sampling strategies. rule of thumb as suggested by Schröder al. (2023) is that 2 has to be informative of Y and of high conditional variance at the same time. This provides the intuition of our proposal in section 4.
4.2. General Definition
Note that the number of strong negatives in Eq.(1 is lim- ited because of the given pair-wise data. This is not ideal for the approximation of IPM's MLE because the number of negatives should be large enough reduce the approxi- mation error (Theorem 3.3). Therefore, we generalize our definition of EPA to circumstances where we can have ac- cess to more strong negatives (i.e., each Vu: is accompanied by (M,, 31/3 ...) instead of just one y)). This is practi- cally feasible because we can always sample less desirable responses from some LLM.
4. EPA: A Practical Approximation
To introduce our loss, we first write the ideal loss in (9) in an equivalent form by removing the constant lMg and moving minus sign out of the logarithm:
C(0
exp(ro(x, y')l
(10)
N N log (xplro(z,)')) + Σ. explro(z, (,,.y"".) y²
Hence, we define loss in a more general form follows:
Now we propose our loss function in this negative log soft- max form with specific negative sampling strategy in mind.
Agenenal EPA
explra v(,)]
Σ log explro vtal + + explra(at of +E exppretart vD))
4.1. Narrow Definition
B
kEAn
For the most classical setup, we assume we only have access to pair-wise preference data. In this setting, our loss for each mini-batch of B samples (((x), yus vt)} B is defined as follows:
(12)
where Ia contains Nstrong indices of available strong neg- atives; Ink contains N Weak indices of weak negatives; y! can be either dig or some yig (j ن i).
CEPA
explro(at, vil)l ! Σ log Σ (explro(x [ + explro(x', of
B
(11)
(E(j)uZa
4.3. Gradient Analysis
where Iun is a non-empty random subset of (1, 2, 1, i+1, 1, ..., B}. introducing negative samples that are mismatched responses originally sampled for other prompts. Its size [Twx| = N / /2e E (0, B - 1] is hyperparameter. Note that our loss without Iuk reduces to the DPO loss. We justify our choice of positives and negatives in EPA as follows:
Using the chain rule, one can easily find the gradient of the EPA loss (the general one) as follows:
general
o EPA
8
Σ
sig (Vo Tog x²) Vo log To (vi, (z²))
B
KETx
(13)
1. Why is you a good approximation of a positive sam- ple from ply xx) For Yor in the dataset, may not be the best y, but there is only finite number of po- tentially possible better ones according to Assumption B. Also, since we know it is preferred over yı and infinitely many other arbitrary token sequences, it is a good approximation of a y that is preferred over all other samples up to a small error.
strong contrist
(Voo (Vo To (o( - Vo log Th (v2l(==))
JE
weak COMErEST
where si and 8 are the softmax-ed values of the strong negative log ratio rewards and the weak negative log ratio rewards, respectively. They control the magnitude of the strong and weak contrast. When there is no weak contrast, the gradient reduces to that of the DPO loss if there is only one strong negative. Therefore, one can interpret the weak
2. Why use both y and mismatched responses as neg- atives? As stated at the end of section 3, we want to
5

---

Bnergy-Basce Preference Model Offers Better Offline Alignment
contrast as a regularization added DPO to prevent 0 from moving to direction that undesirably increases the likelihood of weak negatives.
our core argument about EBM's superiority over BTM. The tricks in consideration are those used in SimPO, R-DPO, CPO and WPO (Details in Table 7 in the Appendix C).
5. Experiments
5.1.4 IMPLEMENTATION
We use mistral-7b- -eetabbt' as the reference model and for the initialization of policy in our paper. We train all models in this paper for epochs with LoRA (r = 16, a 16, dropout 0.05). Whenever comparing among different methods, we pick the one out of the three checkpoints with the best MT-Bench score for each method. For fair compar- ison of baseline models, we fix 3 to 0.01. It is more of a control variable than a hyperparameter because it is a given component of the RLHF objective which all baselines are aimed to optimize. We only vary / for them when probing their KL-Reward frontiers. For comparison of loss modifi- cation tricks, since the RLHF objective is not necessarily the purpose, we use the best 3 and other hyperparameters specific to each method as reported in previous work (e.g.. the tricks used in SimPO are only competitive when 3 = 2.0 for the Mistral model). Learning rate is grid-searched for each method among {le - 5, 5e 6, le 6}.
5.1. Experimental Setup
5.1.1. TRAINING DATA
We consider the dataset of Ultrafcedback (Cui et al., 2024) (denoted as 'UF-all') and widely used pair-wise version of it (Tunstall al., 2023) (UF-binarized). The two datasets are ideal for our purpose besides their popularity. Firstly, in UF-all, there are 4 responses sampled from multiple sources for each prompt. This will allow training with our general version of EPA which can utilize multiple strong negatives. Secondly, in both UF-all and UF-binarized, the positive sample yoe for each æ is the best one out of the 4 responses. This is an arguably close approximation to our assumption that positives are sampled from p(vy' # y: y > 1/)
5.1.2. EVALUATION
Although Ultrafeedback is intended to reflect human pref- erence, it is labeled by GPT-4 in reality. Consequently, we consider MT-Bench (Zheng et al., 2024) which also uses GPT-4 to score a response on scale of 1-10. The metric is the average score for 80 single-turn conversations and 80 multi-turn conversations. We also consider Alpaca-Eval Eval 2.0 (Dubois et al., 2024) because of its high correlation with human preference, the ultimate concern for RLHF. Its met- rics are win-rates (with or without length control) against GPT-4-turbo across 805 test samples with the judge being GPT-4-turbo itself. We report them in the format of "length controlled win-rate win-rate" in the experiment results. For evaluation on Alpaca Eval 2.0, coding parameters in the Huggingface implementation. For MT-Bench, we use the ones specifically required by the benchmark.
Table 2. EPA beats all other baselines either for pair-wise data for data with > 2 responses for each prompt.
TRAINING DATA METHOD
AE 2.0 (%)
MT-BENCH
SFT
8.16/
6.44
+DPO
17.43
15.24
15.24
7.55
+IPO
12.97
/ 10.13
7.31
BINAARIRI
*TTO
12.62
/ 11.29
7.21
+NCA
14.64/11.27
7.39
+EPA
19.20/19.26
7.71
the default de-
+DPO-PL 15.95 14.68 14.68
7.57
+NCA
15.08 11.85
7.28
UF-ALL
+NFFNCA
17.30 / 16.25
7.50
+EPA
22.03 / 21.44
7.58
5.1.3. BASELINES & Loss MODIHICATION TRICKS
For fair comparison, we only consider methods from the ap- proach that explicitly aims to minimize CRLHP with specific probabilistic models about data distributions. Therefore, consider DPO, IPO, KTO and NCA for the classical pair-wise data setting. We consider DPO-PL, NCA, and infoNCA for the general setting where there are multiple responses for each prompt in the dataset.
5.2. Results and Analysis
1.2.1. EPA PERFORMS BETTER THAN BASELINES
As shown in Table 2, we can see EPA consistently achieves the highest scores and hence new state of the art. Note that other baselines generally perform even less well than DPO. This makes BTM the strongest baseline for IPM.
Loss modification tricks are not considered as baselines because they are orthogonal to our proposal. Comparing BTM+tricks to EBM would be comparing apples to oranges. Instead, we consider applying the tricks to both EPA (the narrow one for fair comparison) and DPO to further verify
To compare IPM with its most competitive baseline BTM in detail, we come back to the starting point optimizing LRLIF. We study from two perspectives of the optimization problem. Both perspectives involve multiple checkpoints
6

---

Energy-Baat -aased Preference Model Offers Better Offline Alignment
(a)
high
7.70
7.75
1.25
7.5
DPO
8.75
Ave. [ivea]
#epacha
Figure 2. DPO vs. EPA (1:1:2) from the perspective of (a) KL-Reward frontier and (b) training dynamics.
beyond the single best one for each method (e.g., Table 2), offering more comprehensive comparison.
Table 3. Ablation and variants of EPA with varying number of strong negatives (Natrong) and that of weak negatives (N (Neeak) ) in addition to the positive (N+ = 1) in the denominator of [gemmll
First, we study how well each method balances the KL term and the reward term in CRLHF with varying B E {0.01, 0.02, 0.03, 0.04, 0.05, 0.1, 0.5). Both terms are com- puted on the 80 single-turn prompts the MT-Bench dataset. We estimate KL with 20 response samples per prompt from each policy distribution. We use the GPT-4 score produced by MT-Bench as an alias for the true reward. As shown in Figure 2.(a), in the high-KL region, EPA generally achieves higher reward than DPO. The two become indistinguishable only in the low-KL region.
METHOD
NT strom" &N
weak
AE 2.0 (%)
MT-BENCH
1:100(DP))
17.43 / 15.24 7,55
ABLATION
1:0:2
9.37 / 6.74
6.57
1:1:1
21.14 20.55
7.29
EPA
1:1:2
19.20 / 19.26
7.71
1:1:6
16.63 / 15.78
7.57
ABLATION
1:3:0 (INFONCA)
17.30 / 16.25
7.50
1:3:2
22.03 21.44
7.58
Second, understand how EPA differs from DPO in terms of the dynamics during the optimization process, test the MT-BBnnd score of the checkpoint at every 20% of an epoch. As shown in Figure 2.(b), EPA is less prone to overfitting and fits to the reward signal more steadily than DPO. The performance of DPO starts to degenerate rapidly after the first epoch. However, EPA reaches its peak performance the end of the second epoch and overfits more slowly afterward. This is consistent with gradient analysis in Section 4 that EPA is more regularized than DPO.
1:3:4
/1.31/20.11
7.58
EPA
1:3:6
24.01 23.44
7.35
1:3:8
24.54 23.75
7.19
1:3:10
23.58
22.78
7.43
Table 4. Adding the same number of weak negatives to the data for DPO (as additional y be paired with the original y..) DPO-PL (as additional negatives ranked after the yw and yr) does not show any advantage over EPA. + -UF-weak set of random weak negatives that is of the same quantity of prompts in UF- binarized. +UF-weakx x means add such sets,
add
5.2.2. COMBINING STRONG AND WEAK NEGATIVES IS EFFECTIVE
We also run ablation and variants of EPA for different num- bers of strong and weak negatives (ie., two hyperparameters of EPA). As shown in Table 5.2.2, we can observe the pat- tern that strong negatives and weak negatives should be roughly balanced to achieve the best overall performance. The absence or excessiveness of either kind of the negatives will lead to poorer performance. This is consistent with the intuition that the negative distribution should be both informative and of high variation.
Training DATA
METHOD
A.F. (%)
MT-BENCH
DPO
17.43 / 15.24
7.55
UF-BINARIZED
EPA (1:1:1)
21.14 20.55
7.29
EPA (1:1:2)
19.20 19.26
7.71
DPO
18.99 17.00
7.37
UF-WEAKXI
DPO-PL
15.78 15.17
7.54
DPO
18.49 18.72
7.42
UF-WEAKX2
DPO-PI.
15.59 14.79
7.44

---

Energy-Basse Basee Preference Model Offers Better Offline Alignment
Table BTM IPM with empirically effective loss modification
tricks.
Alpacs Eval 2.0 (50)
PREF MODEL
Loss MODIFICATION
MT- BENCH
21
AE 2.0 (%)
N/A (DPO)
17.43/ 15.24 7.55
usak
+ (CPO)
13.63/ 10.34
7.11
17
N
BTM
N
+lemp (R-DPO)
19.10/ 16.71
7.70
1
1 [18
18
ref lenn + m. (SIMPO)
20.57/20.11
7.61
+Wpp (WPO)
21.90/21.04
7.56
mm
22.33/ 20.61
7.67
Figure 3. Performance modified DPO (N = () and modi- fied EPA (Nweek 0) with a margin me added to (Mrls)) Solid lines represent the length-controlle win-rates, and dotted lines represent the raw win-rates.
N/A (EPA)
19.20 19.26
7.71
IPM
Wop
22.80/
7.61
+T/lc
23.00 / 22.47
7.68
ond one is the on on-policy weight proposed by Zhou al. (2024). It can be viewed as a curriculum learning technique which prioritizes samples that closely relate to the current policy distribution at each step.
5.2.3. COMPUTATION COST-EFFECTIVENESS COMPARED TO ALTERNATIVES
There might be a concern that EPA's effectiveness merely comes from more computations induced by the additional weak negatives. Therefore, we also experiment with the introduction of weak negatives to BTM and the more gen- eral Placket-Luce Model. As shown in Table 4, can see that EPA still performs better than DPO and DPO-PL in the setting where the total number of computations is strictly controlled. This indicates that the effectiveness is not based on the additional computations alone, but essen- tially a consequence of the superior theoretical property of IPM, an EBM that has a unique MLE. Nevertheless, one can still argue that it is somewhat shortcoming that EPA has computation complexity linearly related to the number of contrasting samples. However, when there is an additional computation budget available in practice, EPA is indeed so far the most cost-effective method to fully exploit the resource. In addition, we want to emphasize that the compu- tation cost is not an intrinsic attribute of replacing BTM with IPM (the core contribution/purpose of this paper), but only an attribute of EPA (a not-so-perfen way to estimate the IPM's MLE). As the studies of EBM continue progress, it is reasonable to expect IPM to outrun BTM even more.
As shown in Table 5, we find that both Wo and +m, produce similar performance boost on EPA to DPO. AL- though the marginal boost on EPA generally smaller than DPO, EPA with tricks is still better than DPO with tricks. However, the fact these tricks can still work on EPA also implies that there is still room for improvement. This may come from EPA not necessarily being the best algorithm to approximate IPM's MLE.
We also study in detail how the value of m. influences DPO and EPA. As shown in Figure 3, we observe that combination of higher m. and higher NWeak tends to produce higher performance. A possible explanation for this is that me scales up the logit of the strong negative yi to loosely approximate the existence of multiple strong negatives, we get closer to the performance of the general EPA.
6. Conclusion
In this paper, we show both BTM and our EBM (IPM) have the property that their MLE, if uniquely exists, is equivalent to the minimizer of the RLHF loss. However, the unique ex- istence of IPM's MLE is guaranteed whereas that BTM's MLE is not. This theoretical advantage implies that as long as the IPM's MIEE is accurately found, we are bound to minimize the RLHF loss. But, the same claim does not hold for BTM. Although EPA is just empirical attempt to ap- proximate IPM's MLE, it is already sufficient to outperform its counterpart DPO on open benchmarks, with or without loss modification tricks presented in previous work.
5.2.4. IPM+TRICKS > BTM+
Since most loss modification tricks presented in the re- cent offline aligument literature are originally intended for BTM/DPO and do not necessarily make sense IPM/EPA, only consider two of them when applying EPA. The first one is a constant margin m. added to the logit of yı- The trick can be viewed as a loose numerical approxima- tion to the general EPA where there are multiple Vt,- For example, if me = 1.4, we have exp (re(2,8). 31) + me] = exp [m.] x exp (ro vx)) x x exp [ro(1, v.). The sec-
However, EPA is far from perfect. For example, relatively poorer computation and memory efficiency is major handi-
8

---

Energy-Basse Basee Preference Model Offers Better Offline Alignment
cap of EPA. Foreseeable future work includes finding better ways to perturb data adopting more efficient methods to approximate the MLE. Loss modification tricks particularly tailored for EPA also remain to be explored.
R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. CoRR, abs/21 10.1110.1111. 14168, 2021. URL https:
wwtpg//a/g//.oo...////// 14168.
Cui, G., Yuan, L., Ding, N., Yao, G., He. B., Zhu, W., Ni, Y., Xie, G., Xie, R., Lin, Y., et al. Ultrafeedback: Boosting language models with scaled ai feedback. In Forty-first International Conference on Machine Learning, 2024.
Acknowledgements
We thank Yulong Zhou and Shaoke Lv for many insightful discussions. We also thank our colleagues who provided sta- experiments.
Dai, H., Singh, R., Dai, B., Sutton, C., and Schuurmans, D. Learning discrete energy- -bsedd models via auxiliary- variable local exploration. Advances in Neural Informa- tion Processing Systems, 33: 13:10443-10455 2020.
ble hardware environments for conducting Last but not least, we are grateful to the anonymous review- ers for their meaningful suggestions that helped refine this paper.
Deng, Y., Bakhtin, A., Ott, M., Szlam, A., and Ranzato, M. Residual energy-based models for text generation. In International Conference on Learning Representations, 2020. URL https: http:///ppeeeeeeeeeeeeeeem net.oorunttfftt forum?
Impact Statement
This paper presents work whose goal is advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
id-B114SgHKDE.
Dubois, Y., Galambosi, B., Liang, P., and Hashimoto, T. B. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint vxXiv22444.04477 2024.
References
Amini, A., Vieira, T., and Cotterell, R. Direct pref- crence optimization with an offset. arXiv preprint arXiv:2402.10571, 2024.
Eikema, B., Kruszewski, G., Dance, C. R., Elsahar, H., and Dymetman, M. An approximate sampler for energy- based models with divergence diagnostics. Transactions on Machine Learning Research, 2022.
Azar, M. G., Rowland, M., Piot, B., Guo, D., Calandriello, D., Valko, M., and Munos, R. A general theoretical paradigm understand learning from human preferences. arXiv preprint ar ar. 10.3366, 2023.
Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kiela, D. Kto: Model alignment as prospect theoretic optimization, 2024.
Bong, H. and Rinaldo, A. Generalized results for the existence and consistency of the MLE in the bradley- terry-luce model. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Pro- ceedings of the 39th International Conference on Ma- chine Learning, volume 162 Proceedings of Machine Learning Research, pp. 2160-2177 2177. PMLR, 17-23 Jul httpes::/pprpcceeeeeee h/pp//eeeeinggsss.e mlr www.mmlrrr........ prres// v1 62. 62/bbnng2a html.
Ford, L. R. Solution of a ranking problem from binary comparisons. The American Mathematical Monthly, 64 (8):28-33, 1957. ISSN 00029890, 19300972. URL 23305551..
http: / /www www.wwssto.... org/ wttrrg//taaeeeeeeeeee j/stto..org/saable/33eeeee
Han, R., Ye, R., Tan, C., and Chen, K. Asymptotic theory sparse Bradley-Terry model. The Annals of Applied Probability, 30(5):2491 2515, 2020. doi: 10.1214/ 20- 20-AAP1564. URL https: http://tt........../././.. 20-AAP1564.
2022.
Chen, H., He, G., Su, H., and Zhu, J. Noise contrastive align- ment of language models with explicit rewards. arXiv preprint #rxv:2402.005369 2024a.
Hendrickx, J., Olshevsky, A., and Saligrama, Mini- max rate for learning from pairwise comparisons in the BTL model. In III, H. D. and Singh, A. (eds.), Pro- ceedings of the 37th International Conference on Ma- chine Learning. volume 119 of Proceedings of Machine Learning Research, pp. 4193-4202. PMLR, 13-18 Jul 2020. URL.htt:ss://roooeet v119 1119/hene http///nnnixxx220...fm
Chen, Z., Deng, Y., Yuan, H., Ji, K., and Gu, Q. Self-play fine-tuning converts weak language models to strong lan- guage models. arXiv preprint arXiv:2401.01333
2024b.
Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg. S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information pro- cessing systems, 30, 2017.
ngs. ..p..mm.. m.r...mr..... r.press//
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. CoRR, aws/sa09..03030330 000000000000000...00000000000000 2020. URL
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, I.., Plappert, M., Tworek, J., Hilton, J., Nakano,

---

Energy-Basse -aayed Preference Model Offers Better Offline Alignment
Hong, J., Lee, N., and Thorne, J. Orpo: Monolithic pref- erence optimization without reference model. arXiv preprint arXiv:2403.0769) 2(4):5, 2024.
Meng, Y., Xia, M., and Chen, D. Simpo: Simple preference optimization with a reference-free reward. arXiv preprint arXiv:2405.14773 14734, 2024.
Hyvärinen, A. Some extensions of score matching. Com- putational statistics & data analysis, 51(5):2499-2512, 2007.
Munos, R. and Szepesvári, C. Finite-time bounds for fitted value iteration. Journal of Machine Learning Research, 9 (5), 2008
Kakade, S. and Langford, J. Approximately optimal ap- proximate reinforcement learning. In Proceedings of the Nineteenth International Conference Machine Learn- ing. pp. 267-274, 2002.
Ouyang. I.., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744 2022.
Khalifa, M., Elsahar, H., and Dymetman, M. A distri- butional approach to controlled text generation. In In- ternational Conference on Learning Representations, 2021. URL httpp:://ppezeeee/eee wtwpppereeiweeeeeeeeeooooo www.eeww.eetooooeeoeeeeeee forum? 1d= 1wKw45-9A4
Park, R., Rafailov, R., Ermon, S., and Finn, C. Disentan- gling length from quality in direct preference optimiza- tion. arXiv preprint arXiv:2403. 19159, 2024.
Parshakova, T., Andreoli, J.-M., and Dymetman, M. Distributional reinforcement learning for energy-based sequential models. Optimization Foundations for Reinforcement Learning Workshop at NeurIPS 2019, 2019. URL https:://ptp////t assets http://cctceeeeeeeeee/////// pdf.
Korbak, T., Elsahar, H., Kruszewski, G., and Dymetman, M. On reinforcement learning and distribution match- ing for fine-tuning language models with no catastrophic forgetting. Advances in Neural Information Processing Systems, 35:16203-16220 2022a.
9. github.
Korbak, T., Perez, E., and Buckley, C. RI with kl penalties is better viewed as bayesian inference. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 1083-1091, 2022b.
Perez, E., Huang, S., Song, F., Cai, T., Ring, R., Aslanides, J., Glaese, A., McAlcese, N., and Irving, G. Red teaming language models with language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 3419-3448, 2022.
Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pp. 1207-1216, Stan- ford, CA, 2000. Morgan Kaufmann.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. In Thirty- seventh Conference on Neural Information Processing Systems, 2023, URL https: wptp//veenreeeeeeeeeeeeeeeeeee.m net/
Lazaro-Gredilla, M., Dedieu, A., and George, D. Perturb- uid-max-product Sampling and learning in discrete energy-based models. Advances in Neural Information Processing Systems, 34:928-999 940, 2021.
Schröder, T., Ou, Z., Li, Y., and Duncan, A. B. Training discrete EBMs with energy discrepancy. In ICML 2023 Workshop: Sampling and Optimization in Discrete Space, 2023. URL https://////..... /ttp///pnneeeeeeeeeeeeeee/o net/ fooum?? i dd=KMpJJ775W.
LeCun, Y., Chopra, S., Hadsell, R., Ranzato, M., and Huang, F. A tutorial on energy- based learning. Predicting struc- tured data, 1(0), 2006.
Li, T., Chiang, W.-L., Frick, E., Dunlap, L., Wu, T., Zhu, B., Gonzalez, J. E., and Stoica, I. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406.1 1939, 2024.
Simons, G. and Yao, Y.-C. Asymptotics when the number of parameters tends to infinity in the bradley-terry model for paired comparisons. The Annals of Statistics, 27(3): 1041-1060, 1999.
Majerek, D., Nowak, W., and Zieba, W. Conditional strong law of large number. Int. J. Pure Appl. Math, 20(2): 143-156, 2005.
Song, Y. and Kingma, D. P. How train your energy-based models. arXiv preprini 20XXi:2201.03222 2021.
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. Learning to summarize with human feedback. Ad- vances in Neural Information Processing Systems, 33: 3008-3021, 2020.
Meng, C., Choi, K., Song, J., and Ermon, S. Concrete score matching: Generalized score matching for discrete data. Advances in Neural Information Processing Systems, 35: 34532-34 34545, 2022.
10

---

Energy-Basse -Baeed Preference Model Offers Better Offline Alignment
Tang, Y., Guo, Z. D., Zheng, Z., Calandriello, D., Munos, R., Rowland, M., Richemond, P. H., Valko, M., Pires, B. A., and Piot, B. Generalized preference optimization: A unified approach to offline alignment. arXiv preprint arXiv:2402.05749, 2024.
Álvaro Bartolomé Del Canto, Blázquez, G. M., Lajarín, A. P., and Suero, D. V. Distilabel: An ai feed- back (aif) framework for building datasets with and for llms. https: htpp:://qiiee http///gtttu..tt wwt.eoooommaaaaaeee argilla-io/ d' st il label, 2024.
Tikhonov, A. and Ryabinin, M. It's all in the heads: Using attention heads as a baseline for cross-lingual transfer in commonsense reasoning. CoRR, abs/2106. 12066, 2021. URL https: org/ /210 12066.
Tunstall, L., Beeching, E., Lambert, N., Rajani, N., Ra- sul, K., Belkada, Y., Huang, S., von Werra, L., Fourrier, C., Habib, N., et al. Zephyr: Direct distillation of Im alignment. arXiv preprint arXiv:2310.16944, 2023.
Wu, W., Junker, B. W., and Niezink, N. Asymptotic compar- ison of identifying constraints for bradley-terry models. arXiv preprint arXiv:2205.0453
2022.
Xu, H., Sharaf, A., Chen, Y. Tan, W. Shen, L., Van Durme, B., Murray, K., and Kim, Y. Contrastive preference optimization: Pushing the boundaries of Ilm performance in machine translation. arXiv preprint arXiv:2401.0844 2024.
Yang. J. Q., Salamatian, S., Sun, Z., Suresh, A. T., and Beirami, A. Asymptotics of language model alignment, 2024.
Yuan, H., Yuan, Z., Tan, C., Wang, W., Huang, S., and Huang, F. RRHF: Rank responses to align lan- guage models with human feedback. In seirth- Conference on Neural Information Processing Systems, 2023. URL httppp:///peeeeeiiee.. n forum? id-EdIGMCHK41.
Zhan, W., Huang. B., Huang, A., Jiang, N., and Lee, J. Of- fline reinforcement learning with realizability and single- policy comcentrability. In Conference on Learning Theory, pp. 2730-2775. PMLR, 2022.
Zheng, L., Chiang, W.-L.. Sheng. Y.. Zhuang. S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging almdae-a-judge with mt-bench and chatbot arena. Ad- vances in Neural Information Processing Systems, 36, 2024.
Zhou, W., Agrawal, R., Zhang, S., Indurthi, R., Zhao, S., Song, K., Xu, S., and Zhu, C. Wpo: Enhancing rlhf with weighted preference optimization. arXiv preprint 2rXiv:2406.11827 2024.
Ziegler, D. M., Stiennon, N., Wu, J., Brown, B., Radford, A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning language models from human preferences. arXiv preprint arXiv: 1909 08593,, 2019.

---

Energy-Basse Baaeed Preference Model Offers Better Offline Alignment
A. Proof of the Equivalence Between Slope-1 Linearity and Minimizer of the RLHF Loss
Proof. If mg Ty, then according Lemma A.I, have:
1 x0 Z(x)
Ter(y)z) cxp 3 r( y)
We do not claim any originality for the proofs given in this section because they are largely just paraphrased versions of the work by Korbak et al. (2022a); Rafailov et al. (2023) and others. We include them just for reference and complete- ness the mathematical foundation shared by both DPO and EPA. Throughout this paper, we make the same mild assumption by Rafailov et al. (2023) that Tet is strictly positive.
Take the logarithm of both sides of this equation, have:
1
log w log neef + 3 y) log (xx)
3
After moving the two log terms to the same side, we get slope- linearity:
Lemma A.1. The minimizer of the RLHF objective uniquely exists.
B Blog = r(x, log ((x)
Tref
Proof. We show the minimizer can be analytically expressed by Z(z) Te((U/z) exp rr(z, y) where (x) =
Theorem A.4 (Theorem of sufficiency). If slope-1 linearity holds, then mg = Ty-
En (y)r exp r(( y) (i.e., normalizer to make nt a probabilistic distribution).
Proof. From the Theorem of necessity, we know:
From the property of Gibb's inequality, know:
B Boog Tee
-r(x, 3 log ((x)
n. Teat (v)=)) exp 1 7(2 v)]
7(x)
Substracting the slope-1 linearity from this equation, we
get:
JT arg min SKI.[To
(y)))))) exp r v)
Blog
TT
3 log
T log ((x) - C(x)
((r)
Tel
We will complete the proof by showing the 3KL-Divergence on the RHS of the above equation is the RLHF objective itself plus constant w.r.t A:
Eliminating the non-zero 3 from both sides and taking the exponential, we have:
T f(x)
[oL Z(x)
())r) exp 3 ((x,v)) v)]
where f(x) Z(x))xp]] C(z) Moving xo to the RHS, we get:
To(()x))
3 E log x
((x) Teet (y/or) exp gr y)
Tr - fx)
(xx)
To ( x) Teer (yl
Taking D2 for both sides, we can sum up both distributions Tr. and Ty to one:
=
E
llgg
+ log
(a()(a)
exp 3 (2, y)]
1= f())
no ( (x)
= E
[log (xx) / r 1)) log
Therefore.
))))(
xo(ylæ)
Tr = -Tf(x) = m
E
(r(x,v)) v)] + |KL [TK
=o(D)m))
+Blog ((x)
Note that from the above proof, we casily get the follow- ing corollary because f(x) = e C(x) - -9 Z(x). Corollary A.5. when satisfies slope-1 linearity, it is unique.
CLLHr(()) 1og ((x)
Definition A.2.
We say a slope-1 linearity holds when:
B. Theoretical Aspect of the Infinite Preference Model
ro(o, v) r(x,)) C(x)
where ro(x,y) Blog To (v|z) (v
We will first give our proof of the guaranteed unique exis- tence of our IPM's MLE. Then, we will discuss how BTM is flawed for an infinite space of y/ær.
Theorem A.3 (Theorem necessity). If TT = Ty. then slope-1 linearity holds.
12

---

Energy-Basse -aased Preference Model Offers Better Offline Alignment
B.I. On IPM's MLE
both sides:
We make the following mild assumptions about the structure of human preference:
ro(s,)) v) - Ci(x) r(x,)) - C2(2)
Assumption B.1. =1 (ylp(y/z)) 0} is a finite set.
ro(2, v) r(zz v) C(a)
Assumption B.2. r(x,,) - - -o for any y $ Dij and r(z,)) y) +o0 for any y E D. g
where Ci(r) = log mn explro v')l. C2(2) log by exp/r(x,1) y and C(z) C1 C2(2)
Note that the above two assumptions are just one of many sufficient assumptions that make the partition function exist as finite real number. Also note that the finity of Di la and the infinity of the space of y are two different things that can certainly co-exist. Namely, D. Jar is subset of the space of y|ı. The number of outside of D.je is still infinitely large. The two assumptions in plain words are simply that we assume humans will only possibly prefer a finite set of responses. Note that this does not mean that the finite set Dije cannot be very large.
Note that similar proof does not apply to BTM. The fun- damental reason is that the C (z) and C2(z) only become constants when there is an infinity in the sum to cancel out all y-
B.2. On Bradley-Terry Model's Flaw
We will show in Proposition B.5 that very likely choice of p(y w/(r) will lead to multiple minimizers for the max- imum likelihood estimation of BTM. There are also many other choices of p(de. (z) are known to cause the exis- tence of multiple minimizers (Ford, 1957; Bong & Rinaldo, 2022), such as when there is no full connectivity of the graph made by pairs from p(ue 3r|zz) and when all y candi- dates can only be paired with single shared winning ÿ. etc. Therefore, there is no guarantee for the MLE's uniqueness without imposing additional constraints on p(ymr 3/ (z) (i.e, how the pairs are sampled for DPO). For the loosest suffi- cient constraints discovered so far to ensure the uniqueness, one can refer to Bong & Rinaldo (2022). However, the best of our knowledge, how such constraints can be applied to DPO has never been studied in the offline alignment liter- ature, which is also out of the scope of this paper. Moreover, in the infinite-candidate scenario, a constraint that is both necessary and sufficient for the uniqueness of BTM's MLE remains unknown to this day. What makes BTM even more theoretically troublesome in the case of RLHF is that there is also infinity for the space of x as well. Therefore, strictly speaking, there is infinite number of BTMs used in DPO. And, the p(yee yr (z:) for every should ensure the uniqueness, in order to make DPO really work as expected. Interestingly, although our EPA loss also needs an infinite number IPMs in the strict sense, Theorem B.4 (3.1) ensures the MLE uniqueness of all the IPMs.
Definition B.3. The maximum likelihood estimation objec- tive of IPM is the negative log-likelihood of preference data computed follows:
Eyplyl log (v(lz)
where p(yla))
explr(z.)) and (0(1lx)
vy expir(a.))]
explro(m-1]] 309 explro (a.y)
Given the uniqueness in Corollary A.5, we can argue the following:
Theorem B.4 (Theorem 3.1 in the main content of the pa- per). The ra that satisfies the slope-1 linearity is the unique minimizer of the IPM's maximum likelihood estimation ob- jective.
Proof. Again, from the property of Gibb's inequality, we know:
qo (v(lr) = p(y]z)
qo (y|x) = arg min KLpp())))lllll))))
Proposition B.5. If there exists a y² that will never be sampled (i.e., ply", whenever there is minimizer for Bradley-Terry' maximum likelihood estimation, it is not unique.
For the equation on the right, since p(y|== ) is a constant w.r.t 8, we can find go is the minimizer of IPM's objective:
0 and v(=v"(x)) = 0), then
go(vi (o(u = arg min 9
arg min H[p(y]=)) 2. p(ular log (og()))))
Proof. Without losing generality, we set 3 1.
Given the log ratio reward parameterization, we have an intrinsic constraint on ro:
arg 2. p(y) ( log (
DO Σ 4
We then show that qo (y|z) = p(y|z) is equivalent to slope-1 1 linearity to complete the proof by taking the logarithm of
Teet exp(ro)
T
13

---

Energy-Basse -aargd Preference Model Offers Better Offline Alignment
If we assume that there is unique minimizer ra to BTM's maximum likelihood estimation, it certainly satisfies the above constraint:
and since A(x) 0 and F [0, 1]. have:
1 2 exp[A(x)][TTé 1) +
2 exp[0 - 1) +
Tel exp(rp)
T 20 0
hence 7tg E [0, 1];
We will then show that another reward also follows the constraint (hence a valid log ratio reward) and shares the same expected data likelihood as rg. which contradicts the uniqueness of rg. We define the other reward as:
2) for # y:
n = net exp[
n explr, A(x)
(xplA(e))Terreppl))
(x) = log(exp(ro v) + A(x)] + 1-esp[A(al)) ro(xx v) A(x):
yfy=
Real
-xplA(r])
T
otherwise
ref
= exp[A(x)]TT
where A(x) can be any negative constant w.r.t This reward satisfies the constraint because:
[0,
Then, we show that this valid log ratio reward indeed another minimizer because it leads to the same expected likelihood of data rg.
Tet explF
E
Dog a (fg Mw) - (2 Bi))
Tet expli Teat explig(r.,"
dl (r) p(u > a))
8 #"
plyw m/
E ply=>ullay
[log (Fa Mww ig( v1))
Σ
Nee exxe(pp((x, A(x)]
Mus
8'
p(y y1 (x)
+ ner exp exp(ig(x,
Mu uutt
= =xx[[(xx)
Σ 8'#y"
Tie explaaxxx)
E p(yn (1(z))
[log (Fg - (2(2, v1))
+ + exp(px(xpx y²)
Σ Uw #r an #2
P(u 91 (1)x))-
exp|AAx)] 7 exppr
exp (tppp/(ap/..)//p////p y y²)
+ + (xp]F
E p(yn>ulle vulx))
[log (rg ywm) A(x) (g(2x, A(x))
= exp[A(x)](1 nue exp |r (z, }]]
+ Tet cxppr/(=.//))) + A [x] + (1 - exp[A(x)))
Σ Vo #o vu
p(y 91 (z)
= -
ner expl
(2))
(explA(a)(TTeeexplr y 1) +1
E p(y Mn
[log (rg( yw) - rr( 91))
=1
pluw: Mr
E
[log o (r vwo) (2 vi))
Note that the constraint makes Fa correspond valid policy म net exp(ft) that sums up 1. The policy is also in the range of [0, 1] because:
plu > melar)
E
E
Dog a (r Mw) rz(2, v1))
P(u p(u
M1 (m))
1) for y y":
Finally, because the possible choices of A(z) are infinite and by assuming the full representation capacity of large neural network, मे can be represented by some other Tg.
म = nea explial
Tiet explra A(x)) + ex)(A(x)]
explAAxx) Tee expl/ + exp[A(()))
Note that if we are given the p(y 31 |z) defined in Proposi- tion B.5, even we have an infinite amount of data, we still do not have a unique MLE. This refutes a popular claim that DPO will work better given enough data.
-cxp(A(>) TTz T6 +1 cxp[A(()
Jee
cxp[A(x)](TT) - 1) +1
14

---

Energy-Bass -Based Preference Model Offers Better Offline Alignment
Also, if the rewards of appearing y are shifted by constant, note that the key intuition of the above proof is that the finiteness of the number of candidates considered in the maximum likelihood estimation loss makes itself invariant as long as some y" outside of the finite set of candidates never appears. This means a similar property as follows also holds for the Placket--Luce Model as it also just considers finite set of candidates.
in the derivative becomes identical the first term:
E x.p(yl=). P ( [1(x;; z/)]
E
p(zly') - expl (z. 1/1
E sppla php(aly)
h(2,2')
Σ p(zle/)) (2.
p(zly (pzl))p(ull1)) h(2, 30)
E
x. x.p(ul=).,(Elv
Σ (((l)"))p("" y
Corollary B.6. there exists a y that will never be sam- pled, then whenever there is minimizer for Plackett-Luce's ss maximum tikelihood estimation, it is not unique.
Σ
Ex
p(ul))
Ep(zly")p(""la
In contrast, a similar proof will not work for our IPM as it considers the entire infinite space of candidates.
0
plylz)p(z
Ex > p(y (x)((,,)))) Σ
B.3. Fitting IPM with Energy Discrepancy
pl
In our paper, since we simply adapt the general theorems given by Schröder et al. (2023) to fit the RLHF context, we only provide proof sketches for quick reference and claim originality thereof. We encourage readers to refer to the original full proof if more details are needed.
Σ p(zly)p(ulz)
x6 Ex Σ
ply|2))/(z, (2)((2,)))) 5/) Ep(z/1/
o
Ex
p()'lz)((())))
Ep(El))
= E. 3
Proof sketch for Theorem 3.2. The first functional deriva- tive of ED2, (.Dzp.alls)pp((lll) [E] (we use E = -r to conform to the convention of EBMs) is given by:
With the first variation being O, we will then only need to show that the second variation of ED at E is strictly positive to complete the proof that E = - -Tnue is the global unique minimizer of FD. This can be done by showing that the second derivative becomes an expec- tation of Varp(yle) [h()]] which cannot be negative or zero because we assume Varp(ale) > 0 (i.e., Var [Z] > 0). Concretely, the second derivative of ED is given by:
-inue
d E ED
[E + ch]
de
E [1(x,v)) =.p(y/=)
E =xplvlz) (b'1=)
E.
[1(x,, y')l
d² ED dea (.Da.polle) + ch]
de
E de
E (v)))
[M v/
where
x.p(vl=))p(=)p))
(see (Schröder al., 2023)'s 's Lemma 2)
p(zl1/) exxp[=F(x,)) ch(xx//)))
E *.p(vl=).m=ll))
(1((z,))) -
E
[M
(E)((1/12)=
Σ p(2l)"") y"
v² ch(xr
E *.p(vl=).p(=(u)
Varpp...v'la (M) [Mx,,''))
Setting e = 0 and E -Tirus we will get the positive second variation of ED E Tiree-
Then, setting e = O and E = - Tunee we get the first varia-
tion of F. at -Tinee to be because the second term
Proof sketch for Theorem 3.3. Given the property that
15

---

Energy- -Based Preference Model Offers Better Offline Alignment
(r)
Table Preliminary results on tuning 3 for IPO.
Croes nalr]
B MT-BENCH (EPOCH #1 #2 / #3)
(an(f)
0.1 6.73/6.88/ /
0.01
7.20/7.31/7.23
20g
Caraa ma(r]
tion than the minimizer of the RLHF loss. Increasing the amount of data for DPO can certainly mitigate the fuzziness but can do nothing to avoid the undesirable structure of BTM's maximum likelihood estimation loss governed by P(ve; (x).
Casale)
The global minimiser be (- ruse (J) of Casslrl
C. More Experimental Details
Figure 4. A simplified illustration of the topology of the function- als mentioned in the paper's theorems. The vertical axis represents the value of each functional. The horizontal axis represents the space of
C.I. Implementation Details
C.1.1. MORE HYPERPARAMETE DETAILS
We use 8 A100/8800 GPUs (80G Memory) with ZeRO3 parallelism train each model in this paper. Global batch size is fixed to 64. For experiments in Figure 2, two rounds with different seeds (0 and 1) for each model configuration. Other experiments are only conducted with seed O. The reason we set 33 to 0.01 in the main experiments is two-fold.
Eyp(=ly)f(g)= = fpp(yl=)f(v), the (z)p("))") in the defi- nition of ED becomes an expectation over y. i.e., [x)) This enables a statistic estimate using the normal and mod- ified Strong Law of Large Numbers (Majerek et al., 2005) for the expectations outside and inside of the logarithm, respectively.
Firstly, results (see Figure 2.(a)) of tuning 3 for multi- ple values ranging from 0.01 to 0.5 show that 3 close 0.01 (low-KL region) will make both EPA and DPO achieve higher rewards.
B.4. The Perspective of Functional Analysis
An elegant way to understand the theoretical advantage of EBM/EPA over BTM/DPO is through the lens of functional analysis. As illustrated in Figure 4, our EBM's maximum likelihood estimation loss, the energy discrepancy, and the RLHF loss can be regarded as three functionals sharing the same unique global minimizer. The EPA loss, being a sta- tistical e of the energy discrepancy, yet another functional. However, the Strong Law of Large Numbers ensures that the error between it and the energy discrepancy will almost surely vanish with enough amount of negatives drawn from properly designed data perturbing source Z. Therefore, one can view the EPA loss as a locally fuzzy approximation of the energy discrepancy. The maximum likelihood estimation loss of BTM is a functional that may have multiple minimizers. Therefore, without explicit con- straints prevent this from happening, chances are that optimizing the DPO loss (which is strictly speaking also just statistical estimate of the maximum likelihood of data, probably with less fuzziness) will lead to different solu-
Secondly, 8 0.01 is also the default setting for NCA and infoNCA (Chen et al., 2024a) and also a recom- mended setting for Mistral-based DPO (Tunstall et al., 2023) and KTO (Ethayarajh et al., 2024). For IPO's best 9, we did preliminary experiments with 0.01 and 0.1, and found 0.01 works better (see Table 6).
C.1.2. Loss MODIFICATIONS TRICKS
In Table 7, we list how the tricks are applied to the DPO loss or the narrow EPA loss. For more detailed properties of them, one can refer to the corresponding previous work. In summary, length penalty (Park et al., 2024), length normal- ization (Yuan et al., 2023; Hong al., 2024; Meng et al., 2024), constant or dynamic margins (Meng et al., 2024; Amini et al., 2024), removal of the reference model (Xu et al., 2024; Hong et al., 2024; Meng et al., 2024), on-policy weighting (Zhou et al., 2024) and addition of SFT loss (Xu et al., 2024; Hong et al., 2024), etc.
"Strictly speaking, there is also constant shift log M, which we removed from the energy discrepancy derive (10) because has no topoloicaa impact terms of optimization problems.
16

---

Eeergy-Bassed Preference Model Offers Better Offline Alignment
could cause the lower scores on GSM8K and MMLU.
Table 7. Notation and function of loss modification tricks proposed in the offline alignment literature. We consider on- on-policy weighting (+wop), length penalty (+leny), length normalization (+len,)) constant margin (+mm). removal of the reference model (-ref), and addition of SFT loss (+fr))).
C.2.2. PROBING THE SLOPE-1 LINEARITY
Reward models in the classical sense only need to satisfy ranking consistency with the corresponding true rewards. However, for ideal offline optimization of the RLHF loss, a learned log ratio reward and its corresponding true re- ward have to satisfy the slope-1 linearity, a much stronger requirement than ranking consistency. The reason is that, to satisfy ranking consistency, the relationship between the two rewards can be any monotonically increasing function, including those that are non-linear or linear with the slope being any positive number.
NOTATION
TRICK FUNCTION
-ref
log no (yls) (yur(y
+ log NTe (v((a)
+haft
C - L - log (ow (x)
+lemp
To (2) (z) - To (u/zc) alyl
+len,
(0()x)) + [) re (y(
Therefore, metrics that are based on ranking consistency to evaluate the log ratio reward (i.e., the learned reward model) are not meaningful when it comes to offline alignment. In- stead, we evaluate the log ratio reward by probing how well the linearity is approximated. For this purpose, we need multiple samples of y (responses) given an z (prompt) and their true rewards. The test split of the Ultrafeedback data (Tunstall et al., 2023) can fulfill this purpose because there are four y for each x and they are scored using the same scoring scheme used for our training data (i.c., UF-all and UF- (iF-binarized). We randomly sample 500 prompts from the test split to speed up evaluation while preserving the general reliability.
+me
ro(y(x) - rv 22) + me
+Wop
- 1l. (11,,
mo 1o _x) E Sv' 20 10_
C.1.3. COMPARING EPA WITH OTHER MULTI- RELLI-R LOSSES
appears that EPA and two multi-response losses (InfoNCA and DPO-PI.) are very similar because they are all con- trastive losses in the form of negative log softmax. However, there is a clear distinction between EPA and the other losses in terms of the required information in the training data and the samples being contrasted (cxemplified by Table 8). This is a direct consequence of using different probabilis- tic models. EPA comes from our IPM. DPO-PL comes from the Plackett-L.uce Model, and InfoNCA comes from an energy-based model for the distribution of the optimal policy.
Firstly, we consider only how linear the relationship between the two rewards is, regardless of the slope. This is exactly the essence of Pearson correlation analysis. We compute the Pearson coefficient between the two rewards over the four y for each prompt. Then, we compute the mean over all 500 prompts as metric. We report the results in the first column of Table (in the main content of the paper).
Secondly, we study both the "slope-1" and the "linearity". We do this via linear regression with the slope fixed to 1. Specifically, given a prompt, we need to fit a linear regression model Neaamed = 1 Time + b the 4 coordinates of (Flaamed, F'me). With simple algebra, the optimal value of b that minimize the linear regression error c = Σ E|
C.2. DPO vs. EPA from More Perspectives
C.2.1. ALIGNMENT TAX
Preference alignment is usually associated with an align- ment tax: forgetting certain capabilitiees (e.g., math problem solving) while enhancing others (c.g., safety, truthfulness, and helpfulness). We acknowledge that EPA might exhibit a higher aligmment tax than DPO due to the results in Table 9, where we report metrics on GSM8k (Cobbe et al., 2021), MMLU (Hendrycks et al., 2020), and Winograd (Tikhonov & Ryabinin, 2021). However, a more comprehensive future work on this issue is still necessary for a reliable conclu- sion. For example, although GSM8k and MMLU are of relatively higher correlation with human preference than other Exact-Match-base benchmarks, they have poorer cor- relations than the ones used in the main paper (MT-Bench and Alpaca-Ewe Eval 2.0) (Dubois et al., 2024). Therefore, a more aggressive alignment method with human preference
Pleamed + b12 can be analytically expressed as b (1/4) ([Teamed Tine). Thus, we use à to compute the minimal error for each given prompt, and then compute the average minimal error é over all 500 prompts as the metric (see the second column of Table 1). We can also shift the four Tlearned for each prompt by the constant -6, which should move all regression lines for the slope-1 linearity to the same location: the diagonal Fleaned = 1 Tune This allows us to visualize the overall degree of how well the slope-1 1 linearity is approximated. As shown in Figure 5, we can that for the top 10% prompts with the best (i.c., smallest é), the slope-1 linearity is well approximated for both EPA and DPO. However, we can observe that DPO is slightly off
17

---

Energy-Based Preference Model Offers Better Offline Alignment
Table Comparing EPA, InfoNCA and DPO-PL. in the setting of 3 responses per prompt (omitting an for clarity).
NAME
DATA
LOSS
EPA
(yee M1 W12), (yw m1 (yw M12)
log -oxprr (ra eepplry (Wack 1 1
explrg (
DPO-PL
(Whe u 31 (yw > U1, > 1112)
log exp(ra(vu)) Texpoo (01 -exptra
exp (ww)
log exp (o
exp(ra(V;
INFONCA
(Ne: M11 H12), (Nhw W... (the M12)
log (xprr((y)))==xpppp) -
explry
Table 9, EPA has slightly higher alignment tax than DPO.
RPA (1.02)
EPA (SLAS)
WHeT 1016 0.7
TORE 0.30
VIAT HOS -2 0.
МETHOD GSM8K-5SHOT
MMLU
WINOGRAD
SFT
0.421 (0.014)
0.598 (0.004) 0.800 (0.006)
+DPO
0.463 (0.014)
0.593 (0.004) 0.790 (0.006)
+EPA
0.419 (0.014) 0.591 (0.004) 0.790 (0.006)
from the slope-1 linearity for the medium 10% and much so for the worst 10%. On the other hand, for EPA, although the points are also gradually spreading towards the worst 10%. they are still distributed along the direction of the diagonal. This phenomenon means that EPA is closer to the slope-1 linearity than DPO, especially for the worst group of prompts.
when
Wabe 12% 20U47
KourE 105 :4 2509
WUDKET 10% -E 23.35
Table 10. EPA vs DPO applied to Llama3-8B based SFT model.
METHOD MT-BENCH (EPOCH #1 / #2 / #3)
Prancesd!
DPO
6.94 6.83 6.83
6.71
EPA
6.84 6.98 7.04
Figure 5. EPA DPO in terms of the slope-1 linearity. Each average slope-1 linear regression error is computed on 500 x 10% 50 prompts. Although the difference between EPA and DPO only becomes noticeable for the "WORST 10%" group in the visualization (i.c., how close the partially transparent red dots are to the diagonal), the difference in é is conspicuous.
Table 11. EPA DPO Arena Hard.
TRAINING DATA METHOD
ARENA HARD (%)
DPO
12.0
UF-BINARIZED
et al., 2024) of the widely used but small dataset: In- tel/orca_dpe sptpadpoopppppp pooooppirs..oooaa As shown in Table 10, EPA is still better performing than DPO. In addition, we also report the best checkpoints for EPA and DPO/DPO-PL. on Arena Hard (Li et al., 2024), yet another benchmark that is considered more difficult. The result is consistent with Alpaca Eval 2.0 and MT-Bench reported in Table 2, further validating the superiority of EPA.
EPA
16.3
DPO-PL
13.0
UF-ALL
EPA
16.9
C.2.3. GENERALITY OF THE PROPOSAL
Although we believe the experimental configuration de- scribed in the main content of the paper provides sufficient support for our core theoretical argument, we present addi- tional results here using a different base model and training dataset. Specifically, we train Llama3-8B based SFT model on a cleansed version (Álvaro Bartolomé Del Canto
18


Reviews:
[{"review_id": "2BTTiDTAn9", "reviewer": "Reviewer_PHXo", "summary": "Bradley-Terry model (BTM) has been the default modelling assumption between rewards and preferences, used in RLHF and DPO. This paper challenges the BTM, claiming that the BTM has not guaranteed unique minimizers in DPO training. To solve this issue, they propose Energy-based Preference Model (EBM), which models preferences in a Boltzmann distribution. The paper claims that the EBM has unique minimizers towards the global optimum in the RLHF objective.\n\nWhile EBM is not tractable, the paper proposes Energy Preference Alignment (EPA) to approximate the EBM objective. Specifically, they propose to use two types of negative responses: negative responses from the same prompt and negative responses from different prompts. \n\nThey conducted thorough experiments to compare EPA and other baselines like DPO. Their experiments show superior performances of EPA in benchmarks like MT-bench and AlpacaEval. They also included ablation studies to investigate the impact of \"weak negative examples\" and \"other tricks\".", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Concern #1: The implication of Proposition B.5: the issue of the dataset or the issue of the DPO algorithm?\n\nWe believe a reasonable definition of a good algorithm is one that works for datasets \"easy\" to collect. If there are too many constraints to collect the datasets, it is indeed an issue of the algorithm.\n\nB.5 gives a necessary constraint which is itself not easy to meet, i.e., we can hardly sample all $y$s from an infinite space given a finite data point budget. \n\nWhat makes the datasets \"even harder\" to collect is that the B.5 constraint is not sufficient. This means even if we somehow are able to sample all $y$s in an infinite space, we still do not guarantee such a dataset will allow the DPO algorithm to work. As mentioned in the paragraphs before B.5, one can refer to (Ford, 1957; Bong & Rinaldo, 2022) for more situations for BTM to have non-unique MLEs.\n\n\n> Concern #2: \"why EBM still have unique minimizers when some y* never appear in their training data?\" & \"what bout EPA\"\n\nAs shown in Theorem 3.1 (also B.4), EBM having a unique minimizer to its maximum likelihood estimation is not dependent on any requirements on the structure of $p(y|x)$. Even when some $p(y^*|x)=0$, the proof we provide to Theorem 3.1 still holds.\n\nFor EPA, we acknowledge that there will be an error in the approximation (illustrated as the fuzzy dashed curve in Figure 1). This is why we need to check empirically whether or not the error introduced in EPA is smaller than that of DPO. For example, Table 1 explicitly shows that the slope-1 linear regression error is smaller for EPA, although the error is non-zero.\n\n> Concern #3: Difference between EPA and RPO, infoNCA, etc.\n\nWe noticed that the RPO paper was posted to arxiv.org after the submission of our paper. So, we were not aware of this concurrent work. Although EPA itself may or may not be a special case of RPO, we argue that our paper is more about the underlying energy-based preference model as opposed to EPA (which is only one of many possible ways to fit the EBM).\n\n(see our response to xiBU for EPA vs other losses)\n\n> Concern #4: Why setting $\\beta=0.01$ for Table 2? \n\nAs mentioned in the paper's subsection 5.1.4, the philosophy to set $\\beta=0.01$ in Table 2. is to treat it as a given part of the RLHF objective (Eq.1). As for the reason why we choose 0.01 in particular, it is two-fold. \n* Firstly, we did tune the $\\beta$ for multiple values ranging from 0.01 to 0.5 (see Figure 2.(a)). $\\beta$ close to 0.01 (low-KL region) will make both EPA and DPO achieve higher rewards. \n* Secondly, $\\beta=0.01$ is also the default setting for NCA and infoNCA (Chen et al., 2024a) and also a recommended setting for Mistral-based DPO (Tunstall et al., 2023) and KTO (Ethayarajh et al. (2024) ).  For IPO's best $\\beta$, we did preliminary experiments  with $\\beta=0.1$ and $\\beta =0.01$, and found $\\beta=0.01$ works better. Therefore, we believe setting $\\beta =0.01$ is a reasonable choice. We will add these results (MT-Bench scores) in the final version of the appendix:\n| $\\beta$ | IPO (epoch#1/#2/#3) |\n|--------|----------------------------------------|\n| 0.1    | 6.73/6.88/6.87                         |\n| 0.01   | 7.20/7.31/7.23                         |\n\n> Concern #5: \"how are the weak negatives added to DPO or DPO-PL training?\" & \"what about InfoNCA?\"\n\nIn the caption of Table 4., we have briefly stated how the weak negatives are added. The following are some examples for \"+UF-WEAKx1\".\n\nsuppose the original pairwise dataset has 3 data points: $(y_w^1, y_l^1), (y_w^2, y_l^2), (y_w^3, y_l^3)$\n\n* For DPO, weak negatives are added \"as additional $y_l$ to be paired with the original $y_w$\". This means we **add** something like these to the original dataset: $(y_w^1, y_l^2), (y_w^2, y_w^4), (y_w^3, y_l^1)$.\n\n* For DPO-PL, \"as additional negatives ranked after the $y_w$ and $y_l$\". This means we **replace** the original dataset with something like ${(y_w^1, y_l^1, y_w^2), (y_w^2, y_l^2, y_l^1), (y_w^3, y_l^3, y_l^2)}$. The K-wise ranking information required by DPO-PL can be implicitly inferred here. For example, for $(y_w^1, y_l^1, y_w^2)$, the ranking is $y_w^1 > y_l^1 > y_w^2$.\n\nWe do not consider InfoNCA because the purpose of Table 4. is to study if EPA's main advantage comes from introducing more computation alone or being the product of a better preference model. However, InfoNCA is not based on a \"preference model\". Therefore, we do not think it is relevant for the purpose here.\n> Concern #6: more data points regarding the computational cost.\n\nWe provide our actual training time to support our linear-time-complexity argument in sec 5.2.3:\n| #resp | setting | training time |\n|-------|---------|---------------|\n| 8     | 1:3:4   | 44h41min      |\n| 6     | 1:3:2   | 33h51min      |\n| 4     | 1:1:2   | 23h0min       |\n| 2     | 1:1:0   | 12h53min      |\n\n> Concern #7: the benefit of weak negatives\n\n(please refer to our response to reviewer RQg9's concern#2)", "claims_and_evidence": "* **Non-unique minimums of DPO training**.  As argued in this paper, one main problem of DPO is that its minimizers are not unique. Reading their proposition B.5, it is based on the assumption that some $y^*$ were never sampled within the preference dataset. Then one can build a translated reward model by adding an arbitrary $A(x)$ to the target RM for any $y \\neq y^*$. I don't understand why this is a problem. If $y^*$ is never sampled in the preference dataset, then it is an issue with the dataset instead of with the DPO algorithm.\n\n* Can the authors explain why EBM still have unique minimizers when some $y^*$ never appear in their training data? What about the approximation, EPA? Does it guarantee unique minimizers?\n\n* **Difference between EPA and InfoNCA / RPO** The proposed EPA method is very similar to InfoNCA [1] and Multi-Response RPO [2], except that the weak responses come from other prompts. Can the author further clarify the differences? Intuitively, I am not convinced that adding responses from irrelevant prompts will add any benefit to model training.\n\n[1] Noise Contrastive Alignment of Language Models with Explicit Rewards\n\n[2] Reward-aware Preference Optimization: A Unified Mathematical Framework for Model Alignment", "methods_and_evaluation_criteria": "For the proposed methods, please see \"Claims and Evidence\".\n\nThe evaluation criteria are standard for comparing preference optimization techniques.", "theoretical_claims": "I checked the proof of Proposition B.5, which is correct. However, as discussed in \"Claims and Evidence\", I don't agree with the implications of that proposition.", "experimental_designs_or_analyses": "In general, this paper presents comprehensive studies to understand the impact of the proposed approach. \n* Figure 2 is pretty good. It shows that EPA achieves a better reward-KL tradeoff compared to DPO.\n* Table 3 studies the impact of the number of weak negative samples.\n\nSome questions,\n* For Table 2. They fixed the same $\\beta = 0.01$ for all methods. This feels problematic, cuz different methods might have different optimal $\\beta$. How $\\beta = 0.01$ is selected is not mentioned either.\n* In Table 4, how are the weak negatives added to DPO or DPO-PL training? Can you also include InfoNCA for a comparison?", "relation_to_broader_scientific_literature": "This paper aims to push forward preference optimization algorithm, which could be useful in many problems like LLM post-training and off-policy RL.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "This paper is clearly written. \n\nI especially like how they conduct multiple ablation studies to investigate the performance of each element in their algorithm.\n\nOne weakness is the proposed approach incurs larger computational and memory costs due to the additional weak negative samples. Can the author present specific experimental data points or analysis regard them?", "other_comments_or_suggestions": "None", "questions_for_authors": "As mentioned before, my main concern is that I am not convinced that adding responses from irrelevant prompts will add any benefit to model training.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uJC8oI8eTc", "reviewer": "Reviewer_xiBU", "summary": "The authors highlight a fundamental issue with Bradley-Terry DPO, namely the lack of uniqueness in its solutions. To address this, they propose a novel approach based on energy-based models, termed Energy Preference Alignment. They demonstrate that their model overcomes the non-uniqueness problem inherent in Bradley-Terry DPO. Additionally, they approximate the model’s objective to develop a practical algorithm, which they empirically validate, showing its effectiveness compared to other DPO-based methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Concern #1 (also raised by reviewer ZCWk): generalizability to more datasets and more base models\n\nAs this is a point mentioned by multiple reviewers, we managed to run quick experiments on another relatively small dataset and on another base model. Although the experiments can never be exhaustive, we believe the results (as follows) should be additional evidence for the generality of our findings. We will include them in the final version of the appendix.\n\n* dataset: \n  - source: https://huggingface.co/datasets/argilla/distilabel-intel-orca-dpo-pairs\n  - reference: Álvaro Bartolomé Del Canto et al., 2024; https://github.com/argilla-io/distilabel\n  - description: a cleansed version of the widely used dataset (Intel/orca_dpo_pairs)\n\n* results: (MT-Bench scores for epoch#1/#2/#3)\n| base model | EPA             | DPO            |\n|------------|--------------------|----------------|\n| Llama3-8B     | 6.84/6.98/**7.04** | 6.94/6.83/6.71 |\n\n> Concern #2: A.5 requires an assumption of the non-negativity of $\\pi_{\\text{ref}}(x, y)$\n\nThank you for the suggestion. Yes, the assumption is required, which is true for both DPO (as explicitly stated in (Rafailov et al., 2023)) and EPA. Although this is a mild assumption that does not affect the main theories of our paper, we will add this to the final version of the appendix for comprehensiveness.\n> Concern #3: Does DPO-PL suffer from the same issue as DPO, particularly in terms of the non-uniqueness of solutions?\n\nYes, we think so. The proof will be analogous to that of DPO.  It is not difficult to show that proposition B.5 is also true for the Plackett-Luce Model by only checking that the reconstructed reward $\\tilde{r}_{\\hat{\\theta}}$ also leads to the same expected likelihood of data. We will add the exact proof to the final version of the appendix.\n\nBesides the proof per se, there is an intuitive understanding of why this is the case. As we have mentioned in the first paragraph of subsection 3.1., the Plackett-Luce Model also just models human preference among a **finite** number of candidates. This is problematic when the actual space of candidates is **infinite**.\n\n> Concern #4 (also raised by reviewer PHXo): detailed differences between EPA and other multi-response losses such as DPO-PL, infoNCA, etc.\n\nWe are thankful that some reviewers point out the similarity shared by EPA and other multi-response losses. This is exactly the reason why we include many such losses as baselines for comparison in Table 2. However, for a clearer description of the differences, we hope the following table can suffice. We will add it to the final version of the paper.\n\n| name | probalistic model        | data                                                                  | loss |\n|------------|--------------------------|-----------------------------------------------------------------------|------|\n| EPA        | IPM                      | $(y_w, y_{l_1}, y_{l_2})$ and $(y_w > y_{l_1})$ and $(y_w >y_{l_2})$     | $-\\log\\frac{\\exp(r_{\\theta}(y_w))}{\\exp(r_{\\theta}(y_w)) + \\exp(r_{\\theta}(y_{l_1})) + \\exp(r_{\\theta}(y_{l_2})) + \\exp(r_{\\theta}(y_{wk_1})) + \\exp(r_{\\theta}(y_{wk_2}))}$     |\n| DPO-PL     | Plackett-Luce Model       | $(y_w, y_{l_1}, y_{l_2})$ and $(y_w > y_{l_1} > y_{l_2})$       | $-\\log\\frac{\\exp(r_{\\theta}(y_w))}{\\exp(r_{\\theta}(y_w)) + \\exp(r_{\\theta}(y_{l_1})) + \\exp(r_{\\theta}(y_{l_2}))} - \\log\\frac{\\exp(r_{\\theta}(y_{l_1}))}{\\exp(r_{\\theta}(y_{l_1})) + \\exp(r_{\\theta}(y_{l_2}))}$    |\n| InfoNCA    | EBM for the ideal policy | $(y_w, y_{l_1}, y_{l_2})$ and $(y_w > y_{l_1})$ and $(y_w >y_{l_2})$ | $-\\log\\frac{\\exp(r_{\\theta}(y_w))}{\\exp(r_{\\theta}(y_w)) + \\exp(r_{\\theta}(y_{l_1})) + \\exp(r_{\\theta}(y_{l_2}))}$    |", "claims_and_evidence": "- The theoretical results appear to be well-supported.  \n- The connection between the theoretical and practical results could be strengthened. For instance, in Theorem 3.3, the result holds only asymptotically as both M and N approach infinity. Providing insights into the rate of convergence would make the theoretical findings more informative.  \n- The practical evaluation seems limited, as the proposed approach was tested on only one dataset. Expanding the experiments to multiple datasets would help better assess its generalizability.", "methods_and_evaluation_criteria": "- The theoretical results appear to be valid.  \n- The current practical evaluation is insufficient and could benefit from further validation.", "theoretical_claims": "I thoroughly checked the proof of B.5 but only skimmed the rest.", "experimental_designs_or_analyses": "See above.", "relation_to_broader_scientific_literature": "- The authors discuss various related works, particularly different objectives for DPO. \n-The final algorithm can be interpreted as DPO-PL with a technique to expand the dataset for K-wise comparisons. Specifically, given a dataset with k responses, the approach involves adding unrelated k' responses to them from other queries and then solving the problem. This formulation effectively leads to the final practical objective derived from DPO-PL.", "essential_references_not_discussed": "Not to my knowledge.", "other_strengths_and_weaknesses": "See above!", "other_comments_or_suggestions": "In **A.5**, I believe the argument relies on the condition that **\\(\\pi_{\\text{ref}}(x, y^*)\\) must be non-zero** for the reasoning to hold.", "questions_for_authors": "- Does DPO-PL suffer from the same issue as DPO, particularly in terms of the non-uniqueness of solutions?\n- Could you clarify the relationship between the proposed approach and DPO-PL?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1hlAByI0DA", "reviewer": "Reviewer_ZCWk", "summary": "This paper identifies a major limitation in Direct Preference Optimization (DPO): the underlying Bradley-Terry Model (BTM) does not always guarantee a unique Maximum Likelihood Estimator (MLE), leading to suboptimal alignment in Reinforcement Learning with Human Feedback (RLHF).To address this, the authors propose an Energy-Based Model (EBM), termed the Infinite Preference Model (IPM), which inherently ensures a unique MLE. They introduce Energy Preference Alignment (EPA)—a contrastive loss function that better enforces the required slope-1 linearity between learned and true rewards.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Concern #1: generalizability to more datasets and more base models\n\n(Please refer to our response to reviewer xiBU for our additional experiments using a different dataset and a different model)\n> Concern #2: limited discussion on computational trade-offs\n\nThank you for pointing out the limitation of this paper. As the main focus of this paper is on the theoretical benefit of using our EBM instead of BTM to model human preference, we have not included many possible further analyses regarding the computational cost of EPA (which is one among many other possible ways to fit the EBM). However, there is an explicit question raised by reviewer RQg9 (Concern#5) about a computational comparison between DPO and EPA. We hope our response there can provide additional insights.\n  \n> Concern #3: \"Could you clarify how robust that uniqueness advantage remains if your negative sampling is incomplete or unrepresentative (e.g., limited coverage in real-world data)? Specifically, do suboptimal negative samples risk undermining uniqueness—leading to the same drawbacks that can occur under BTM—and, if so, how severe is the impact?\"\n\nThank you for raising these very interesting questions that we think suffice for many future research directions. \n\nFrom a general point of view, we agree that suboptimal negative samples do risk undermining the uniqueness. Although there are no designated experiments in the paper to show this, we do think there are some related observations in the paper. \n\n* Observation 1. In Figure 2.(b), EPA also degenerates (slightly) after some epochs, which means the dataset might still be not enough. \n* Observation 2. In Table 5, as we stated in the paper, the fact that there are two tricks (adding a margin and using a dynamic weight on the loss) that can further boost the performance of EPA means it is empirically not perfect. \n* Observation 3. In Table 1, the slope-1 linear regression error is non-zero.\n\nHowever, in all these cases, we see EPA is still better than DPO. Therefore, although EPA is not robust enough, it is safe to say that it is more robust than DPO. In other words, the error caused by using EPA to estimate the unique MLE of the EBM is milder than the intrinsic flaw of BTM of not being guaranteed to have a unique MLE in the first place.", "claims_and_evidence": "The paper’s claims are well-supported by a mix of theoretical proofs, empirical benchmarks, and ablation studies. The primary claims regarding the limitations of BTM, the uniqueness of EBM’s MLE, and EPA’s improved alignment are convincingly demonstrated.\n\nMinor Weaknesses:\n1.Higher computational cost (acknowledged but not mitigated).\n2.Tested mainly on Mistral-7B (broader validation needed).", "methods_and_evaluation_criteria": "The paper employs a well-justified Energy-Based Preference Model (EBM) to address Direct Preference Optimization (DPO) limitations, with strong theoretical support (Theorem 3.1). The Energy Preference Alignment (EPA) loss effectively enforces slope-1 linearity, outperforming DPO, IPO, KTO, and NCA.", "theoretical_claims": "The paper's theoretical claims are largely well-justified with rigorous mathematical proofs. Theorem 3.1, which guarantees the uniqueness of the Maximum Likelihood Estimator (MLE) for the Infinite Preference Model (IPM), is correctly derived and logically sound. The equivalence between slope-1 linearity and the minimizer of the RLHF loss (Theorems A.3 and A.4) follows standard derivations in reinforcement learning theory and is correctly structured. The proof of Proposition B.5, which demonstrates the non-uniqueness of the MLE in the Bradley-Terry Model (BTM) under certain conditions, is valid and aligns with prior work on ranking models.", "experimental_designs_or_analyses": "The experimental design is generally sound, with a clear methodology comparing the proposed Energy Preference Alignment (EPA) loss against strong baselines, including Direct Preference Optimization (DPO), IPO, KTO, and NCA. The use of intrinsic (Pearson correlation, slope-1 regression error) and extrinsic (MT-Bench, Alpaca-Eval 2.0) evaluation metrics provides a well-rounded assessment of alignment performance.", "relation_to_broader_scientific_literature": "The paper builds on the foundation of preference optimization for RLHF, particularly Direct Preference Optimization (DPO), which was introduced as a reward-model-free approach to align preference. It identifies a major limitation of DPO stemming from the Bradley-Terry Model’s (BTM) non-uniqueness of the Maximum Likelihood Estimator (MLE), a problem well-documented in ranking literature. By introducing the Infinite Preference Model (IPM), an energy-based alternative, the paper aligns with prior work in energy-based modeling and construct the Energy Preference Alignment (EPA) loss.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Strengths:\n1.Clarity of Motivation: The paper clearly explains the limitations of BTM and how the proposed EBM circumvents them, making a compelling argument for its adoption.\n\n2.Robust Evaluation: The authors conduct extensive experiments, comparing against strong baselines, and include multiple evaluation perspectives.\n\nWeaknesses:\n1.Lack of experiments on diverse model architectures beyond Mistral-7B limits the generalizability of its findings.\n\n2.Discussion on computational trade-offs is relatively brief.", "other_comments_or_suggestions": "None", "questions_for_authors": "Could you clarify how robust that uniqueness advantage remains if your negative sampling is incomplete or unrepresentative (e.g., limited coverage in real-world data)? Specifically, do suboptimal negative samples risk undermining uniqueness—leading to the same drawbacks that can occur under BTM—and, if so, how severe is the impact?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fWHuEZOTtI", "reviewer": "Reviewer_RQg9", "summary": "This paper introduces a novel alternative to DPO for training LLMs, using an energy-based preference model (EBM). The proposed method, Energy Preference Alignment (EPA), semmly guarantees a unique MLE)and consistently outperforms DPO in empirical benchmarks, providing better alignment with human preferences and faster convergence during training.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Concern#1: Does this imply that the proposed training objective relies on a perfectly clean dataset??\n\nThank you for raising this interesting question. No, it is not a requirement to have a perfectly clean dataset. The reason for the weak negative samples is to provide high variance. Even if some $y_w^j (j \\neq i)$ is accidentally better than $y_w^i$ in the training dataset, it should not be a problem. However, if all or too many such $y_w^j$ becomes better than $y_w^i$, we might end up with lower variance as a result, which is not desired according to the theorems (see our response to concern#2 for more information about weak negatives and the variance of negatives).\n\n> Concern#2: What is the rationale behind computing the reward for irrelevant x and y pairs? Could the authors provide a more solid theoretical justification for introducing negatives in Eq. (11) and Eq. (12)?\n\nThe rationale, as mentioned in the response to Concern#1, is that the irrelevant x and y pairs can provide high variance. The premise of Theorem 3.2 and Theorem 3.3 only states that the Var[Y|Z] has to be positive. However, from the proof of Theorem 3.2, we can also see that the value of Var[Y|Z] is positively associated with the 2nd-order derivative of the energy discrepancy. Therefore, the higher variance will lead to a more convex Energy Discrepancy functional/loss.\n\nWe also discuss their intuitive usefulness as a regularizer in section 4.3. The negative gradients in DPO might push the policy parameters towards unexpected directions because they are likely not pointing to the RLHF minimizer. Therefore, the additional negative gradients introduced by the weak negatives can alleviate this issue.\n> Concern#3: does the performance improvement over DPO depend on the use of a large batch size?\n\nNo, because we use 64 as the global batch size for all the models and methods (including DPO) reported in the paper. As for why 64 in particular, this is because it is the highest possible number we can set to prevent our most memory-consuming setting (i.e., EPA's 1:3:10 setting in Table 3) from OOM. \n\n> Concern#4: sampling more mismatched responses from the training data instead of the minibatch.\n\nWe believe an irrelevant response from a mini-batch is not meaningfully different from an irrelevant response elsewhere, because the mini-batches are randomly sampled from the dataset in the first place. Also, as we mentioned in the response to Concern#2, the value of weak negatives is from the high variance as opposed to where they are sampled from.\n\n> Concern#5: EPA and DPO under the same training time?\n\nThe x axis in Figure 2.(b) is proportional to the number of prompts (ie., x in the datasets). However, since EPA (1:1:2) has exactly twice the amount of the responses (ie, y in the datasets) used in DPO, we think scaling the dynamics curve of EPA rightward by a coefficient of 2 is close to what you ask for. In such a graph,  we do not think EPA achieves faster convergence compared to DPO.  However, we also do not intend to make such a claim in the paper. Instead, we do agree the computation cost of EPA is its disadvantage compared to DPO. \n\n> Concern#6: more benchmark evaluations, such as Arena-Hard\n\nThank you for the suggestion. We quickly ran Arena-Hard for EPA against the two most important baselines: DPO and DPO-PL, using the same checkpoints we use in Table 2. The results are as follows. And, we can see that EPA is still better. We will add them to the final version of the paper. However, we believe the widely used MT-Bench and Alpaca-Eval 2.0 are enough for most of the experiments in the paper.\n\n| Training Data | Method | Arena-Hard (Win Rate, %) |\n|---------------|--------|--------------------------|\n| UF-binarized  | DPO    | 12.0                     |\n| UF-binarized  | EPA    | 16.3                     |\n| UF-all        | DPO-PL | 13.0                     |\n| UF-all        | EPA    | 16.9                     |", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Please refer to the questions 1&2.", "experimental_designs_or_analyses": "Please refer to the questions 3-6.", "relation_to_broader_scientific_literature": "Please refer to the questions.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n\n1. The combination of strong and weak negatives in the EPA loss function shows a novel and effective way to improve the model’s performance, as supported by ablation studies.\n\n\nWeaknesses:\n\nRelative weak experiments. Please refer to the questions.", "other_comments_or_suggestions": "Please refer to the Questions.", "questions_for_authors": "1. My primary concern and question lie in Eq. (11) and Eq. (12). The authors introduce cross-pairs of responses between mini-batches to increase the number of negatives (mismatched responses, when index $i ≠ j$). Does this imply that the proposed training objective relies on a perfectly clean dataset? In other words, does $ r_\\theta(x^i, x_w^i) $ always need to be greater than $ r_\\theta(x^i, x_w^j) $ and $ r_\\theta(x^i, x_l^j) $ when $ i \\neq j $?\n\n2. What is the rationale behind computing the reward for irrelevant $x $ and $y$ pairs? Could the authors provide a more solid theoretical justification for introducing negatives in Eq. (11) and Eq. (12)?\n\n3. Could the authors provide an ablation study regarding the batch size? I noticed that the global batch size is set to 64—does the performance improvement over DPO depend on the use of a large batch size?\n\n4. In the current experiments, the introducation of more negatives are within a mini-batch. Could the authors explore the possibility of randomly sampling more mismatched responses from the training data to increase the number of negatives in the mini-batch, thereby achieving further performance improvements?\n\n5. Furthermore, could the authors provide a performance comparison between EPA and DPO under the same training time? Can Figure (b) be interpreted as showing that, given the same computational budget, EPA achieves faster convergence and performance improvement compared to DPO?\n\n6. Additionally, some performance results in the current experiments seem relatively weak. Only the AlpacaEval and MT-Bench results are reported. I would expect to see more benchmark evaluations, such as Arena-Hard.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: To address the limitation that the DPO loss may admit multiple minimizers, this paper introduces an energy-based preference model (EBM) that guarantees a unique maximum likelihood estimator (MLE), inherently satisfying the desired linearity condition. To demonstrate the practical benefits of replacing DPO with the proposed EBM in offline alignment, the authors adopt a simple yet scalable objective function inspired by recent advances in EBM training, which they term Energy Preference Alignment (EPA). Empirical results on standard benchmarks show that EPA consistently outperforms DPO. The paper received four reviews. The authors provided rebuttals to address the concerns raised by the reviewers. Following the rebuttal and discussion, all four reviewers reached a consensus to accept the paper, recognizing its innovation, strong theoretical justification, and promising empirical results. The Area Chair concurs with the reviewers’ assessment and recommends acceptance.
Decision: Accept (poster)