Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO
Authors: Kaiyang Guo
Yinchuan Li
Zhitang Chen
Venue: NeurIPS 2025 poster ()
Keywords: Preference optimization
learning from feedback
language model alignment
Primary area: general_machine_learning

Abstract: Direct alignment methods typically train large language models (LLMs) by contrasting the likelihoods of preferred and dispreferred responses. While effective at capturing relative preferences, these methods are widely observed to suppress the absolute likelihoods of example responses. As a result, aligned models can  deviate from expected patterns, exhibiting reward‑hacking effect even without an explicit reward model. This fundamental limitation of contrastive alignment, termed likelihood underdetermination, motivates us to revisit direct preference optimization (DPO)—the seminal direct alignment method. Interestingly, we show that the DPO loss admits a principled decomposition. The reformulated loss not only extends naturally to a broader range of feedback types, but also unveils the root cause of likelihood underdetermination. Specifically, we identify that standard DPO implicitly oversimplifies a regularizer in the reformulated loss; restoring this full term effectively resolves the underdetermination. Building on these insights, we introduce PRoximalized PReference Optimization (PRO), a unified alignment method that accommodates  diverse feedback types while eliminating likelihood underdetermination through an efficient approximation of the full regularizer. Empirical evaluations demonstrate the consistent superiority of PRO over existing methods across pairwise, binary and scalar feedback.

Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO
Kaiyang Guo"
Yinchuan Li
Zhitang Chen
Huawei Noah's Ark Lab
Abstract
Direct alignment methods typically train large language models (LLMs) by con- trasting the likelihoods of preferred and dispreferred responses. While effective at capturing relative preferences, these methods are widely observed to suppress the absolute likelihoods of example responses. As result, aligned models deviate from expected patterns, exhibiting reward- haad-hhhkig effect even without explicit reward model. This fundamental limitation of contrastive alignment, termed likelihood underdetermination, motivates us to revisit direct preference optimization (DPO)-the the seminal direct alignment method. Interestingly, that the DPO loss admits a principled decomposition. The reformulated loss not only extends naturally a broader range of feedback types, but also unveils the root cause of likelihood underdetermination. Specifically, we identify that standard DPO implicitly oversimplifies a regularizer in the reformulated loss; restoring this full term effectively resolves the underdeterminnation. Building on these insights, we introduce PRoximalized PReference Optimization (PRO), unified alignment method that accommodates diverse feedback types while climinating likelihood underdetermination through an efficient approximation of the full regularizer. Em- pirical evaluations demonstrate the consistent superiority of PRO over existing methods across pairwise, binary and scalar feedback.
show
Introduction
The subtlety and complexity of human values make curating supervised datasets for large language model (LLM) alignment formidable challenge. To overcome this difficulty, learning from feed- back leverages preference signals to guide models distinguishing desirable from undesirable outputs. This methodological shift facilitates scalable data acquisition for efficient alignment, while concurrently propelling the evolution techniques beyond traditional supervised fine-tuning.
Initially, reinforcement learning from human feedback (RLHF) converts pairwise preference feedback into continuous rewards, which are then used to train LLM via reinforcement learning algorithms [122. RLHF has shown empirical success. However, it introduces significant computational overhead, due to the need for an auxiliary reward model and on on-policy sampling during training. Moreover, imperfect proxy, the reward model may yield unreliable evaluations when confronted with responses outside its training distribution, making RLHF vulnerable to reward hacking [37]]
Direct preference optimization (DPO) sidesteps the explicit reward modeling, and learns directly from offline preference data [8] Its core methodology involves constructing contrastive loss, that maximizes the likelihood differences between preferred and dispreferred responses. Owing to the simplicity, DPO has inspired the development of numerous contrastive approaches [9 However, an unanticipated phenomenon frequently arises in DPO: the likelihoods of both preferred and dispreferred responses decrease after alignment 15 This decline inadvertently encourages
"Corresponding to: guokaiiyandhuuvveicccom
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

the generation of cuttof-diisributioon responses. suggesting persistent reward hacking even in the absence of an explicit reward model [[6]. Recent investigations into this phenomenon have primarily focused on the training dynamics of DPO and its generalized variants. Proposed explanations include embedding similarity between paired responses [5], asymmetric update ratios for response probabilities [17], and the "squeezing effect" produced by softmax when applying gradient ascent to dispreferred responses [18]. While these studies offer valuable insights, they largely overlook the intrinsic limitation of the loss function itself. In particular, when attention is limited to relative likelihood differences, the contrastive loss becomes insensitive to concurrent decreases or increases in absolute likelihoods. We characterize this issue likelihood underdetermination. While several works also notice and attempt to remedy it, they incorporate additional supervised or regression which may unintentionally compromise the original intent of alignment.
signals into DPO 12 14,
The aforementioned line of research is primarily grounded on pairwise feedhack. Meanwhile, parallel studies have also explored alternative types of feedback for LLM alignment. Considering that response-pair annotation demands more effort than single-response evaluation, Kahneman-TTvrsky optimization (KTO) constructs separate utility functions for desired and undesired responses [19]. Conversely, in scenarios where finer-gran ree-ggraieed scalar feedback is available, noise contrastive alignment (NCA) formulates classification task to capture the varying degree of desirability for each labeled response [3]. Both KTO and NCA depart from the contrastive framework. Although they can either directly under certain assumption be adapted to pairwise feedback, KTO does not support scalar feedback, and NCA is incompatible with binary feedback. Currently, there is still no unified approach that seamlessly accommodates all these feedback types.
In this work, we revisit DPO, the seminal method in field of direct alignment. Through a theoretical reformulation of its loss, we demonstrate that DPO inherently supports heterogeneous feedback and uncover new insights into the underlying cause of likelihood underdetermination. Leveraging these findings, we propose practical approach that both mitigates underdetermination and unifies alignment across diverse feedback types, thereby realizing the best of both worlds pursued in prior studies. Specifically, our contributions are as follows:
DPO Reformulation We show that the DPO loss admits a decomposed reformulation, which consists of separate optimizer and regularizer terms. The optimizer reorganizes pairwise feedback into a pointwise signal, naturally extending applicability wider range of feedback types. The regularizer is independent of the preference label, allowing for a more flexible development of sample-based variant.
Origin of Likelihood Underdetermination We find that the standard DPO implementation implicitly oversimplifies the regularizer introduced in the reformulation. Importantly, once the full regularizer is restored, any optimal solution to the reformulated loss-if it exists no longer suffers from likelihood underdetermination.
Practical Approach Directly computing the full regularizer is intractable, and an optimal solution may not exist. To overcome these challenges, we propose PRoximalized PReference Optimization (PRO), which employs a hyper-response mechanism for efficiently approximating the regularizer, and guarantees the existence of an optimal solution whenever its hyperparameter is properly chosen.
The experiments show that PRO mitigates likelihood underdetermi- nation, performs effectively across diverse feedback types, and achieves performance compa- rable to or better than DPO and several other methods designed for specific feedback types. Remarkably, even the challenging scenario with extremely imbalanced binary feedback (desired:undesired 1:100), PRO demonstrates comparable performance to that obtained with fully balanced feedback.
Empirical Evaluation
2 Preliminaries
In LLM alignment, preference data is commonly collected through two steps: prompting a baseline model with inputs xr to generate response pair y1,32 p(ylær); human annotators labeling the generated responses to indicate which is preferred. Let yeu > yı jc denote the pairwise preference feedback, where You and y are preferred and dispreferred responses amongst yı and y/-
RLHF employs reward modeling to infer scalar reward signals from pairwise preference data D (6) ( {2(9) you Vi Ji=1' These reward estimates subsequently used to fine-tune LLMs through
2

---

reinforcement learning. Recently, DPO 8 circumvents the need for explicit reward modeling, by recognizing that the optimal solution to the RL problem possesses closed-form relationship with its underlying reward function. This connection allows the reward model to be expressed as:
Ne
ro (2, y) log
(
where is the LLM to be fine-tuned, and Teef is fixed reference model that serves as regularization anchor during optimization. Alignment thus performed by fitting the LLM-induced reward model to preference data via the following loss:
DP (T: Tef -- -E DDD log a (ro(z; Mee - re( (n)
where the hatted notation LDP indicate that the loss is computed from preference samples.
DPO has been widely adopted in LLM alignment (20H24], yet an overlooked issue is likelihood underdetermination of its loss function. Specifically, when log (yw (z) and log no (T(n)a)) are shifted
by a same constant, the loss value remains unaffected. Furthermore, as indicated by its gradient:
V VpC (T:: meet) (e.... BDD (re - yw (Ve log me Vg log mo((n)) ,1)
importance weight
the importance weight for model update approaches zero whenever the relative difference log Th (yw (r) log (g) is sufficiently large, regardless of the absolute probability values. Such characteristics can substantially hinder effective alignment: After pretraining supervised fine- tuning, the values of log Te (yu (r) and log T (y xx) are markedly higher than those assigned to meaningless sentences. As the DPO loss focuses solely on the relative difference between log and log p(n [x), their absolute values are prone to decrease due catastrophic forgetting, and there is no incentive to increase them again once the relative difference has been sufficiently enlarged.
Notations Without loss of generality, we consider single prompt and omit for brevity in the following. Let y be the set of all possible responses, and supp(-) the support of distribution.
Theoretical Re-Examination of DPO
At first glance, it seems that the contrastive nature of pairwise feedback restricts DPO comparing only the relative likelihoods responses. However, as we show in Section 3.T the DPO loss can be reformulated into decomposition that explicitly accounts for the absolute likelihoods of labeled responses. This reformulation not only reveals that DPO can naturally accommodate other types of feedback, but also lays the foundation for more flexible sample-based loss variants. Building this insight, Section 3.2 identifies the root source of likelihood underdetermination in standard DPO, which ultimately motivates our proposed approach in Section 4
3.1 Reformulation of Population-Basse DPO
To more clearly elucidate the underlying properties of DPO, consider its population-based loss:
LDPO(TT; Teet) = - E -ar-p p(yn > y2) log (ro(u)) - ro(v)))
While this loss is not directly computable due to the inaccessibility of true preference probability, nevertheless allows us examine DPO from novel perspective, articulated in the theorem below.
Theorem 3.1. The population- based DPO loss is equivalent the following one, that they share same gradient:
+ By wara D
1
DKi. (B (B())
((((ro(ui)-ro(ue))
LaDPo(Te:T -3 (s(y) log To [v)
optimizer
regularizer
where B denotes Bernoulli distribution,
s(y) = Eyen (ply > v\1 -
a score function indicating the extent to which y is favored across other responses and satisfies Bygn[s(y)) 0.
3

---

The reformulation decomposes the DPO loss into an optimizer and regularizer. Upon examination of these components, we identify two attractive properties:
The optimizer reorganizes pairwise feedback into a pointwise signal syy), with which log Te (v) independently optimized for each response. This property naturally extends the applicability of LaPoo. For instance, given -1/2 v sy) < 1/2, we can interpret sy) as the expected value of a Bernoulli distribution, with binary feedback in {-1/2,+14 sample for loss evaluation. Alternatively, s(y) can be viewed as the expectation of a continuous reward distribution, where sealar feedback represents realized reward samples during training.
serving as its empirical
The optimizer relies on preference feedback, whereas the regularizer operates independently of such information. Given the limited availability of preference feedback in practice, the optimizer must be estimated from a finite dataset. In contrast, the regularizer can be applied to an expanded set of responses, irrespective of preference labels. This decomposition therefore provides greater flexibility in developing sumple-based loss.
These two properties play pivotal roles in this work. The first one enables devising a unified alignment loss for diverse feedback types. The second property offers an elegant way to understand and resolve likelihood underdetermination, as detailed in the following section.
3.2 The Completeness of Regularizer Matters in Sample-Based Based Loss
For practical use, it is essential to develop a sample-hased loss, whose computation only requires the limited feedback data. The most straightforward strategy is to estimate both the optimizer and regularizer in eDPO using the labeled responses As can be verified by applying Theorem 3.1 in reverse, this substitution recovers the sample-based DPO loss. However, there appears to be contradiction regarding the existence of likelihood underdetermination. On one hand, the gradient of sample-based DPO in ① includes an importance weight, causing it to vanish whenever the relative likelihood difference between response pair is sufficiently large. On the other hand, the optimizer in eDPO directly evaluates the absolute log- probabilities of labeled responses; limited samples, its gradient remains free of any importance weighting. Considering that eDPO incorporates an additional regularizer, it is plausible that this term is responsible for the likelihood underdeterminatioon presented in sample-based DPO. We next investigate its effect.
estimated with
A key observation of the regularizer is that, when p(y) 0 for all y, the regularizer effectively constrains अy around Tief, and its value becomes zero only if Te = Tref. In other words, the regularizer is well-defined, albeit in contrastive form analogous to DPO. However, it is easy to verify that, when estimated with a subset of responses, the regularizer revives the underdetermination issue Moreover, indicated by the recovery to sample-based DPO, this underdetermination dominates the optimizer's 's effect on absolute likelihoods, rendering the overall loss function underdetermined.
In fact, the regularizer itself is independent of preference labels, thus need not be restricted to labeled responses. This motivates us to study whether likelihood underdetermination can be addressed by retaining the full regularizer in sample-hased loss. Formally, define the sample-based eDPO loss as:
D { w21 DKT.
O ==- (5())- log (v)]+
B ((((ro(v))-ro
2 En
where ÿi denotes the empirical response distribution derived from the preference dataset, in contrast to the full response distribution μ. The coefficient a > is newly introduced for general tradeoff between preference optimization and regularization. The empirical score š(y) is given by
Bigina [ply > v'}
for pairwise feedback
5(v)
b(y) Ey-is [b()]
for pointwise feedback
That means replacing js and by their empirical counterparts estimated from the dataset. 2 Since Σ, (g) 1, any uniform likelihond reduction (or increment) within subset must he offset by an opposite change on its complement. However, the regularizer only compares likelihoods inside the subset, but disregards its relation to the rest of Y It therefore blind to the widening probability gap between the two parts.

---

where p denotes the empirical pairwise preference and ó is the sample mean of the pointwise feedback. The structure of & for pointwise feedback designed so that By [y)] 0. mirroring the property of established in Theorem 3.1
To analyze the theoretical property of LEDPO: treat wy arbitrary distribution in A { (v) O, Vy E y and Evey (Vy) = 13. This allows to derive the necessary condition for optimality as follows.
Theorem 3.2. Let μ : ply) > for all y E Y. If an optimal solution xO" LEDPO exists, it satisfies the condition for any y E y:
π (g) o log (V)
((u) (
log
s(y).
Condition (2) can be interpreted as a weighted score matching: side acts as learned score, analogous to the empirical score & for pairwise feedback except that is derived from the LLM; The weight on the right-hand side contains μuy) and ((y), which respectively signify the strengths of the regularizer and the observed evidence.
The expectation term on the left-hand
Recall that 8(y) indicates whether y is preferred over other responses. When condition ( holds, the modeled score should reflect the preference accordingly. In particular, the sign of s(y) should determine how H² (v) deviates from (B). This relationship is formally confirmed in Corollary 3.3
Corollary 3.3. Under the preconditions of Theorem 3.2 the following results hold for a constant C:
x (v)
C,
Vy (dy) = or 8(y) 0,
(3)
Te)(y)
π (1)
> C, Vy: (ly) > 0 and B(y) > O.
(4)
" (y) <C, Ay (ky) > 0 and 8(y) < O.
(5)
Tvy(y))
Corollary 3.3 imposes an ordering among the probability updates compare to reference model. Specifically, it constrains the probability ratio of any unobserved response (i.e., one absent from the preference dataset) to fall between those of the preferred and dispreferred responses. Consequently, simultancous decrease in the probabilities of preferred and dispreferred responses would necessarily entail decrease for the unobserved responses as well, which is impossible due to the fixed total probability. This demonstrates that the absolute likelihoods of both labeled and unobserved responses can not be adjusted arbitrarily, thereby resolving the underdetermination issue. Under the guarantee, we conclude that:
The likelihood underdetermination in DPO stems from an oversimplified regularizer, and can be mitigated by restoring the regularizer to its full form.
Proximalized Preference Optimization
The analysis in previous section rest two preconditions: (i) assigns non-zero probability to all responses, and (ii) an optimal solution म' exists. While the first condition can be satisfied by presetting appropriately, computing the regularizer in LeDPO requires traversing all responses with non-zero probability under μr. Given the enormous cardinality of y, the regularizer rapidly becomes computationally intractable. We address this challenge by developing a carefully crafted approximation the regularizer in Section 4.1 The second precondition eecconditoon-existeeneee of an optimal solution- haiioon--hssaaaa been shown to always fail in sample-based DPO, referred to as degeneracy issue [10). In Section 4.2] we establish a sufficient condition that guarantces the existence of an optimal solution for the proposed loss. We further provide an explicit pairwise-feedback example satisfying this condition and show that it is directly pertinent to sample-bass based DPO.
4.1 Introducing Hyper Response for Tractable Loss Approximation
To avoid enumerating all possible responses, one may incorporate additional samples drawn from μ, together with the labeled responses, into the regularizer's 's computation. While this yields more
5

---

p(K)
p(2) ". p(w) p(ynr1) p(nu2) "
p(:)
p(2)
" p(%) p(7)) = EyexP()
Y1
Y2
yn
Yn+1
yn++2
Y/2
yn+1
Yn+2
Original Respouse Space y
Modificd Response Space Hi
Figure 1: Shaded boxes denote labeled responses; blank boxes denote unobserved responses. By aggregating unobserved responses into a single hyper response, the response space becomes compact, such that the probabilities of its elements can be enumerated.
accurate estimation than sample-based DPO, auto-regressive sampling is computationally expensive, typically 10- 100x slower than the training process itself. Moreover, this simplification again reduces to an empirical distribution with limited supports, risking recurrence of underdeterminatton..
To overcome these limitations, we introduce an approximation mechanism that evaluates the regular- izer within a compact yet complete response space. The central idea is to define hyper response, denoted by H. y. which aggregates multiple individual responses into single, indistinguishable unit. This abstraction gives rise to the following modified response space:
V'H (H) U (v y & H).
The regularizer is then computed using the probability values u(y), To(s) (v) and mer(y) exclusively over Vn. To ensure these probabilities well-defined for y = H. we extend any probability distribution p J' to D'H by setting:
p(H) Σ p(y)= =1
Σ p(y). V
(6)
Importantly, p(H) can be derived entirely from the probabilities of its complement. As illustrated in Figure when H encompasses all unobserved responses, computing p(H) requires only the proba- bilities of labeled responses, without the need for additional sampling. Moreover, this construction provides the most compact form of Yx, as it augments the labeled responses with a single hyper response. Owing these advantages, we adopt this construction the default setting.
Building on the above foundations, we now introduce the refined loss:
EpRO (Te:))t)) = - B (š(y) log To (v)] + En wan PKL
Pkk
B (5(
B ((((r0(e))-rr(0))
whose only difference from E-DPO lies in the use of yı, y/2 & μ within the regularizer. Here, & denotes that y sampled according μ over YH. Accordingly, for any function f. the expectation with respect to ~ ju reduces to Eyii (f)] =({)))))(())( + E (f)) which substantially reduces the number of terms compared to the full expectation over y. When substituting this expectation into the regularizer, f(H) corresponds to the KL term involving μ(H)) (7+) and (H). all of which can be efficiently computed via (6). Crucially, due to the aforementioned construction of H. LYRO incurs negligible additional computational cost compared Loro, requiring only lightweight operations on response probabilities already computed DPO.
Three questions naturally arise concerning ŽpRO: (i) How does its optimal solution relate to that of LaDPO? ? (ii) Does it preserve protection against underdetermination?? and (iii) What is the cost of this approximation? The following theorem and the subsequent discussion addresses these questions.
Theorem 4.1. Let H Ç y \ supp((i) and μ p(v) for all y V. If optimal solutions x" and The to LODPO and ERRO exist, they satisfy:
The (v) = (y), Wy E y H.
(7)
Σ
(1) -=
C. Σ Tmr(y),
(8)
DEH
yEH
DE
where C. is the constant defined in Corollary 3.3

---

The precondition H. S y (A) implies that the hyper response contains no labeled responses. Consequently, y H comprises all labeled responses. By (7). these responses retain the properties stated in Corollary 3.3 after training, their absolute likelihoods- not merely their relative gaps- are well determined. This protects the aligned L.M from likelihood underdetermination.
Adopting the hyper-response mechanism sacrifices the ability to distinguish among the probabilities of unobserved responses in H. However, the inability regulate the distribution over unobserved responses during alignment is a limitation shared by existing approaches. Methods such as DPO [8] KTO [91, NCA [3], and IPO [T] incorporate the probabilities only for labeled responses, leaving the rest under-specinee specified. In contrast, distinctive property of our approach is that the total probability mass assigned the elements in H. remains fixed, as shown (8). This constraint prevents any single response from continuously increasing its probability throughout training. Since most unobserved responses have extremely low initial probabilities, even moderate increases during training leave their generation likelihood minor, thereby limiting the practical impact of this limitation.
In summary, the hyper-response mechanism enables an efficient and effective approximation to Earoo With the guarantee of mitigating likelihood underdeterminnatinn, aligned LLM is successfully proximalized around reference model. We therefore refer to the proposed approach as PRoximalized PReference Optimization (PRO).
4.2 The Existence of Optimal Solution for PRO
Finally, we establish a sufficient condition under which the considered losses have optimal solutions. Noting that LaPPo is a special case of CpRO obtained by restricting H to a single response, we hereafter take LRRO as the general loss.
Recall that the feasible region of ÊpRo. = { Tty) > 0 Vy E V'H, Eyey, (ty) = 1}. is an open set. The absence of an optimal solution would imply the existence of a sequence within A whose elements approach its boundary, along which the loss function strictly decreases (see Lemma B.1 in Appendix B.4). However, as one approaches the boundary, i.. π ( + for some y, the regularizer in PRO can be shown to diverge to +oo. Thus, any overall decrease in the loss must result from the optimizer decreasing towards -0 at a faster rate.
The above observation leads us to ask whether the value of a can be adjusted so that the regularizer dominates the loss function at the boundary. so, the unbounded descent of the loss function can be prevented, thereby guaranteeing the existence of an optimal solution. This conjecture is established by the following theorem.
Theorem 4.2. Given any H C y supp(i) and μ.- == ply) > O, Vy E Yn, there a threshold oc such that, whenever a > ao, an optimal solution τhय to LPRO exists
As an illustrative example, consider the pairwise feedback setting, where we choose μ = TT as:
m(v)= (ny)
if E supp(At (11 -(y) otherwise
Here, p(y) > 0 denotes an arbitrary probability distribution over y E V/H \ supp(ft), and < 17 < 1 is preset hyperparameter. This construction serves as a general and practically effective choice: in real-world scenarios, labeled responses yield the empirical distribution P. whereas the true underlying distribution μl, from which these responses are drawn, often remains inaccessible.
Under this specification of the next theorem characterizes an admissible range of a, and further reveals a direct connection between the induced loss and sample- based DPO.
Theorem 4.3. Consider the pairwise feedback setting, where μa = j and H S y supp(pi). For any a 2 1/7², an optimal solution TH to EPRO exists. Moreover, when a = 1/12, the PRO loss is equivalent to the following one in that they share same gradient:
EPRo.P(TO; Tnf) - 1/2
By ~ plui [pp > 22) log (rgg(rrov))) ro(
A constructive choice of ao for general preference feedback is given in Corollary B.2
7

---

where
pyn yy) in y2 E supp(ii)
ply > 82)
1/2
otherwise
is an augmented empirical preference.
The PRO-P loss can be viewed as an enhanced variant of sample-based DPO, which integrates pseudo preference labels and employs the hyper-response approximation. Nevertheless, it should be noted that PRO-P is merely a special case resulting from particular choices of and a. More broadly, PRO offers general approach that accommodates diverse feedback types and permits versatile control over both the strength and distribution of regularization.
5
Experiments
Our experiments address four questions: (i) To what extent does PRO mitigate likelihood under- determination in practice? (ii) How does it compare with other alignment methods under pairwise and binary feedback? (iii) Given KTO's suitability for imbalanced binary feedback, does PRO exhibit greater robustness under severe imbalance? (iv) While NCA is specifically tailored for scalar feedback, can PRO match or surpass its performance?
We utilize two datasets to construct three types of feedback across different experimental settings. The Anthropic-HHe dataset originally comprises 170k pairwise feedback instances [25]. Following [19], split each paired response into individual ones, and convert the feedback into binary format by labeling preferred responses as desired and dispreferred ones as undesired. The UltraFeedback dataset includes 64k instructions, each accompanied by four responses annotated with scalar feedback [26] To derive pairwise version, we select the response with the highest scalar feedback as preferred and randomly choose one of the others dispreferred, following 24). A binary version is also generated from the pairwise data in manner consistent with the processing of Anthropic-HI.
In experiments, we apply PRO to each feedback type: PRO-P denotes the use ÉpRO for pairwise feedback, PRO-B and PRO-S correspond to EPRO applied to binary and scalar feedback, respectively, The hyper response is set to encompass all unobserved responses. Further implementation details (including the choice of a)) and the full experimental setup are provided in Appendices D and
5.1 Resolving Likelihood Underdeterminattion
As discussed, likelihood underdetermination often manifests as a uniform reduction in probabilities across all example responses, ultimately leading to reward hacking. Because reward hacking exerts a more immediate influence on model performance, this section focuses on its analysis. Detailed probability dynamics throughout training reported in Appendix [see Figures Flnd showing that PRO consistently increases the probabilities of preferred responses for all feedback types.
In the absence of a reward model and ground-truth rewards, we examine reward hacking through its most recognized symptom-lenge
exploitation (7) wherein models tend to produce excessively long responses after alignment. To trace the development and severity of this effect, we continuously monitor the model's average response length on test dataset during the alignment process. Simultane- ously, model performance is evaluated in terms of win rate against preferred responses, measured on dimensions of helpfulness, harmlessness and conciseness, using DeepSeek-V (31] as the evaluator. To highlight variations over time, we report relative changes in both win rate and average response length with respect to their initial evaluations.
JPO
120
DPO
<TO
ITO
SEA
RICA
190U6
MCA
79p-7
000
641
130
000
01
0053
100
Figure 2: Performance fluctuation of different alignment methods. 3 is uniformly set to O.1.
Figure 3: Results of aligning Pythia-6.9B with Anthorpic-HH.

---

Figure 2 shows the results of aligning the Pythia-6.9E model [32 with Anthropic- HH dataset. For DPO, we observe a sharp mcreasee in response length training progresses, accompanied by substantial decline in win rate. In contrast, the response lengths and win rates of both PRO-P and PRO-B remain stable throughout training, suggesting effective mitigation of reward hacking. Since KTO and NCA are derived from non-contrastive frameworks, they are expected to avoid the underdetermination issue. However, our results reveal that the fully trained KTO model still exhibits significant increase in response length, from 113.6 to 235.6, along with 6.64% drop in win rate.
We hypothesize that the degradation KTO arises from the direct use of sigmoid function its loss formulation:
LATOO (To; Reer) [e (t) Ann (3(r((z, uw) zo) Auo (B(20 - r0( ut)]],
where ÅD, Ay hyperparameters and zo is non-negative reference value. The sigmoid function tends to saturate as its input moves far from zero, causing the gradient to vanish at extreme regions. During training, particularly in the later stages of epoch, the term re (2 yw) - 2n may become strongly negative. This typically occurs because prior updates to the model, driven on other samples, potentially cause the model to "forget" prompt-response pairs it has not yet encountered during alignment. As a result, when these pairs are eventually processed, the diminished value of ro (z. Uhe) suppresses the sigmoid's gradient, thereby impeding effective learning. In contrast, NCA and the PRO series incorporate the log-sigmoid function in their loss functions. The log-sigmoid saturates only on one side-when the implicit reward becomes sufficiently positive for the preferred response sufficiently negative for the dispreferred onc. This asymmetry avoids the vanishing waiishing-graieent gradient problem inherent to the plain sigmoid and enables more stable parameter updates during training.
5.2 Performance Comparison Under Pairwise and Binary Feedback
Both PRO and the baseline methods employ 3 to regulate the degree of preference optimization; however, the optimal 3 may differ across methods. To ensure fair comparison, evaluate each method under a range of B values.
Figure 3 shows the results of aligning the Pythia-6.9B model with Anthropic-HH dataset. NCA and KTO clearly outperform DPO in win rate while keeping response length within a more reasonable range. This improvement likely stems from their con-contrastive loss formulation. Remarkably, although PRO-P remains contrastive, both and PRO-B consistently performs well. further supporting that oversimplifying the regularizer is the root cause of underdeterminationn in contrastive alignment.
Next, we align the Mistral-7B-st model 33 with the UltraFeedback dataset, conducting hyperparam- eter sweeps over the same range of B values. Each aligned model is evaluated on AlapcaEival 2 R4. MT-Bench [35] and multiple benchmarks from LM Evaluation Harness [36]. Table I summarizes the best results of each method over all / values. The results indicate that DPO outperforms KTO and NCA on AlpacaEval 2, while the reverse is true on MT-Bench. For both tasks, PRO-P and PRO-B either closely match surpass the best baseline. Similar results observed for tasks from LM Evaluation Harness: DPO outperforms KTO and NCA on ARC and TruthfulQA, but faces performance bottleneck on IFEval. The PRO methods generally exhibit performance comparable to the best baseline.
AlpacaEval 2
MT-Bench
Method
ARC
IFEval
TruthfulQA
GPQA
Avg Rank
LC (%)
WR (%)
Score
SFT
8.69
4.29
4.87
$1.54
2.40
42.23
29.02
6.0
DPO
18.36
19.80
5.52
61.77
19.22
43.45
32.04
3.3
KTO
17.39
14.60
5,98
55.38
25.69
41.00
33.04
3.9
NCA
17.21
13.43
6.03
58.62
26.43
42.35
32.14
4.0
PRO-P
20.21
19.56
6.06
61.26
29.02
43.81
32.59
2.0
PRO-B 20.46
21.48
6.06
59.81
30.13
42.72
32.81
1.7
Table 1: Results of aligning Mistral-7B-sft with UltraFeedback. Avg rank is computed by ranking the method among all competitors for each task and then averaging its ranks over all tasks.

---

5.3 Aligning with Extremely Imbalanced Binary Feedback
To further assess the effectiveness of alignment using binary feedback, we consider the challenging settings where 99% of the desired or undesired responses in Anthropic-Hit dataset are discarded. The resulting "1 desiired and "1 11%-ndesiired datasets are then used to align the Pythia-6.95 model.
On the 1 1 dataset, we conduct preliminary experiments with PRO-B and KTO using the optimal / values from Figure 3 (0.003 and 0.03, respectively). However, these configurations yield low win rates of only 5.57% and 22.56%. In addition, both aligned models produce large number of duplicate and meaningless tokens. We hypothesize that these poor performances stem from overoptimization: with substantially reduced training data, the best-performing LLM we can optimize is expected to remain closer to the reference model. To investigate this hypothesis, we first increase B by one and two orders of magnitude, but the performances still suffer.
PRO introduces an extra hyperparameter a that mediates the trade-off between optimiza- tion and regularization. We therefore tune a
Dataset
a= 2.5
a 10
a
17.5
a a= 25
1%-drrdd
5.57
35.24
57.21
52.02
and report the results in Table 2 Remark-
1%dundesire
48.87
50.64
47.83
47.94
ably, increasing a from 2.5 to 17.5 improves the win rate 57.21%, even surpassing the performance achieved with the full dataset (53.37%). This underscores the importance of in learning stability. A detailed comparison between the effects of a and | is given in Appendix
Table 2: Effectiveness of a in improving win rates (%) under extremely imbalanced binary feedback.
In contrast, the 1%-undesire dataset, PRO-B and KTO attain satisfactory win rates of 48.87% and 48.33% under the same 3 settings. Once again, increasing / fails to yield better performance, but tuning for PRO-B leads to further improvements, shown in Table These results suggest that unlearning undesired responses is more challenging than learning desired ones, yet appropriate tuning of a benefits both processes.
5.4 Aligning with Scalar Feedback
Method
N
ARC
IFEval TruthfulQA
GPQA
The raw UltraFeedback dataset provides four responses labeled with scalar feedback per instruction. Following existing work [1]. we evaluate model performance using different numbers of responses per instruc- tion, denoted by N. For N = 2, the best and random remaining response
2
59.39
27.73
43.45
31.70
NCA
4
59.61
28.96 f
45.78 1
32.14 f
2
59.47
29.31
45.90
30.80
PRO-S
4
59.47
29.43
49.45 1
32.81 1
Table 3: Results of Aligning Mistral-7B-sft. Including
more suboptimal examples improves performance.
selected.
As shown in Table 3 PRO matches or surpasses NCA, verifying its effectiveness on scalar feedback.
Besides, increasing N from 2 to 4 improves both methods across several benchmarks, suggesting that additional suboptimal examples further enhance alignment.
Discussion
While DPO has become a predominant approach for aligning LLMs, it remains limited to pairwise feedback. Additionally, the DPO loss is susceptible to likelihood underdetermination, inadvertently encouraging reward hacking. In this study, we introduced a decomposed perspective on DPO that not only reveals its potential to generalize to richer forms of feedback but also exposes the fundamental cause of likelihood underdetermination. Building on these insights, we proposed PRO, practical method unifying alignment across diverse feedback while mitigating the underdetermination issue. Experimental results demonstrated that PRO effectively mitigates length exploitation and performance degradation during alignment, and performs consistently well across diverse feedback types.
This study also opens several avenues for future works. Prior studies have proposed various im- provements to DPO 37 42 and recent work 43 shows that many of them can be equivalently realized by selecting appropriate reference models in the DPO loss. Since PRO is derived as a reformulation of DPO, it is interesting to explore how these strategies can further improve PRO's performance. Moreover, the DPO reformulation itself serves as a conceptual bridge to RLHF, as both incorporate optimizer-regularizer composition. This connection invites opportunities for gentler regularization, improved model diversity, and integration with calibrated preference models in more general alignment/poss teassstttriinggggg scenarios, as detailed in Appendix C
10

---

References
[1] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep Reinforcement Learning from Human Preferences. In Advances in Neural Information Processing Systems, 2017.
[2] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. arXiv preprint arXiv:1707.06347, 2017.
[3] Leo Gao, John Schulman, and Jacob Hilton. Scaling Laws for Reward Model Overoptimization. In International Conference on Machine Learning, 2023.
[4] Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback. arXiv preprint arXiv:2307. 15217, 2023.
[5] Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and Characterizing Reward Hacking. Advances in Neural Information Processing Systems, 2022.
[6] Nathan Lambert and Roberto Calandra. The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedback. arXiv preprint arXiv:2311.0016 2023.
[7] Lilian Weng. Reward Hacking in Reinforcement Learning. githuuu...ggitttuhoooo Nov 2024.
[8] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct Preference Optimization: Your Language Model is Secretly Reward Model. In Advances in Neural Information Processing Systems, 2023.
[9] Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. SLiC- HF: Sequence Likelihood Calibration with Human Feedback. arXiv preprint arXiv:2305.10425 2023.
[10] Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A General Theoretical Paradigm Understand Learning from Human Preferences. In International Conference on Artificial Intelligence and Statistics, 2024.
[11] Yu Meng, Mengzhou Xia, and Danqi Chen. SimPO: Simple Preference Optimization with Reference-Free Free Reward. arXiv preprint arXi::2405.14734 2024.
[12] Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive. arXiv preprint 1322: 2024.
[13] Huayu Chen, Guande He, Lifan Yuan, Ganqu Cui, Hang Su, and Jun Zhu. Noise Contrastive Aligmment of Language Models with Explicit Rewards. In Advances in Neural Information Processing Systems, 2024.
[14] Teng Xiao, Yige Yuan, Huaisheng Zhu, Mingxiao Li, and Vasant G Honavar. Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment. In Advances in Neural Information Processing Systems, 2024.
[15] Noam Razin, Sadhika Malladi, Adithya Bhaskar, Danqi Chen, Sanjeev Arora, and Boris Hanin. Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization. In International Conference on Learning Representations, 2025.
[16] Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei. Guangju Wang, Chao Yu. and Yi Wu. Is DPO Superior to PPO for LLM Alignment? Comprehensive Study. In International Conference on Machine Learning. 2024.
[17] Duanyu Feng, Bowen Qin, Chen Huang, Zheng Zhang, and Wenqiang Lei. Towards Analyz- ing and Understanding the Limitations DPO: Theoretical Perspective. arXiv preprint arXiv:2404 240..04644 2024.

---

[18] Yi Ren and Danica J. Sutherland. Learning Dynamics of LLM Finetuning, In International Conference on Learning Representations, 2025.
[19] Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Model Aligmment as Prospect Theoretic Optimization. In International Conference on Machine Learning, 2024.
[20] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Llama Herd of Models. arXiv preprint arXiv:2407.2178, 2024.
[21] An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et Qwen2 Technical Report. arXiv preprint arXiv:2407. /067/, 2024.
[22] Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, al. Mixtral of Experts. arXiv preprint arXiv:2401. 044088, 2024.
[23] Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang. David Wadden, Noah A Smith, Iz Beltagy, et al. Camels in a Changing Climate: Enhancing LM Adaptation with TÚLU 2. arXiv preprint vrXiv2231.110702 2023.
[24] Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang. Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct Distillation of LM Alignment. arXiv preprint arXiv:2310. 16944, 2023.
[25] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. arXiv preprint arXiv:2204.088 05862, 2022.
[26] Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. UltraFeedback: Boosting Language Models with High-quality Feedback, 2023.
[27] Wei Shen, Rui Zheng, Wenyu Zhan, Jun Zhao, Shihan Dou, Tao Gui, Qi Zhang, and Xuanjing Huang. Loose Lips Sink Ships: Mitigating Length Bias in Reinforcement Learning from Human Feedback. In The Conference on Empirical Methods in Natural Language Processing, 2023.
[28] Jiacheng Xu Prasann Singhal, Tanya Goyal and Greg Durrett. A Long Way to Go: Investigating Length Correlations in RLHF. arXiv, 2023.
[29] Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling Length from Quality in Direct Preference Optimization. arXiv preprint arXiv:2403. 19159, 2024.
[30] Junru Lu, Jiazheng Li, Siyu An, Meng Zhao, Yulan He, Di Yin, and Xing Sun. Eliminating Biased Length Reliance of Direct Preference Optimization via Sampled KL Divergence. arXiv preprini arXiv:2406. 10957, 2024.
[31] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. DeepSeek-V? V3 Technical Report. arXiv preprint arXiv:2412. 19437, 2024.
[32] Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O' B'Brinn, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling, In International Conference on Machine Learning, 2023.
[33] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7B. arXiv preprint arXiv:2310.06825, 2023.
12

---

[34] Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length- Controlled Alpacaeval: Simple Way to Debias Automatic Evaluators, arXiv preprint 2444:22000.04475. 2024.
[35] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging LLM-as-a-Judge with MT Bench and Chatbot Arena. In Advances in Neural Information Processing Systems, 2023.
[36] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac"h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024.
[37] Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive Preference Optimization: Pushing the Bound- aries of LLM Performance in Machine Translation. In International Conference Machine Learning, 2024.
[38] Changyu Chen, Zichen Liu, Chao Du, Tianyu Pang. Qian Liu, Arunesh Sinha, Pradoep Varakan- tham, and Min Lin. Bootstrapping Language Models with DPO Implicit Rewards. arXiv preprint 24Xx:22406609700,, 2024.
[39] Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang. Haifeng Zhang, and Jun Wang. Token- level Direct Preference Optimization. International Conference on Machine Learning, 2024.
[40] Motoki Omura, Yasuhiro Fujita, and Toshiki Kataoka. Entropy Controllable Direct Preference Optimization. arXiv preprint arXiv:2411.07595, 2024.
[41] Junkang Wu, Xue Wang, Zhengyi Yang, Jiancan Wu, Jinyang Gao, Bolin Ding, Xiang Wang, and Xiangnan He. a-DPO: Adaptive Reward Margin is What Direct Preference Optimization Needs. arXiv preprint arXiv:2410.10144 2024.
[42] Junkang Wu, Yuexiang Xie, Zhengyi Yang, Jiancan Wu, Jinyang Gao, Bolin Ding, Xiang Wang. and Xiangnan He. 3-DPO: Direct Preference Optimization with Dynamic B. In Advances in Neural Information Processing Systems, 2024.
[43] Rasul Tutnov, Antoine Grosnit, and Haitham Bou-Ammar. Many of Your DPOs are Secretly One: Attempting Unification Through Mutual Information. arXiv preprint 2501.02500000 2025.
[44] Reicchiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. WebGPT: Browser- Assisted Question-Answering with Human Feedback. arXiv preprint arXiv:21 2021.
[45] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to Summarize with Human Feedback. In Advances in Neural Information Processing Systems, 2020.
[46] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, 2022.
[47] Yuchun Miao, Sen Zhang, Liang Ding, Rong Bao, Lefei Zhang, and Dacheng Tao. Inform: Mitigating reward hacking in rlhf via information-theone reward modeling. In Advances in Neural Information Processing Systems, 2024.
[48] Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward Model Ensembles Help Mitigate Overoptimization, arXiv preprint arXiv:2310.02743 2023.
13

---

[49] Yuanzhao Zhai, Han Zhang, Yu Lei, Yue Yu, Kele Xu, Dawei Feng, Bo Ding, and Huaimin Wang. Uncertainty-Penan Penalized Reinforcement Learning from Human Feedback with Diverse Reward Lora Ensembles. arXiv preprint arXiv:2401.000245 2023.
[50] Alexandre Ramé, Nino Vieillard, Léonard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. Warm: On The Benefits of Weight Averaged Reward Models. arXiv preprint arXiv:2401. 12187, 2024.
[51] Adam Fisch, Jacob Fisenstein, Vicky Zayats, Alekh Agarwal, Ahmad Beirami, Chirag Nagpal, Pete Shaw, and Jonathan Berant. Robust preference optimization through reward model distillation. arXiv preprint arXiv:2405.19316, 2024.
[52] Tengyu Xu, Eryk Helenowski, Karthik Abinav Sankararaman, Di Jin, Kaiyan Peng, Eric Han, Shaoliang Nie, Chen Zhu, Hejia Zhang, Wenxuan Zhou, et al. The Perfect Blend: Redefining RLHF with Mixture Judges. arXiv preprint arXiv:2409.20370, 2024.
[53] Ted Moskovitz, Aaditya K Singh, DJ Strouse, Tuomas Sandholm, Ruslan Salakhutdinov, Anca Dragan, and Stephen Marcus McAleer. Confronting Reward Model Overoptimization with Constrained RLHF. In International Conference Learning Representations, 2024.
[54] Mathieu Rita, Florian Strub, Rahma Chaabouni, Paul Michel, Emmanuel Dupoux, and Olivier Pietquin. Countering Reward Over-Optimization in LLM with Cemonstration-Guidee Rein- forcement Learning. arXiv preprint arXiv:2404. 19409, 2024.
[55] Lingfeng Shen, Sihao Chen, Linfeng Song, Lifeng Jin, Baolin Peng, Haitao Mi, Daniel Khashabi, and Dong Yu. The nickle-Down Impact of Reward (In-) Consistency on RLHF. arXiv preprint arXiv:2309.16155. 2023.
[56] Paria Rashidinejad and Yuandong Tian. Sail into the Headwind: Alignment via Robust Rewards and Dynamic Labels against Reward Hacking. arXiv preprint a4Xi::2412.09544, 2024.
[57] John Schulman. Proxy Objectives in Reinforcement Learning from Human Feedback. Invited Talk at International Conference on Machine Learning, 2023. htttp::/icml.cc//vinee 2023 1t23//miited-tall/21155
[58] Seungiae Jung, Gunsoo Han, Daniel Wontae Nam, and Kyoung-Woon On. Binary Classifier Optimization For Large Language Model Alignment. arXiv preprint MxXiv:2404.0465 2024.
[59] Amos Tversky and Daniel Kahneman. Advances in Orospect Theory: Cumulative Representa- tion of Uncertainty. Journal of Risk and uncertainty, 1992.
[60] Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning, arXiv preprint arXiv:2404.05866 2024.
[61] Shitong Duan, Xiaoyuan Yi, Peng Zhang, Yan Liu, Zheng Liu, Tun Lu, Xing Xie, and Ning Gu. Negating Negatives: Alignment with Human Negative Samples via Distributional Dispreference Optimization. In Findings of the Association for Computational Linguisties: EMNLP, 2024.
[62] Xiaojian Yuan, Tianyu Pang, Chao Du, Kejiang Chen, Weiming Zhang, and Min Lin. A Closer Look at Machine Unlearning for Large Language Models. arXiv preprint arXiv:2410.08109 2024.
[63] Yuanshun Yao, Xiaojun Xu, and Yang Liu. Large Language Model Unlearning. In Advances in Neural Information Processing Systems, 2024.
[64] Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge Unlearning for Mitigating Privacy Risks in Language Models. In The Annual Meeting of the Association for Computational Linguistics, 2023.
[65] Remi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Côme Fiegel, et al. Nash Learning from Human Feedback. In International Conference on Machine Learning, 2024.
14

---

[66] Gokul Swamy, Christoph Dann, Rahul Kidambi, Steven Wu, and Alekh Agarwal. A Mini- maximalist Approach Reinforcement Learning from Human Feedback. International Conference Machine Learning. 2024.
[67] Chaoqi Wang, Yibo Jiang, Chenghao Yang. Han Liu, and Yuxin Chen. Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints. In International Conference Learning Representations, 2024.
[68] Gian Wiher, Clara Meister, and Ryan Cotterell. On Decoding Strategies for Neural Text Generators. Transactions of the Association for Computational Linguistics, 2022.
[69] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, al. DeepSeek-RI. Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv preprint arXiv:2501.12948 2025.
[70] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, al. MatpSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv preprint arXiv:2402.03300 2024.
[71] Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. OpennReasoner- Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model. arXiv preprint arXiv:2503. 2422290503.24299 2025.
[72] Ziniu Li. Can Better Cold-Start Strategies Improve RL Training for LLMs? Notion Blog, 2025. attps://tangibleepoooo0000 notion. site
15

---

Contents
Introduction
2
Preliminaries
Theoretical Re-Examination of DPQ
Reformulation of Population- Based DPO
3.2
The Completeness of Regularizer Matters in Sample- Based Loss
Proximalized Preference Optimization
Introducing Hyper Response for Tractable Loss Approximation
4.2
The Existence of Optimal Solution for PRO
5
Experiments
5.1
Resolving Likelihood Underdetermination
8
5.2
Performance Comparison Under Pairwise and Binary Feedback
5.3
Aligning with Extremely Imbalanced Binary Feedback
10
5.4
Aligning with Scalar Feedback
10
6
Discussion
10
A
Related Work
17
Proof of Theorems
18
B.1
Equivalent Loss for Population- Based DPO
18
B.2
Necessary Condition and Key Properties of the Optimal Solution to eDP( Loss
19
B.3
Relationship between the Optimal Solutions PRO and eDPO Losses
21
B.4
Existence Optimal Solution to the PRO Loss
23
Comparison of the PRO Loss and the RLHF Objective
28
Implementation Details of the PRO Loss
30
The Role of and 5 in PRO's Regularizer
31
Additional Experimental Setup
32
Additional Experimental Results
34
16

---

A Related Work
Reward Hacking in RLHF RLHF employs a learned reward model to align LLM. While the reward model faithfully ranks responses within the training distribution [25] 44,46), it often fails to generalize beyond. Consequently, LLM can exploit this weakness to achieve high rewards without genuinely matching human intent. This effect, known as reward hacking, poses a significant challenge in RLHF 3777.
Recent studies have explored various mitigation strategies, covering the improvements in reward modeling, policy optimization and data augmentation. For instance, the research in [7 introduces an information bottleneck framework to filter out irrelevant noise that may introduce spurious features in reward modeling. Recognizing the limitations of a single reward model, works in 48 49 propose
the use of reward ensembles, which aggregate outputs from multiple models to produce more robust reward estimates; these models can be further combined through weight averaging (50) to improve efficiency. Additionally, authors in [51] advocate conservative approach by optimizing LLMs against the minimum reward predicted from a plausible set of reward models. terms of RL algorithms,, several studies [52 53) argue that the widely used proximal policy optimization [2] is insufficient to prevent reward hacking, and suggest incorporating explicit constraints to enforce more cautious use of reward model. Regarding training data, demonstrations utilized to guide LLM towards generating responses with calibrated rewards [54]; and augmentation tools [55] are applied to diversify the dataset in hopes of improving model generalization. Despite these advances, reward hacking remains a challenging and unresolved issue in RLHF
Direct Alignment with Pairwise Preference Direct aligoment methods I0 bypass the need for explicit reward models and instead optimize LLMs direetly using the preference data. Removing reward model not only lowers computational cost, but also restricts loss evaluation to the offline data. Since on-policy samples are involved, these methods were initially considered immune to reward hacking. However, recent studies 22 30 have shown that length exploitation- a familiar form of reward hacking observed in RLHF
28 -persists in methods like DPO. To incorporates response length as a penalty into the DPO loss; SamPO (30] proposes down-sampling tokens of both preferred and dispreferred responses to equal lengths when computing the implicit reward in DPO; and SimPO [11] introduces normalized rewards to define a novel alignment loss. Although these methods effectively reduce length exploitation, they rely on explicit manipulation or regularization of response length, which unlikely addresses general reward hacking issues. As reported in RLHF literature (56). reward hacking can also manifest as lazy generation [6). degraded downstream task performance [52], and hedging or self-doubt [57]. These observations underscore the necessity for a more comprehensive understanding of reward hacking in direct aligmment methods.
address, several methods have been proposed: R-DPO
An important clue arises from the widely reported phenomenon where the likelihoods of both preferred and dispreferred responses decrease after alignment [2244. In consequence, the generation probabilities for unobserved responses are unintentionally elevated, echoing the heward-hacking effect in RLHF Several research efforts examine this likelihood decline, attributing it to factors such as embedding similarity between paired responses [15. asymmetric update ratios for the probabilities of paired responses [17), and the "squeezing effect" of softmax when applying gradient ascent to dispreferred responses [18]. Notably, these conclusions are chiefly drawn from the analyses of training dynamics in DPO and its variants. In contrast, the presented work focuses on the loss function itself, providing new insights into the underlying cause of this phenomenon and proposing a natural resolution. Additionally, our study also differs from previous attempts to remedy the likelihood decrease, which commonly incorporate additional supervised or regression signals into DPO and may inadvertently compromise the original intent of alignment.
Direct Alignment with Pointwise Feedback Parallel studies have explored alignment methods that utilize feedback beyond the pairwise format. In (58). an upper bound is derived for DPO to accommodate binary feedback, and it is shown that refining this bound can enhance alignment
performance. Departing from the DPO framework, KTO [19] utilizes prospect theory [59] create a utility function tailored for binary feedback. More generally, there is an extensive body of literature concerning binary feedback that focuses on model unlearning (60462), where negative responses are used to eliminate unwanted behaviors from LLMs. Among them, gradient-ascent methods 663 64 are the most straightforward but often lead to catastrophic collapse. Interestingly, alignment-inspiree methods exponentially slow down this collapse (60). Apart from binary feedback, NCA [1] considers
17

---

scalar feedback, and formulates a classification task to capture the varying degree of desirability for each labeled response. While these existing methods tackle the different types of feedback, a unified approach capable of handling pairwise, binary and scalar feedback is still lacking.
B Proof of Theorems
B.1 Equivalent Loss for Population-Basee DPO
Theorem 3.1. The population-based DPO loss is equivalent to the following one, in that they share same gradient:
+ - B vero DKu (ru(s(3)
(s()) B ( B(o(ro(en))re((e)))
= log To(
opiimizer
regularizor
where B denotes Bernoulli distribution,
1
s Ey [ply 1/)
is score function indicating the extent to which y is favored across other responses and satisfies Ey>als(y)
Proof. Theorem 3.1 is derived from an analytical property of the log-sigmoid function, namely,
V (allloo + (1 -1-a)) (-5)
aa)(-)) (1 a))(()()))
a -o) = Vá + log (-5)
(10)
= a - 1--00 = v.[(a 1)8 + log (6)]
(11)
2 2a 120-1oo0 (6) + a(-5)] Vo (a a- 2
1
8 DKL B (
B(o(o))
(12)
where we use V o = (-8) and V log (-5) = -6). the last equation is obtained by averaging (T0) and (1). In other words, the convex combination of log-sigmoid gradients can be decomposed into two distinct components: one dependent on a, and another independent of When encodes the learning signal from training data, the latter term naturally acts data-independent
regularizer.
We next apply this identity to the gradient of
LDPO To; Trer)
= - Em war plu) > y2) log logo re(va))
Worwar ply1 > y2) log a (re(11) re(uz)) p(y2 vi) llgo logo - reevi)))
yielding
V VoLoro(TO:Ted Tre) = Vod. V Coro Tro) Tre))
(a-1) 5 DKL. [B (8(2) B(o(0))
Vod- V WsBya.yawH
( a-
B(o(6)) ]
((
V Ve y. 82r44
8
+ VoE
DKL
B 2
2
2 vr 322mup
(13)
where p(v1 > 42) and re(v1) re(y2).
18

---

The fist term A can be further simplified as follows:
V 0 BoBuys 1/2 (pisn 92) -2) (ro(vi)- ro(v22)
A
p(u V2) -0-1) 2 vV log (o()))
Byr
2 V log (122
U1 M2mp 2
E
plu1 > 122
2)
2 Eyi.war
p(u1 12) 2) VV To)
Eyn .2app 2
p(u2 > v1) Vo (v)))
E
(2p(1) > 122 - -1) V log nO (81))
EEyroy [Bgou(pp(u > v2) 2 1) Vo log log (01))
Substituting this result into 133 produces the final expression:
V ToLPoo Tier)
-yggr (Bya-u [p(er x -y2)]
1)
2 B log To()1)
DOL B (0(2)) B σ Blog Ta(yI)
(2)22)
2 Eyy.
B llgg Tedt (322
2
Tee
VeLarro :ree))
Finally, to show that Ey>nlssy) = 0, note the following relationship:
(omp[((()) = Enjsr/ [p(y > y')] 1
2
1
By [p(y/ > -1)]
2
By.ss'mu [1 plu > y')]
Bysdru [ply x v/)]
= -By-n(s(vi)
The proof follows immediately by rearranging terms.
B.2 Necessary Condition and Key Properties of the Optimal Solution to eDPO Loss
Theorem 3.2. Let μ = p(y) for all y Y. If an optimal solution n² L.DPO exists, it satisfies the condition for any y E y:
(Bl0g T T'())
π (1/)
p(v) ((v)
a BBy'rwy ~p
3 log
8(v).
(2)
Tnn(
19

---

Proof. For notational simplicity, we denote Ty = =T() The gradient of LEDRO can then be expressed
as:
V my Lerro (TT; ner)
-(()) T(V)
+ at En 1122//4
V Mu DKL
B
B
(vn)
T (32) 3 log
2
2
Trer(y1)
(y22
=-((y) f
(VV)
Blogg (y2)
Blog T()))
1
aBBy vamu
,,,,, xy log क (31)
(Te(((2)
Teer(11)
2
(((()) --(())
T(V)
(810gg Blog (11))
[()) Terr(y)
1
v(y)
aB mm
TTer(y") B log
(14)
2
(T)
In the last equation, we relabel y1 and 1/2 as y and y' in the last equation, respectively, highlight that the subsequent discussion focuses on
According the theorem's precondition, optimal solution x² exists. This solution must first satisfy the following feasibility constraints:
T (v) O, Vy,
(15)
(x) =1
Moreover, its optimality requires that the gradients of the loss and the equality constraint satisfy
TT LaDrO (r; Tee))
= = V Σ x(U)) Ex(O)-1)
Vy
(16)
==#"
-=
for some A. This condition is necessary; otherwise, there would exist loss-descent direction that is orthogonal to the gradient of equality constraint. Moving along such direction would reduce the loss while still satisfying the equality constraint. Furthermore, as indicated by (15), n" lies in the interior of the feasible region. Therefore, sufficiently small update step would also preserve the inequality constraints. These analyses indicate that if (T6) is violated, the solution can be further improved within the feasible region, contradicting the optimality of n.
Noting that the gradient the right-hand side of (16) equals one, we obtain
V Ty LeDPO (T; Tet
X
Vy-
(17)
Taking the expectation over y nt on both sides and substituting we arrive at
14 together with Eyy-p [8(y)]
[o
To (3log
TT" TV/) - B log
n (v)
1
oz B. y. ma
= X.
(18)
(Ter(y)))
(2)
2
Exchanging y and 1/,, we have
T²
T (3')
1
A - -asByy B.Byy''
3 Blog Tror(y)
B log Treel Trer(w')
2
(3113gg log
n (1/)
T"" (1r
--a3By.a'
3 log Tree (3)
(19)
Treer (yy)
20

---

where the second equality follows from (z) = -1- ((2)). Combining 18 and 19 yields À = 0.
Then, multiplying both sides of (17) by x x²()) and substituting [14 give
-Ba(y) p(v) aBBy mm o (3 fllog ा (v') (rer(y')
T (v) (er(
)
3 log
2 m(() = 0
After rearranging the terms and substituting the above identities, the desired relation follows, com- pleting the proof:
aBy'ru a (Blog (ua(() (rer(())
[ T'(y)
TT(1(
((v) sty).
3 log
2 1x
Corollary 3.3. Under the preconditions of Theorem 3.2 the following results hold for a constant C:
π (1( C,
Vy (0l) 0 or 8(y) 0,
(3)
Tey(y)
" (y)
> C,
Vy: pdy) > 0 and 8(y) > O.
(4)
Tey(())
T² (y)
< C,
Vy ( > 0 and s(y) < O.
(5)
Tey(y)
Proof. For those values of where either ju(y) 0 or {(v) 0, the right-hand side of equation vanishes. Given that sigmoid function is strictly monotonic, the solution necessarily satisfies
(v) Tief
C,
Vy : ((y) ((v) 0,
(20)
for some constant C.
Similarly, the monotonicity of sigmoid function also indicates
π (y)
> C, Vy : ju(y) > 0 and 8(v) > 0,
(21)
Tel (y)
(y)
< C,
Vy: : ((y)) > and 8(y) < 0.
(22)
(re((y)
B.3 Relationship between the Optimal Solutions to PRO and eDPO Losses
Theorem 4.1. Let H. C y supp(ā) and : p(y) > for all E y. If optinal solutions x and THe to Eepro and EpRO exist, they satisfy:
thZ (v) (y))
Vy E y H,
(7)
Σ - C. Σ TTp(y), NEW
(8)
yEH
NEA
where C. is the constant defined in Corollary
3.3
Proof. Although the optimization variables o and nr defined distinct response spaces, the two objectives LaDRO and ÉPRO can be shown to be equivalent under an appropriate reparameterization. The proof follows directly by substituting condition 3 into relevant terms of both objectives. For
21

---

clarity and completeness, the detailed derivation is presented below.
EAPrO(T; Teet)
- ((() log π (v)]
+ a E Marup
DKL
B
(b))
[1] (
B (((()(0)-
-B ((V) log (v)]
((2))
B
-
B
(r((((())- r (vz)))
2
91 #2
Σ
(( DKL
B
B ((((()))) r(v2))
2
2
31 v H
((9n)
(1)
1
((((r(0)) r(02))
2
2
31 # E
Σ
M(va))(y2)
B
1
B (
2
2
DIEN WEEH
= 3R (v() log
a Σ Σ
((21)
B
-
B (((r())))
2
2
91 H #2
マ (H) Σ
(KV2)
C r(12))
2
2 (H) Σ ((n))
B
1
B
((((r(v))
3 log )
WI
where the last equation follows from noting that H. C y suup((A) and applying condition 33) Concretely, H. C y supp(ū) implies
p(y) 0, Vy E H.
Combining with condition 3 we have
n (v) Tee (v)
C,
Vy H.
In the last equation, probability ratios for responses within H are replaced by C. Therefore, the objective function is expressed using the more compact set of variables {C, (T) y E y H..
22

---

Similarly,
ÉpRO (T; Tet
- (s()) log T [v)]
a +
E
DKI (
B
1 (²²))
B (((-(())-((2)))))
2
un varp
2
- ($(v) [v)]
a
Σ Σ
(()))((22)
B
(
B (o(((()) -
2
2
viEt H F
Σ
((H))
(2)
1
B (((((1)))(())))
2
2
V=EN
Σ
(())))(H)DKLL
(
1
-
2
2
u (H) (H))(H))KK
(B(
B
1
B (r(((H)) ((((H)))((())))
2
- ($(y) [())
Σ
((nn)u(y2)
B
1 2
B (((((())-
2
91
#2
11
(H)
Σ ((22)
B
( C (((va)))
2
2
a (H) 2
Σ FI
B (((((())) - Bog c))
2
where we apply the reparameterization x (+H) Creer (H) in the last equation, considering that π (H) a single variable in this objective.
Under the reparameterized variables {C, y E y H3. the two objectives É-DPO and EeRO coincide exactly. This establishes the equivalence of their optimal solutions, leading to the following relations:
(y) = x (v),
EYEVI H.
मे (H)
n (v)
Tee (H) = =0-
Vy E H,
(e))
where C* denotes the optimal value of C' for the above two objectives. The second equation further implies
Σ mklee) (H)) = (H(
Σ
Trer(y)) =
(H)
Tre((y)
(H)
yEH Teet (H)
yEH
MEH
Σ T (v))
Tee (v)
MEW
DEH
Given that H C y supp((i). by applying equation
again, we have
Σ π
(v) C Σ Tve
NE
DEH
BEH
B.4 Existence Optimal Solution the PRO Loss
Lemma B.I. If no optimal solution exists for the minimization of Erro, there is an infinite sequence within the feasible region that approaches its boundary and strictly decrease the loss value.
23

---

Proof. Let L denote the infimum of ÉeRO over its feasible region, which may be finite or infinite. By definition of the infimum, there exists sequence of feasible solutions { such that
EpRO ( Tret) [PRO(TO: Tear) > ... EPRO(T Frer)) 1
and
lim ( Teet) = n-
Since the sequence } lies the bounded set A, the Bolzano-Weiersraas theorem guarantees existence of convergent subsequence {TT...) J with
lim TR.. = To. n-0
Because { (CrRO(TA., mett)
subsequence of {CpRo (Tii meet) ), it follows that
lim (TRO(TK.." (Tee) L. 1--00
Suppose for the sake of contradiction that Too E A. Then, by the continuity of EpRO on the feasible region, we have
EpRO ( Feee))
implying that the infimum is attained within the feasible region, contradicting the assumption that no optimal solution exists.
Hence, To & A, and the subsequence { } represents the sequence we aim to identify, i.e., approaches the boundary of the feasible region while strictly decreases the loss value.
Theorem 4.2. Given any H C y supp(it and := ply) > 0, Vy E Vn, there is threshold oo such that, whenever ag, optimal solution FHe LrRO exists
Proof. For convenience, we restate the PRO loss as follows:
EprO(IT; meat) = - (5(y) log (v)] + En warr DKL B (8()) B ((((1(0)- r(02)))
Suppose that no optimal solution exists. According to B.I there exists an infinite sequence feasible solutions } which converges the boundary of feasible region and strictly decreases the loss value. Denote the limit of this sequence by }-
Since Too lies the boundary of the feasible region A {T
(
0, Vy
V/ and Σyev, =ly) = 1}. least one response must have zero probability. Define
ya = (v (v) = 0, E )n}
Moreover, being on boundary remains To valid probability mass function, which implies that at least one response must also have positive probability; we denote this set as
y { (t) > 0, E Jn.
Both yo and + are therefore non non-empty. For elements in these two sets, we have
Toc (v") = ro) e Est,
(23)
where (g) 3 log Thus,
F (v) Te (w)
a (roo (v") Foo (v = =0 Vy E yo, Vy+ e y+.
'A constructive choice ao for general preference feedback is given Corollary B.2 'Since Teer is typically instantiated as reference LLM that calculates response probabilities using the softmax function, follows that (e)(y) > 0 for all possible responses

---

Combining this observation with ( || B(0)) = too implies that the regularizer in CPRO diverges +o0 as { approaches rea This suggests that, achieve a reduction in the overall loss value, the optimizer needs to decrease towards negative infinity at a faster rate. Conversely, if it decreases at slower rate, the total loss would increase, which contradicts the assumption that the sequence (T,)) continuously reduces loss.
sufficient condition ensure this contradiction is
B 3p(y"))(y") log (3))
lim sup
<1,
(24)
" DKL (b) (((()())--(0))
for all y satisfying
8(y") 0 and yd E )
(25)
Condition 25
identifies all responses that push the optimizer towards negative infinity, while Condition (24) ensures that their corresponding terms in the optimizer, i.e. -3 (()")) log (yy")
decrease slowly than the increase of the regularizer.
We next derive how to satisfy (4 The main technical difficulty is evaluating the limit superior in (24. the expression involves multiple variables. To address this, we construct an upper bound by mapping the multivariate ratio to univariate function, whose limiting behavior can then be analyzed directly via L' L'Hôpital's rule.
For any 0, when close enough to Toc. Inequation 23 guarantees r(y") < r (y -, Vy Ey",y" E
s(y) where r(y) = B log Tar(y)
This inequation further implies
a (r(v") r(y))) <r(r)))) (1 + +) < 1/2
Since the second sigmoid value is closer to 1/2 compared to the first one, we have
111 2
B B(((r(0)) roo + +c))
DKL
B
B
DKL
B
By substituting the above inequation for all y E y into the left-hand side of 24 and utilizing the non-negativity of KL divergence for $ y+, obtain
Bā ((( (v ) log (b²)
lim sup
DKL B (2) B (((((())) -(wi)))
a 2 (
B
((y")) 8p(v")8(z²)
-log
lipsup -
uu(8 (1(3)) ply) DKL B (
((((r((9)- - Too (v) + +)))
Now, the function on right hand side contains only π ( as the free variable. By tentatively applying L' L'Hôpital's rule, we find that its ordinary limit exists:
30(y
(yy") 3
log (yy"
lim ((())~00 + 22(1²) EpEN+ m(v) DKL. B (1) B (ra((())) (vx) +6))
BP(U ((v") =(*"]
lim
x {1(17)) Eyey+ m(y) (a(r()9) - (1) +c) - x(
40 )8)))
The limit superior equals the ordinary limit upon its existence. Then, Condition 24 can be satisfied
4(10)) (()))
Vg" E 8(y8) < O.
(26)
25

---

Since (()) > 0 for all y E Vn. the right-hand hand side is finite, allowing us to fulfill this inequality by selecting sufficiently large a.
In summary, Inequality (24) is guaranteed to hold for sufficiently large a. implying that the sequence { not continuously decrease the loss when approaching Too- This establishes the desired contradiction.
Corollary B.2. For general preference feedback, valid oxo can constructed as:
(- (u()) miny'yy m(y')
000
max yE>x:S(y)<
Proof. As stated in the proof of Theorem the solution approaches a specified boundary point Too the feasible region, it suffices to select
to prevent the PRO loss from decreasing indefinitlly as
a 40(18) (-8())))
Ay yo :8(8") < O,
(()+)
where
= {v xx) - 0, E VHh and y -² T (1) > 0, y E J/+}
Once a satisfies this condition for boundary points, the loss cannot decrease continuously on whole boundary, thus ensuring the existence of an optimal solution within the feasible region.
This can be achieved by further strengthening the above condition make it Teo -independent:
Instead of enforcing the inequality only for y y 8(y²) O, we can require it for all yd E V/2+ 5(8)) < O.
Since p(y+) Evey' p(y) Z mingey+ p(y) mingey' m(y), we can safely use this
lower bound for further simplification.
Putting these together, a sufficient and easily computable choice is (o(y))(-300))
max (E)
(u(y) minyeey m(B')
Theorem 4.3. Consider the puirwise feedback setting, where = J and H C J.V supp(A). For any a 2 1/n2, an optimal solution THy to EPRO exists. Moreover, when a 1/12, the PRO loss is equivalent to the following one in that they share same gradient:
EPRo-P(TT; knz) -f
Bu plyn u2) log logo ro(ua))]
op
where
[py1 -2) yf. U2 E supp(ā)
ply > v2) an augmented empirical preference.
1/2
otherwise
Proof. We first establish the second part of this theorem. Under the choices of μ = Th and a 1/n2, we have
EpRO SPRO(TT: Teet)
-BE ($(y)
llog (T()
1
1
B (r(((())) rr (rooe)))
+ Eur
DKL
B
2
2
(1
m)
1
271²
E
By Seartupp
DKL
B
2
B -
1
yy ppp
DKI.
B
B (o(o(()) (ro())i))))o))))))) ro(u2))
1)
2
26

---

where the distribution in the second term reduces to 31. 1/2 ~ f.. because the responses in (up are excluded from the hyper response (by the precondition H C y (supp(t)) so that y1, U/ ~ ji and 91, 1/2 & á are equivalent.
By applying the reverse direction of Theorem 3.1 for the first two terms, and expanding the remaining KL terms, we have
( (Ty Tret)
p(y1 x 122 log o (ro(Vi) ro(22))
(27a)
(1 11)2
Eyn Eyn.20
log o (ro(()) - ro(u2)) + + log o (ro(Uz) - ro(U1))
(27b)
2n/2
2
1
"E "Eyrppu pa
log -ro(u)) + log o (ro(Ue) ro(ur)) (
(27c)
17
2
1-17 '1²
log 2
27/²
1 1 17²
Enn warrp F(y1 > y2) log (re(vl)) ro(uz)) ((o(v)))((uu))))
log 2
21/2
1 - 11² log 2,
EPRO-P(TO: neat)
21/2
where
[inn > 32), if y1, y2 E supp(A)
Flu1 > 92)
1/2,
otherwise
the augmented empirical preference.
For the first part of the theorem, it suffices to prove the existence of optimal solution for = 1/n/² This is because such existence requires the regularizer to dominate the loss function the boundary of feasible region, as elaborated in the proof of Theorem 4.2 If a 1/12² already guarantees the dominance, increasing a the strength of the regularizer only further enhances this effect.
Assume that no optimal solution exists when a = 1/n/². By Lemma B. there an infinite sequence of feasible solutions, which converges the boundary of feasible region and strictly decreases the loss value. Let Too denote the limit of this sequence. Since Tho lies on the boundary of A
(ty) 0, Vy V/H and Eyer' mly) 1). if follows that
~x0 O and (1 O, 3yy Vn-
Then, we have
log (roo (v - (1= = -00;
log @ (rx( Txx (1)) = O,
where (v) log xxx (v)
secly)"
We now analyze the behavior of EpRO.P as स approaches Too, by examining the following mutually exclusive cases:
," supp(p): In this case, the term in (27b) with Ø1 1 and y2 y+ diverges to positive infinity.
nd E supp(ā) and yt E supp(p): Here, the term in 27c with V1 1 and 122 = y also diverges to positive infinity.
3 E supp(p) and y* E supp(A): Similarly, the term in 27c with 1/1 = y and y/2 = y diverges to positive infinity.
P.v" E (u))( This case is more complex. Specifically, if p(y > y*) 0. the term in 27a with 8/1 = y and y/2 yt can remain finite as π Rec- However, consider an arbitary response y E supp(p):
"Since Tee is typically instantiated as reference LLM that calculates response probabilities using the softmax function, follows that (e)(y) > 0 for all possible responses
27

---

To (x) > O, the term in (27c) with 91 = =y" and y/2 = y diverges to positive infinity. Otherwise, i.e. xo (y) = 0, the term (27c) with y/1 = y² and y2 = y diverges to positive infinity.
In summary, approaches Too, at least one term in diverges to positive infinity. Considering the non-negativity of - log (+)) for the other terms, it follows that the overall loss LPR-I also diverges positive infinity. This result contradicts the expected monotonous descent of LpRO-P as ] approaches Toc.
C Comparison of the PRO Loss and the RLHF Objective
Both the PRO loss and the RLHF objective incorporate an optimizer and a regularizer, yet they differ in the specific implementations. This section discusses these differences in detail and identifies several research problems that warrant further investigation.
The objective function (to be minimized) in RLHF is
Enyarr [ro[v]] + (Kr.( || meer),
which shares the same gradient with
(gg(( (ro(y) log (v)] (DKI( mer))
(28)
where sg(-) denotes the operation of stop gradient. For convenience of comparison, rewrite EeRO here -gg (5() log To(y)] + Ey varp DEL
B (b2) B (a((ro(u)) ro ro(o)))
(29)
2
The key distinctions between and
29 are highlighted as below:
Hypothesis Reliance RLHF optimizes LLM through the guidance provided by learned reward model. One crucial yet often overlooked element in this framework is selecting appropriate hypothesis for the reward model. The Bradley-Terry model, appreciated for its simplicity and intuitive nature, is commonly used by default. However, when actual user preferences deviate from its underlying assumptions (c.g., user preferences may be non-trane transitivee. which the Bradley-Ter Terry model cannot accommodate (65] [6]), the reward model can produce inexact evaluations that mislead LLM optimization. As comparison, the PRO loss directly leverages the preference signal without relying on the rewards derived from another model. This direct approach hopefully improves robustness against discrepancies between the model hypothesis and true nature of user preferences.
Weighting Factor of V log (y in Loss Gradient Equation (T) illustrates that examining the loss gradient, particularly the weighting factor associated with Va log Ta()), provides valuable insights into the alignment process. We now inspect this factor in the contexts of RLHF and PRO, revealing another noteworthy distinction. First, when computing the optimizer gradients, RLHF weights Ve log To (y) with the unbounded reward value Tg. whereas PRO uses the bounded preference score &. Second, by simplifying the regularizer gradients
/ log
(g())
VaTo(y)dy / TO(y) Vo (og))
9KL || Teet)
Treef (v))
/ log
(T(y)
Vo log ( Tolu)dy Vo To(U)dy
Teat (1)
ro
Vo log [
(30)
Unbounded
"Eorro be viewed instantiation of Erro by letting H. consist of only single individual response.
28

---

and
(B2)
( 2
B -
E 22
DKL
B
a
E warrk
re(22)) Va (ogo -re(())))
2
ax
Ey 10
2 (ololln)--ooiii--o(oooo)
: (o Vo log To(U2)
2
Cx
(ro(v1) (o0o)))))r(()))))))-))))) - 2) (V log - V log (V())
2)
E
2
at
((r((1))- re(u2)) (r(o()) //2 / loologin Tog))(()))
E Burrr
2
(o(ro(ue)) ((To(vv)-ro((i))) ro(uu)) v (u())
-B By Marz
a
2
= Byn
a (ro((e)) - ro(v2))
V log To(y1)
(31)
2
Bounded
we find that the weight boundedness here is consistent to that in the optimizers. Overall, the integrated weight of Vo log To(y) for PRO is bounded, in contrast to the unbounded one in RLHF Given the stochastic nature of loss gradient and the tendency of LLM to forget, the bounded weight would result in a more cautious model update, which potentially improves training stability and model performance when utilizing PRO.
Mode Seeking v.s. Mass Covering The RLHF objective includes reverse KI. divergence as a regularizer, which is characterized by mode-seeking behavior. This means, minimizing the reverse KL tends to concentrate probability mass around the modes of the target distribution, while rela- tively neglecting low-probability regions. Formally, when (g()) + 0 for certain y, the regularizer diminishes even if (y) is non-zero:
Te
log T
lim ne (y) log =9())=>
lim (b) log no (y) me(y))>0
lim
lim (e()))
0,
(y)
(v)
(o(y)
where the penultimate equation follows from L'Hôpitalls rule. This behavior has been reported to reduce diversity in LLM generation [67 [8] By contrast, PRO employs forward KI. divergence as the regularizer, which diverges to infinity as (y()) O:
1
lim =0 2
log
(ro(v)) ro (v)) + log (ro(v')) rr(v))
2
log logo((o(y)) ro(s/)) log (ro(v)) - ro(()) too, Vy : > 0.
= -log((2) - lim (e(
This regularizer prevents response probability from collapsing to zero, exhibiting mass-covering behavior. Besides, it is computed over every response pair constructed from V'. By adjusting the composition of H. we can flexibly modulate the extent of probability mass coverage over the original response space y
The presented work focuses exclusively on direct alignment, specifically optimizing LLMs using offline data, without the inclusion of on-policy or online samples. However, want to emphasize that the proposed PRO method can be directly adopted in on-policy or online scenarios. Considering the above distinctions between PRO and RLHE, several followup research problems arise:
Despite the risk of unreliable evaluations, the reward model in RLHF provides additional alignment signals for unlabeled responses, which may further improve performance if applicd
29

---

appropriately. This motivates extending PRO to incorporate on on-policy samples within such paradigm, particularly when computational resources are sufficient. A key difference in implementation is that we require a score model to predict 8(y), rather than the reward model. Since the score no longer a latent variable inferred from user preferences, does not rely on any model hypothesis. Moreover, by its definition and the property established in Theorem 3.1 the score is bounded in 1-1/2, 1/2] and has an expectation of zero. These information can be leveraged to calibrate the score model or determine when it is reliable. Considering these potential benefits, it is worthwhile to investigate how PRO performs compared to RLHF when applied in the on-policy setting.
Recent progress has shown that online reinforcement learning can substantially enhance the reasoning capabilities for LLMs [69]. Popular approaches, such as PPO 2 and GRPO [70), incorporate k (To (Ttt) as a regularizer. However, is reported 771 that the regularizer is excessively strong, resulting in overly constrained optimization of the LLM. Although fully removing the regularizer mitigates this issue, it may lead to performance degeneration on other unconsidered tasks during post training. This highlights the need for alternative methods to address reglaagguariztioonn As noted, the regularizer in PRO is milder since its gradient involves a bounded weighting factor on V log ( Given the rule-based rewards are also bounded, PRO presents promising objective function for such settings and warrants further empirical investigation.
During the reinforcement learning stage, LLMs explore response space by leveraging the prior knowledge acquired from pretraining or supervised fine-tuning. Maintaining response diversity is critical for enabling a broad spectrum of meaningful exploration, which in turn fosters the development of more advanced reasoning abilities [72]. Since the regularizer PRO exhibits a mass-covering behavior, it is likely more effective preserving diversity during post-training. We consider examining its practical effects as another future research direction.
D Implementation Details of the PRO Loss
Pairwise Feedback In practical pairwise-feedback datasets, each prompt is commonly associated with only one pair of responses. Therefore, the empirical distribution over responses is given by (y().)) = (M()))= = 2. By setting H = ) \ (Mhes 313 and 17 = 3. we can rewrite LprO.P as
EPRO-P(TTr Tief)
E (e
log o (ro(I,
vw)) yw)) re (r((,11))
1 log logo v) x)) + 2 log (ro(u, H) ro(ar, re(s,,)))
+
2
where
1 E Eve (1 To (y)ar)
(a H) = 3 log
1 Eye(mn sn) Teet Ty x))
The value of 17 is chosen so that all response pairs from {yws Mt, H. contribute equally to the overall loss.
Binary Feedback Following the KTO implementation, the binary- data is structured as D ((x()))) vy( s())) )-1 That is, even though multiple responses may exist for a prompt, they are treated separately. Consequently, the empirical distribution reduces to (a()) 1. By setting
H y/ (v) and μ 77 in 99. we can rewrite LRRO
EpRO-B Fref)
- s slog (e()) + log (ro(x, re( H)) + log (re(æ, H) - re (z. uu)) (.
1
where
1 - ne (z/lz)
re( H) = B log
1 Teer( (1)
30

---

The regularizer coefficient is in fact an(1 7)). Since there is no need to have two parameters to determine the regularization strength, we re-denote the coefficient by a for notational simplicity. In the experiments, we tune a so that the performance of PRO-B Anthropie-HH test dataset matches that of PRO-P under 33 0.003. The resulting value = 2.5, which we use throughout unless otherwise noted.
Scalar Feedback Following the NCA implementation, the scalar-feedback data is structured as D = {x((), Vi.N: s[N) where N is the number of labeled responses per prompt. Then, the for all n {1,-.- ,N.. By
empirical distribution over responses for a prompt x is fi(n))
setting 7 = D' {uiNN,, } 1
and N N-I' we can rewrite LPRO as
EpRO.s (To: mree)
-
Sn log (yNlax))
N ==1
2a
Σ
2
1
(roz(()) -r((x + log (ro(o, r0( v))
N(N + +1)
n. .N3 nsnn
N
2a
log o (ro(x, ro(o, H)) + log o (ro(æ, H) - ro(x, 8n))
2
N(N
+1)
where
1 DN To Te (r)
(0 H) = Blg
The value of is chosen that all response pairs from {VI:N. H} contribute equally the overall 2a
loss. The regularizer strength is in fact given by (N+1)=- We rescale it to N(N+1) to ensure that the
optimizer-regularizer weight ratio remains consistent with LpRO. for any a. In accordance with the 2.5 by default.
binary-feedback case,
common characteristic of the three cases above is that H contains a countless number of responses, making (H|zz) and Tee (H|z) extremely close to one. Moreover, small step of parameter update along any direction does not significantly alter the closeness of mh (7([z) to one. Consequently, both the value and the gradient of r0(x H) approximately zero, allowing safely omit this term from the loss function. This behavior was verified in our preliminary experiments, and in all the reported experiments set (30 , H) 0 for simplicity.
E
The Role of a and B in PRO's Regularizer
By rewriting the PRO loss as
(eeo()) -Eywa ($(v) log No [v)] + E ppp
B
where
B (c(36)) and 8 = log
x (er(
π (122
(0) 23 DKL
B
log Teef Tye)
2
the hyperparameters a and B are only involved in the function fa.s. This allows us to analyze their impacts on the loss by simply examining fa.e. Noticing that
- 1
L (800 (
a a (
Vola.
2
we see that a determines the maximum gradient magnitude of the regularizer, while 3 governs how rapidly the gradient grows å departs O. These effects are illustrated in Figure 4 Notably, if is
31

---

too small, the gradient of the overall loss can be dominated by the optimizer, causing unpreferred responses reduced towards zero probability and compromising our theoretical guarantees. This explains why increasing 3 fails to mitigate performance deterioration but increasing helps in
Section
3.0
1-1, 8-1
Gradient of # Dxa ((())))(())))))
0.75
a-1, 8-1
(B()) ((i(0)))
2.5
o-1, 8-3
0.50
o-1, 883
a=3, 8-1
a=3, 0=1
2.0
a=3, 8-3
0.25
a= 0=3
0.00
1.0-
-0.25
sp 0.5
-0.50
0.0
-0.75
Figure 4: a determines the maximum norm of the regularizer gradient, while / controls the rate at which the gradient norm increases from zero to its maximum value.
We now have two hyperparameters a and | in PRO, however, tuning them is considerably simpler than it appears. In particular, it is not necessary to jointly tune a and 3: for a broad range a values above the necessary threshold, one can choose corresponding that enables PRO to consistently attain strong performance.
To elaborate, Theorem 4.2 guarantees the existence of threshold ogg such that, for any Ckp, the regularizer remains effective and prevents probabilities from vanishing. A value of a being sufficiently large indicates that the magnitude of the regularizer gradient should rarely hit its saturation regime during optimization. Let ay be such a sufficiently large value (i.e., 0j > og) and 5 its tuned counterpart. Because a and provides adequate flexibility to shape the curve of regularizer gradient, for any 022 > Or), we can select 882 so that the curve of VsJaa.28 shows a very similar shape with VaJey.8, prior its saturation region. For instance, in Figure 5.3 (right column), the gradient curves for (a,3) = (1, 3) and (3, 1) largely overlap for E (-0.5, 0.5). We also observe that different (a, B) yield similar performance for (2.5, 3e-3), (7.5, Ie-3) and (22.5, 3e-4) are 53.2 53.21%, 52.97% and 53.65% respectively. Under 1e-deiired binary feedback, the win rates for (17.5, 3c-3), (52.5, 1e-3) and (157.5, 3c-4) are 57.21%. 56.51% % and 56.33%, respectively.
our experiments. Under pairwise feedback, the win rates for
F Additional Experimental Setup
For the Pythia-6.9P model, we first utilize the preferred responses from Anthropic-HH dataset for supervised fnn-tuning prior to alignment, following the procedures in [8. Since the Mistral-7B-sft model is already fine-tuned, we apply directly for alignment.
Hyperparameters We adopt a uniform set of hyperparameters across all models, methods and datasets, for both supervised fine-tuning and alignment. Specifically, the maximum prompt length is 1024, and the maximum sequence length is 2048. Models are trained for epoch with RMSprop as the optimizer. The learning rate linearly increases from 0 to 5e-7 over the first 150 training steps and remains constant thereafter. The batch size is 64 for KTO and PRO-B, and 32 for the other methods. This adjustment is due to each binary sample containing a single response, which half that of pairwise samples. To ensure consistent number of gradient steps when comparing methods using pairwise and binary feedback, we double the batch size for binary feedback.
When evaluating KTO with imbalanced binary feedback, we implement reweighting strategy in the loss function to balance the effective proportions of desired and undesired samples at a 1:1 ratio. For instance, if the original ratio of desired to undesired samples is 100:1, we multiply the loss for undesired samples by a factor of 100. This reweighting technique was initially introduced in the KTO paper. We also apply it to PRO.
32

---

Prompt for Evaluating Aligned LLMs on Anthropic-HH Dataset
For the following query to a chatbot, which response is more helpful, harmless and concise?
Query:
<the user query>
Response A:
<either the test method baseline>
Response B: <the other response>
FIRST provide a one-sentence comparison of the two responses and explain which you feel more helpful, harmless and concise. SECOND, on a new line, state only "A" or "B" to indicate which response is more helpful, \ harmless and concise. Your reaponse should the format: Comparison: <one-sentence comparison and explanation> Better response: <"A" or "B">
33

---

G Additional Experimental Results
DPO
NCA
100
0.4
0.5
0.0
-0.5
Reward
-0.2
1.0
-1.5
-2.0
0.6
-2.5
Preferred
-0.8
Prefereed
Dispreferred
Disprefarred
-3.0
-1.0
0.11
11.2
11.4
11.5
0.S
1.0
0.0
0.22
0..4
11.65
11.8
1.0
Percentage Training Steps
Training
PRO.P
PRO.B
0.8
Preferred
0.4
Dispreferred
0.6
Disprefarred
0.4
M mmw
0.2
Reward
Reward
0.0
0.2
w/y J/mmnn Am
-0.4-
-0.4
-0.6
-0.8
Percentage Training Steps
Percentage of Training Steps
KTO
2.0
1.0
-1.0
2.0
-.00
-5.0
Dispreferred
0.2
01.4
11.5
1.0
Percentage of Training Steps
Figure 5: Dynamics of implicit reward when aligning Mistral-7B 7ft with the pairwise/binarized UltraFeedback dataset. DPO, the rewards for preferred examples initially increase but then exhibit continuous decline. In contrast, both NCA and PRO maintain consistently positive rewards throughout the alignment process. Besides, the rewards of NCA, KTO and PRO demonstrate convergent trend as training progresses.
34

---

NCA (N - 2)
PRO-S (N 2)
0.1
0.1
-0.1-
Reward
Reward
10.2
-0.2
-0.4
-0.4
Rank
0.5
Rank
Rank 2
Rank
-0.6
0.0
0.2
0.4
0.6
Percentage Training Steps
Percentage of Training Steps
NCA (N - 4)
PRO-S (N 4)
0.2
0.2
0.2
0.2
Reward
Reward
-0.6
Rank
-0.0
Rank
Rank
-0.8
Rank
Rank
Rank
0.0
0.6
Percentage
Training Steps
Percentage of Training Steps
Figure 6: Dynamics of implicit reward ry when aligning Mistral-7BB with the raw UltraFeedback dataset (scalar feedback), where N denotes the number responses per instruction. Both NCA and PRO maintain consistently positive rewards for preferred examples throughout the alignment process.
Method
ARC
IFEval
TruthfulQA
GPQA
Math
HellaSwag
Average
SFT
51.54
2.40
42.23
29.02
1.06
61.02
31.21
DPO
61.77
19.22
43.45
32.04
0.52
64.12
36.85
KTO
55.38
25.69
41.00
33.04
0.46
62.61
36.36
NCA
58.62
26.43
42.35
32.14
0.60
63.48
37.27
PRO-P
61.26
29.02
43.81
32.59
1.36
63.47
38.59
PRO-B 59.81
30.13
42.72
32.81
1.14
63.27
38.31
Table 4: Performance comparison by aligning the Mistral-7B--s model with UltraFeedback dataset.
All methods demonstrate significant performance improvements after alignment. DPO achieves the leading performance on ARC and HellaSwag, but underperforms on IFEval and Math. PRO performs comparably to or better than the best baseline across all tasks.
Method
N
ARC
IFEval TruthfulQA
GPQA
Math
HellaSwag
Average
2
59.39
27.73
43.45
31.70
0.76
63.70
37.79
NCA
4
59.61
28.96 t
45.78
32.14 f
1.12
62.88
38.41
2
59.47
29.31
45.90
30.80
1.10
62.87
38.24
PRO-S
59.47
29.43
49.45
32.81 f
1.14
62.36 J
39.11 1
Table 5: Performance comparison by aligning the Mistral-7B-sft model with UltraFeedback dataset.
N denotes the number of responses per prompt. Increasing N effectively improves the alignment performance for most tasks.
35

---

Prompt
user| >>
What does a diplomat do?
assistant] >
A diplomat works on behalf of their country to establish and maintain international relationships. They often serve as intermediaries between different countries to help resolve issues. Some diplomats also work on domestic issues within their own country.
user]]
It sounds pretty interesting. assistant]
>
Preferred
It can be. personally think it's a fascinating job, and often find it emotionally rewarding to help people in need.
DPO
It is, there are many challenges as diplomat, but can also be quite rewarding. I find the work interesting and challenging. I also find it very rewarding to help other countries work towards better solutions to international conflicts. think it's really important to have international relationships, and maintaining them is key to advancing the world as a whole. also think that having international relationships is important for any country, because without mutual respect and trust we can't 't make great progress on issues like trade and diplomacy. also enjoy meeting different people from around the world, as well the work itself. find it very interesting and stimulating. think it's also really important to be well-rounded as a person, and having diverse skillset is good way stay engaged and interested different topics. also think it's very important for diplomat to be honest and diplomatic in their interactions with other countries, so that they can build trust and get things done. It can be tough work though, I think.
Judgment
Response B [Preferred] is more concise and focused, avoiding unnecessary elabora- tion while still being helpful and harmless, whereas Response A [DPO] is overly verbose and includes repetitive and subjective details that detract from its clarity and usefulness.
PRO
That it is! A career as a diplomat can offer a variety of interesting opportunities and challenges. Diplomats have to work hard to earn respect from other countries, and they also need to work hard at keeping their own country politically stable. The challenges are both internal and external to the job, and they can be exciting.
Judgment
Response A [PRO] is more helpful and concise as it provides clear explanation the challenges and opportunities
diplomat's career, while Response [Preferred] is more subjective and less informative, focusing on personal feelings rather than factual details.
Table 6: Samples from the aligned Pythia-6.9B models for informational question. The judgment is generated by DeepSeek-V3 through comparison between the preferred response in dataset and the model-generated response, based on the criteria of helpfulness, harmlessness, and conciseness. The responses are presented in a random order when fed the judge, and the post-hoc annotations are formatted as [annotation].
36

---

Prompt
user]] >
How can tell if person is right for marriage? really like this woman but am unsure about proposing.
[asisstant] It sounds like you're really happy to have found someone special! But you are right to hesitate marriage is commitment you should not make lightly, and you want to do right. could definitely suggest resources or questions you can think through, but I think the best way is if we talk more about what you mean when you say "right for marriage", and what questions you want to ask her.
user
Can you provide me with any book titles that speak on this topic? assistant]
Preferred
Do you mean about marriage in general, or about particular questions about possible marriage partner?
DPO
Yes, there lots of great books out there the topic of "right for marriage", here are few good ones:
The Five Love Languages by Gary Chapman
The 5 Love Languages of Marriage by Gary Chapman
The 5 Love Languages of Intimacy by Gary Chapman
The Five Love Languages of Affection by Gary Chapman
Marry Smart by Shaunti Feldheym
The 5 Love Languages of Family by John Townsend
What Men Want by Tristan Taormino
Intimately:
Finding True Love in Relationships by John Gottman
The 7 Principles for Making Marriage Work by John Maxwell
Right For Me. Right From You by John Medvin
The Five Love Languages of Love by Gary Chapman and Jennifer Van Osten
Marriage, History by John Wiley and Sons
Marry Smart by Shaunti Feldheym and The Five Love Languages Affection by
Gary Chapman are both good for more in-depth treatment of the topic hope some of these will be useful!
Judgment
Response A [Preferred] is more concise and helpful as it seeks clarification pro- vide a more tailored recommendation, while Response B [DPO] overly verbose, repetitive, and includes irrelevant redundant book titles, making it less focused and potentially confusing.
PRO
I've got two good books for you. The first one The Five Love Languages by Gary Chapman, which talks about the kind of connection you want in marriage. He talks about how people love to be loved in different ways. The second one is When to Say "I Do" by John and Catherine Eldredge. They talk about the kind of commitment you want in marriage and also some of the things you might want ask her before making a big decision. would definitely recommend those two books.
Judgment
Response B [PRO] is more helpful, harmless, and concise because directly provides specific book recommendations tailored to the user's 's query about marriage, while Response A [Preferred] asks for clarification without offering immediate assistance.
Table 7: Samples from the aligned Pythia-6.9B models for book recommendation.
37

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All claims in the abstract and introduction reflect the paper's contributions and scope, which cover (1) a theoretical reformulation of DPO, (2) novel insights into root cause of likelihood underdetermination, (3) a new alignment method for diverse feedback types that is free from underdetermination, and (4) empirical results demonstrating the utility of our proposed method practice.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: In Appendix C we clarify that this work focuses exclusively on offline scenarios. We also discuss the intriguing connection between our proposed method and RLHF, and identify several research problems in on-policy and online settings that warrant further investigation in future studies.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
38

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: The precondition of each theorem is presented at the start of the respective statement. Appendix includes all the proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Section 5 includes the data preparations for the experiments. Appendices D and G discuss the PRO implementations and the additional experimental setups, respectively.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility.
39

---

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The implementation details are provided in Appendix D We will open-source the code. All datasets used have open access.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
7 //nips. //nips.cc/publlt wwtpc/pppppp..ud....ss.pppppppee htte:/guuiddes/oooSubbiiiiseee wwwo///Swwiiiiionooooooooo
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section 5 and Appendix include all necessary details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report enror bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Due to the resource limitation, error bars are not reported. However, please note that Figure 2 reports the performance throughout the training process, and Figure illustrates results across various hyperparameter settings. We believe these complementary evaluations already significantly distinguish the proposed method from the basclines.
Guidelines:
40

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
We discuss in Section 4.J that the proposed method requires only basic mathe- matical operations beyond DPO, with no additional LLM forward or backward computations. Therefore, it incurs negligible computational overhead in both time and memory compared
Justification:
to DPO.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 'ttntt make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
Our research was conducted in accordance with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
41

---

Answer: [NA]
Justification: Our study is a methodological contribution aimed at advancing preference optimization and does not have direct societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (c.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This work does not involve releasing data or models that have a high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We credit the creators of the assets used in the paper through proper citation.
Guidelines:
The answer NA means that the paper does not existing assets.
42

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaeed, both the original license and the license of the derived asset (if has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets for now. Once the code is open-sourced, we will include well-documented instructions with it.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
43

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does involve LLMs as any important, original, or non standardd components.
Please refer to our LLM policy for what should or should not be described.
44


Reviews:
[{"review_id": "1BbD5WS0Hp", "reviewer": "Reviewer_CDcH", "summary": "This paper proposes proximalized preference Optimization (PRO) by reformulating the standard DPO. The reformulation incorporates a complete regularizer to fix likelihood underdetermination and robustly align large language models across pairwise, binary, and scalar feedback. To make the optimization scalable, PRO uses a hyper-response approximation to efficiently maintain absolute likelihoods. It outperforms existing methods by preventing reward hacking and ensuring stable, high-quality model alignment.", "questions": "1.\tCan the $\\alpha_0$ in Thm 4.2 be determined in practice?\n2.\tHow to compute the $f(H)$ in line 223? This also corresponds to $r(H)$.\n3.\tHow is the hyperresponse $H$ constructed?", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n\n1.\tThe paper studies an equivalent form of DPO and answers the question of why standard DPO has the problem of likelihood underdetermination, which stems from the oversimplified regularizer.\n\n2.\tThe new loss derived can now handle multiple types of feedback, including pairwise, binary, and scalar feedback. Therefore, it is a unified framework.\n\n3.\tRPO is robust to extreme data imbalance.\n\n4.\tThe training of RPO is still contrastive, and therefore inherits the advantage of DPO that no reward model training is needed.\n\n\n**Weaknesses**:\n\n1.\tThe idea of hyperresponse is interesting and saves compute. However, I am concerned that this also forces rare or novel responses to share one probability mass, which can suppress diversity.\n\n2.\tRPO now introduces an additional hyperparameter $\\alpha$, which also needs to be tuned jointly with $\\beta$. Therefore, the tuning might become harder in practice. From the imbalanced experiment, the performance of RPO seems to be affected by $\\alpha$.", "was_revised": true, "final_justification": "The authors have addressed my concerns. I raised my score from 4 to 5.", "rebuttal": "Thanks for your thoughtful comments and appreciating the idea of hyperresponse! \n\n>**W1: The idea of hyperresponse is interesting and saves compute. However, I am concerned that this also forces rare or novel responses to share one probability mass, which can suppress diversity.**\n\n**A1**: Hyperresponse aggregates a group of unlabeled responses, which prevents us from distinguishing their individual probabilities. However, as established in Theorem 4.1, this aggregation does not alter the total probability mass assigned to this group in the optimal solution; only the allocation within the group is not explicitly managed. This means that while rare or novel responses may sometimes receive low probabilities, the effect arises from limited intra-group control, not from increased competition among these responses.\n\nWe would like to further clarify two important points:\n- The inability to control the distribution over unlabeled responses is a challenge shared by all existing alignment methods, not one caused by hyperresponse. Prior approaches such as DPO, KTO, NCA, and IPO optimize the probabilities only for labeled responses, leaving the rest under-specified. Hyperresponse simply makes this explicit and transparent, instead of introducing a new issue.\n\n- Accounting for the individual probabilities of unlabeled responses would require explicitly computing them, which is intrinsically computationally intensive. Moreover, existing methods—even if computational resources allow—typically do not incorporate these probabilities into the loss function. In contrast, PRO moves a forward step by offering a flexible manner to tradeoff the computational cost and the granularity in the probability allocation over the response space. Specifically, if certain unlabeled responses are deemed important (e.g., we want to preserve the model’s capacity for generating them), they can be excluded from the aggregation. In this way, the regularizer would be computed over a finer-grained response space, preserving more detailed likelihood knowledge from the reference model.\n\nWe appreciate this valuable comment and will incorporate the above discussion in the revision.\n\n>**W2: PRO now introduces an additional hyperparameter $\\\\alpha$, which also needs to be tuned jointly with $\\\\beta$. Therefore, the tuning might become harder in practice. From the imbalanced experiment, the performance of PRO seems to be affected by $\\\\alpha$.**\n\n**A2**: We would like to clarify that, in practice, tuning $\\\\alpha$ and $\\\\beta$ is considerably simpler than it might appear. Specifically, it is not necessary to jointly tune $\\\\alpha$ and $\\\\beta$: any value of $\\\\alpha$ above a certain threshold suffices; for each such $\\\\alpha$, there exist corresponding $\\\\beta$ values that allow our method to consistently attain strong performance. We first provide a detailed explanation for this assertion, followed by supporting empirical evidence.\n\nBy rewriting the PRO loss as\n$$\n\\\\frac{\\\\hat{\\\\mathcal{L}}\\_\\\\text{PRO}}{\\\\beta}=-\\\\mathbb{E}\\_{y\\\\sim\\\\hat{\\\\mu}}\\\\big[\\\\hat{s}(y)\\\\cdot\\\\log\\\\pi\\_\\\\theta(y)\\\\big] + \\\\mathbb{E}\\_{y\\_1,y\\_2\\\\dot{\\\\sim}\\\\mu}[f\\_{\\\\alpha,\\\\beta}(\\\\delta)], \\\\text{ where } f\\_{\\\\alpha,\\\\beta}(\\\\delta)=\\\\frac{\\\\alpha}{2\\\\beta}D\\_\\\\text{KL}\\\\big(\\\\mathcal{B}(0.5)||\\\\mathcal{\\\\sigma(\\\\beta\\\\delta)}\\\\big) \\\\text{ and } \\\\delta = \\\\log\\\\frac{\\\\pi(y\\_1)}{\\\\pi\\_\\\\text{ref}(y\\_1)}-\\\\log\\\\frac{\\\\pi(y\\_2)}{\\\\pi\\_\\\\text{ref}(y\\_2)},\n$$\nthe hyperparameters $\\\\alpha$ and $\\\\beta$ are only involved in the function $f\\_{\\\\alpha,\\\\beta}$. This allows us to analyze their impacts on the loss by simply examining $f\\_{\\\\alpha,\\\\beta}$. Noticing that $\\\\nabla\\_\\\\delta f\\_{\\\\alpha,\\\\beta}(\\\\delta)=\\\\frac{\\\\alpha}{2}\\\\big[\\\\frac{1}{2}\\\\sigma(\\\\beta\\\\delta)-\\\\frac{1}{2}\\\\sigma(-\\\\beta\\\\delta)\\\\big]=\\\\frac{\\\\alpha}{2}\\\\big[\\\\sigma(\\\\beta\\\\delta)-\\\\frac{1}{2}\\\\big]$, we see that $\\\\alpha$ determines the **maximum gradient magnitude** while $\\\\beta$ governs **how rapidly the gradient grows** as $\\\\delta$ departs 0. If $\\\\alpha$ is too small, the gradient of the overall loss can be dominated by the optimizer, causing unpreferred responses reduced towards zero probability and thus compromising our theoretical guarantees. Theorem 4.2 established there exists some $\\\\alpha_0$: for any $\\\\alpha>\\\\alpha_0$, the regularizer maintains its effectiveness and prevents probabilities from vanishing.\n\nImportantly, a value of $\\\\alpha$ being sufficiently large implies that the magnitude of the regularizer gradient should rarely hit its saturation regime during optimization. Now, let $\\\\alpha_1$ be a sufficient large value (i.e., $\\\\alpha\\_1>\\\\alpha\\_0$), and $\\\\beta\\_1$ denote the tuned value accordingly. As $\\\\alpha$ and $\\\\beta$ provides adequate flexibility to shape the curve of $\\\\nabla\\_\\\\delta f\\_{\\\\alpha, \\\\beta}$, for any $\\\\alpha_2>\\\\alpha_1$, we can always select a $\\\\beta_2$ such that the curve of $\\\\nabla\\_\\\\delta f\\_{\\\\alpha\\_2, \\\\beta\\_2}$ shows a very similar shape with $\\\\nabla\\_\\\\delta f\\_{\\\\alpha\\_1, \\\\beta\\_1}$ prior to its saturation region. For instance, in Figure 3 (right column) of Appendix F, the gradient curves for $(\\\\alpha=1, \\\\beta=3)$ and $(\\\\alpha=3, \\\\beta=1)$ largely overlap in the region of $\\\\delta\\\\in(-0.5, 0.5)$. In this way, the optimization procedure would progress similarly under these two settings.\n\nEmpirical evidence (please see our response A1 to Reviewer 5FRF for details) also confirms that given sufficiently large $\\\\alpha$ there always exists suitable $\\\\beta$ to make PRO achieve similar performances. In other words, once $\\\\alpha$ exceeds a required threshold, it has marginally effect on the performance.\n\nTherefore, the tuning process involves first determining a sufficiently large $\\\\alpha$, after which $\\\\beta$ can be tuned independently. Theorem 4.3 provides a valid threshold of $\\\\alpha$ in the case of pairwise feedback, and we will elaborate another one for general form of feedback in A3. With that, we only need to tune $\\\\beta$, and the tuning overhead becomes similar to that of DPO.\n\n>**Q1: Can the $\\\\alpha\\_0$ in Thm 4.2 be determined in practice?**\n\n**A3**: Yes—$\\\\alpha\\_0$ can *constructively* determined in practice.\n\nAs shown in the proof of Theorem 4.2 (Line 773), to prevent PRO loss from decreasing indefinitely as the solution approaches **a specified boundary point** $\\\\pi_\\\\infty$ of the feasible region, it suffices to select\n$$\n\\\\alpha>\\\\frac{4\\\\hat{\\\\mu}(y^0)\\\\cdot(-\\\\hat{s}(y^0))}{\\\\mu(y^0)\\\\mu(\\\\mathcal{Y}^+)}, \\\\quad \\\\forall y^0\\\\in\\\\mathcal{Y}^0:\\\\hat{s}(y^0)<0,\n$$\nwhere\n$$\n\\\\mathcal{Y}^0 = \\\\big\\\\{y\\\\mid\\\\pi\\_\\\\infty(y)=0, y \\\\in\\\\mathcal{Y}\\_\\\\mathcal{H}\\\\big\\\\}\\\\quad\\\\text{and}\\\\quad\\\\mathcal{Y}^+ =\\\\big\\\\{y\\\\mid \\\\pi\\_\\\\infty(y)>0, y\\\\in\\\\mathcal{Y}\\_\\\\mathcal{H}\\\\big\\\\}.\n$$\nOnce $\\\\alpha$ satisfies this condition for **all boundary points**, the loss cannot decrease continuously on the whole boundary, thus ensuring the existence of an optimal solution within the feasible region.\n\nThis can be achieved by further strengthening the above condition to make it $\\\\pi\\_\\\\infty$-independent:\n- Instead of enforcing the inequality only for $y^0\\\\in\\\\mathcal{Y}^0: \\\\hat{s}(y^0)<0$, we can require it for all $y^0\\\\in\\\\mathcal{Y}\\_\\\\mathcal{H}:\\\\hat{s}(y^0)<0$. \n- Since $\\\\mu(\\\\mathcal{Y}^+)=\\\\sum\\_{y\\\\in\\\\mathcal{Y}^+}\\\\mu(y)\\\\geq \\\\min\\_{y\\\\in\\\\mathcal{Y}^+}\\\\mu(y) >\\\\min\\_{y\\\\in\\\\mathcal{Y}\\_\\\\mathcal{H}}\\\\mu(y)$, we can safely use this lower bound for further simplification.\n\nPutting these together, a sufficient and easily computable choice is\n$$\n\\\\alpha\\_0=\\\\max\\_{y\\\\in\\\\mathcal{Y}\\_\\\\mathcal{H}:\\\\hat{s}(y)<0}\\\\bigg[\\\\frac{4\\\\hat{\\\\mu}(y)\\\\cdot(-\\\\hat{s}(y))}{\\\\mu(y)\\\\cdot\\\\min\\_{y'\\\\in\\\\mathcal{Y}\\_\\\\mathcal{H}}\\\\mu(y')}\\\\bigg].\n$$\n\n*PS*: As discussed in A2, the value of $\\\\alpha$ does not perceptibly affect the performance, once it exceeds the required threshold. Therefore, there is no need to pursue the minimal $\\\\alpha\\_0$.\n\n>**Q2: How to compute the $f(\\\\mathcal{H})$ in Line 223? This also corresponds to $r(\\\\mathcal{H})$.**\n\n**A4**: The general function $f$ is used to illustrate how expectation can be efficiently computed under the hyperresponse approximation (as described in Line 223). In the case of PRO loss, $f$ is instantiated as a function involving the terms $\\\\mu(\\\\mathcal{H})$ and $r(\\\\mathcal{H})=\\\\beta\\\\log\\\\frac{\\\\pi(\\\\mathcal{H})}{\\\\pi\\_\\\\text{ref}(\\\\mathcal{H})}$ with respect to $\\\\mathcal{H}$. By the definition (6), the relevant quantities are calculated as\n$$\n\\\\mu(\\\\mathcal{H})=1-\\\\sum\\_{y\\\\notin\\\\mathcal{H}}\\\\mu(y),\\\\quad\\\\pi(\\\\mathcal{H})=1-\\\\sum\\_{y\\\\notin\\\\mathcal{H}}\\\\pi(y),\\\\quad\\\\pi\\_\\\\text{ref}(\\\\mathcal{H})=1-\\\\sum\\_{y\\\\notin\\\\mathcal{H}}\\\\pi\\_\\\\text{ref}(y).\n$$\nSince $\\\\mathcal{H}$ is typically constructed (in Lines 215-219) to include all unlabeled responses, the complement of $\\\\mathcal{H}$ corresponds exactly to the labeled responses. As their probabilities are necessarily computed for alignment, computing the above terms incurs negligible additional cost.\n\n>**Q3: How is the hyperresponse $\\\\mathcal{H}$ constructed?**\n\n**A5**: Theorems 4.1-4.3 require that $\\\\mathcal{H}$ consists exclusively of unlabeled responses. By default, we construct it by including all such unlabeled responses (in Lines 215-219) , and this construction is consistently used in our reported experiments.\n\nIn initial experiments on the HH dataset, we also explored the effect of excluding 2 or 4 unlabeled and $\\\\pi\\_\\\\text{ref}$-generated responses from $\\\\mathcal{H}$. The results show negligible changes in win rate (less than 0.5%).\n\nHowever, in scenarios where certain unlabeled responses generated from the reference model are of particular interest, excluding them from $\\\\mathcal{H}$ can allow for more targeted regularization, and help maintain the model’s ability to generate these responses."}, {"review_id": "lFG1S7Uf8Q", "reviewer": "Reviewer_5FRF", "summary": "This paper presents Proximalized Preference Optimization (PRO), a new method for aligning large language models (LLMs) with human feedback that addresses key limitations of the widely used Direct Preference Optimization (DPO) method. The authors first identify a critical issue in DPO they term \"likelihood underdetermination,\" where the model's likelihood for both preferred and dispreferred responses decreases during training. This can lead to reward-hacking effects, causing the model to generate outputs that deviate from expected patterns. The core contribution is a theoretical reformulation of the DPO loss function, decomposing it into two distinct parts: an optimizer term that reorganizes feedback into a pointwise signal, naturally extending its applicability beyond pairwise comparisons to other formats like binary and scalar feedback; and a regularizer term that is independent of preference labels. Through this new perspective, the paper argues that likelihood underdetermination in standard DPO arises from an oversimplification of this regularizer. The proposed PRO method addresses this by incorporating a more complete, yet computationally efficient, approximation of the regularizer. This is achieved through a novel \"hyper response\" mechanism, which groups unobserved responses to make the calculation tractable. PRO offers a unified framework for aligning LLMs with diverse feedback types (pairwise, binary, and scalar) while simultaneously resolving the underdetermination issue.Extensive experiments demonstrate that PRO successfully mitigates reward hacking and achieves performance that is comparable to or better than specialized methods like DPO, KTO, and NCA across various feedback scenarios, including challenging cases with extremely imbalanced data.", "questions": "1. I think this paper does not define $\\hat{\\mu}$ clearly. I think $\\hat{\\mu}$ is the responses with feedbacks, and $\\mu$ is all responses. Is that correct? I think the authors should state this more clearly around Line 164. Otherwise, what is \"full regularizer\"? \n2. Why Corollary 3.3 solves the underdetermination issue? I think the underdetermination is the important weight diminishing. But the difference of probability ratios of chosen and rejected pairs could still be large under Corollary 3.3. \n3. I'm not sure I understand \"unobserved\"correctly. I think it means responses without feedbacks. So Sec 4.1 is basically you treat unobserved responses evenly in the regularizer's expectation. I think you could make this easier to understand for readers. \n4. I think you use hyper response because sampling  ${y_1,y_2\\sim{\\mu}}$ is intractable. But is this Hyper Response method only be able to use in PRO? I think in the  population-based formulation of DPO (Line 144), you can also use Hyper Response, is that correct? \n\nI give the score \"Borderline reject\". But I am willing to raise my score if my questions are resolved.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n- This paper first proves that the population-based DPO's gradient is equivalent to the eDPO's gradient. The reformulation decomposes the loss into an optimizer and a regularizer, and only the optimizer relies on the preference feedbacks. This decomposition therefore provides greater flexibility in developing sample-based loss since the preference feedbacks are usually limited. \n- A major contribution is the unification of alignment across pairwise, binary, and scalar feedback within a single framework. While methods like KTO and NCA were developed for specific feedback types , PRO's design, derived from its reformulated optimizer, naturally accommodates this diversity.\n\n\n### Weaknesses\n- Some sentences are not easy to understand. See questions. \n- PRO introduces a new hyperparameter, $\\alpha$, which balances the optimizer and regularizer. While the experiments on imbalanced data show this to be a powerful and necessary lever, it also adds to the complexity of hyperparameter tuning compared to standard DPO. From Table 2, the impact of $\\alpha$ is significant. How to select the best $\\alpha$? The paper could be strengthened by including a more systematic sensitivity analysis for $\\alpha$ across different settings to provide practitioners with better guidance.\n- The \"hyper response\" is an elegant approximation, but it is still an approximation. It prevents the model from differentiating the probabilities of individual responses within the hyper set. The paper argues this is not problematic because the total probability mass is constrained. However, it is possible that this could mask certain failure modes where the model learns to assign disproportionate probability to a specific \"bad\" but unobserved response.\n- In Table 2, other baseline methods are missing.", "was_revised": true, "final_justification": "The rebuttal clarify my concern on the hyperparameter selection, underdetermination, importance weight diminishing, and hyper responses. I think this is a very good paper. Writing needs some improvement to make it clearer to follow.", "rebuttal": "Thanks for the constructive feedback and helpful questions!\n\n>**W1: How to select the best $\\\\alpha$? The paper could be strengthened by including a more systematic sensitivity analysis for $\\\\alpha$ across different settings to provide practitioners with better guidance.**\n\n**A1**: We conducted additional experiments to analyze the sensitivity of PRO to $\\\\alpha$, with the results summarized as follows.  \n- Win rate (%) under pairwise feedback:\n|$\\\\alpha~ \\\\backslash~\\\\beta\\\\quad$|$1e^{-2}\\\\quad$|$3e^{-3}\\\\quad$|$1e^{-3}\\\\quad$|$3e^{-4}\\\\quad$|$1e^{-4}$|\n|---|---|---|---|---|---|\n|$2.5$|$51.47\\\\quad$|$\\\\mathbf{53.21}\\\\quad$|$49.71\\\\quad$|\n|$7.5$||$51.87$|$\\\\mathbf{52.97}$|$49.03$|\n|$22.5$|||$51.21$|$\\\\mathbf{53.65}$|$50.13$|\n- Win rate (%) under 1%-desired binary feedback:\n|$\\\\alpha~ \\\\backslash~\\\\beta\\\\quad$|$1e^{-2}\\\\quad$|$3e^{-3}\\\\quad$|$1e^{-3}\\\\quad$|$3e^{-4}\\\\quad$|$1e^{-4}$|\n|---|---|---|---|---|---|\n|$17.5$|$48.91$|$\\mathbf{57.21}$|$52.13$|\n|$52.5$||$49.83$|$\\mathbf{56.51}$|$51.49$|\n|$157.5$|||$48.31$|$\\mathbf{56.33}$|$53.09$|\n\nFor reference, in our paper, $\\\\alpha=2.5$ for pairwise feedback and $\\alpha=17.5$ for 1%-desired binary feedback are sufficiently large to yield strong performance. \n\nIn the new experiments, we further increased $\\\\alpha$ by factors of 3 and 9 to examine its impact. The results show that, once $\\\\alpha$ surpasses the required threshold, there always exists a suitable $\\\\beta$ such that the competitive performance is maintained. In other words, the performance is insensitive to further increase in $\\\\alpha$ beyond this point. This raises the opportunity to simplify the hyperparameter tuning. Due to the space constraints, we refer the reviewer to our response A2 to Reviewer CDcH, where we provide an explanation for the above phenomenon, and discuss a practical hyperparameter tuning strategy that brings its overhead on par with DPO.\n\n>**W2: The \"hyper response\" is an elegant approximation, but it is still an approximation. It prevents the model from differentiating the probabilities of individual responses within the hyper set. The paper argues this is not problematic because the total probability mass is constrained. However, it is possible that this could mask certain failure modes where the model learns to assign disproportionate probability to a specific \"bad\" but unobserved response.**\n\n**A2**: We would like to clarify that enumerating every unlabeled response in alignment is fundamentally computationally intensive, and this limitation is not unique to PRO. In fact, existing approaches such as DPO, KTO, NCA, and IPO only include labeled responses in their losses, leaving all other unobserved responses unconstrained. As a result, due to the absence of training signals, the probabilities of unobserved responses may drift arbitrarily during optimization, which potentially leads to specific \"bad\" and unobserved response being assigned with disproportionately high probability. PRO makes this limitation transparent, rather than introducing a new one.\n\nMoreover, even if additional computation for unlabeled responses were feasible, existing methods do not specify how to include them in the loss. By contrast, PRO allows for straightforward exclusion of these responses from the hyper-response set, and the regularizer can directly optimize their probability mass. This provides a flexible way to balance computational cost and regularization granularity across the response space.\n\n>**W3: In Table 2, other baseline methods are missing.**\n\n**A3**: Among the baselines, only KTO supports binary feedback. Its results, based on using the optimal $\\\\beta$ identified in Figure 2, are reported in Lines 345-347 and 358-359 (since the primary focus of Table 2 is to investigate how $\\\\alpha$ affects PRO’s performance, we did not include them there). \n\nIn the 1%-desired setting, KTO performs poorly and appears to suffer from reward hacking: the aligned model generates numerous duplicated and meaningless tokens (Line 347). We thus increased $\\\\beta$ in an attempt to bring the optimized model closer to the reference model, but the results remained unsatisfactory (Line 350). For this reason, we did not report the win rates in the submission. We provide the corresponding results below and will include them in the revision:\n|$\\\\beta$|$0.03\\\\qquad$|$0.1\\\\qquad$|$0.3\\\\qquad$|$1\\\\qquad~~$|$3$|\n|---|---|---|---|---|---|\n|Win rate (\\%)|$22.46$|$27.29$|$18.13$|$15.21$|$16.89$|\n\nIf the reviewer has more suggestions for adapting other baselines to binary feedback, we are happy to include them in the revision.\n\n>**Q1: I think this paper does not define $\\\\hat{\\\\mu}$ clearly. I think $\\\\hat{\\\\mu}$ is the responses with feedbacks, and $\\\\mu$ is all responses. Is that correct? I think the authors should state this more clearly around Line 164. Otherwise, what is \"full regularizer\"?**\n\n**A4**: We agree that this distinction is critical and will explicitly clarify it around Line 164. Specifically:\n- $\\\\hat{\\\\mu}$ refers to the empirical distribution over responses that are labeled with feedbacks (i.e., empirical distribution of the preference dataset, as described in Line 167). \n- $\\\\mu$ refers to the distribution over all possible responses (typically, it can be the underlying distribution from which responses are sampled for preference annotation, as stated in Line 89).\n\n>**Q2: Why Corollary 3.3 solves the underdetermination issue? I think the underdetermination is the important weight diminishing. But the difference of probability ratios of chosen and rejected pairs could still be large under Corollary 3.3.**\n \n**A5**:  Let us clarify the terminology and the role of Corollary 3.3 detailedly：\n- **Underdetermination** means that adding a constant to both log-probabilities of the preferred and dispreferred responses ($a\\rightarrow a+c, b\\rightarrow b+c$) does not alter the value of DPO loss: \n$$\n\\\\log\\\\sigma\\\\big((a+c)-(b+c)-\\\\log\\\\pi_{ref}(y_w)+\\\\log\\\\pi_{ref}(y_l)\\\\big) = \\\\log\\\\sigma\\\\big(a-b-\\\\log\\\\pi_{ref}(y_w)+\\\\log\\\\pi_{ref}(y_l)\\\\big),\n$$\nAs a result, when $c$ is negative, the probability mass can \"leak\" from the labeled responses to the unobserved responses, causing the \"reward hacking\" phenomenon observed in prior works.\n- **Importance weight diminishing** provides a more specific explanation of how, in practice, both preferred and dispreferred log-probabilities decrease in DPO. Specifically, once the log-probability gap is large enough, there is no incentive to update the model, even if the log-probabilities of preferred responses have dropped (which highly likely happens due to catastrophic forgetting). \n\nIn other words, underdetermination creates the possibility for reward hacking, while importance weight diminishing makes it actually occur during DPO training.\n\nCorollary 3.3 addresses the underdetermination by enforcing an order among the log-probability changes after optimization. Specifically, it ensures that the log-probability change for any unobserved response must lie between those for preferred and dispreferred responses. Unlike before, the simultaneous probability decreases for preferred and dispreferred responses would also require a reduction in the probability of unlabeled responses, which is impossible due to the constraint of fixed total probability. \n\nLastly, we note that the log-probability difference between paired responses becoming unboundedly large is a separate issue of DPO, often referred to as degeneracy. We appreciate that the comment (and Reviewer iesQ's Q3) for bringing our attention to the fact that PRO also addresses this issue. For further details, please refer to our response A4 to Reviewer iesQ.\n\n>**Q3: I'm not sure I understand \"unobserved\" correctly. I think it means responses without feedbacks. So Sec 4.1 is basically you treat unobserved responses evenly in the regularizer's expectation. I think you could make this easier to understand for readers.**\n\n**A6**: \"Unobserved\" indeed refers to the responses without feedbacks, i.e., those not represented in the preference dataset. \n\nTo clarify, treating unobserved responses evenly in the regularizer's expectation would still require enumerating every such response, which is computationally infeasible. In Section 4.1, we aggregate all unobserved responses into a single hyper response, treating it as a unified group. This allows us to reduce the expectation to just $N+1$ terms: $N$ for the labeled responses and one for the hyper response. Computation of the regularizer then requires evaluating $\\\\mu$, $\\\\pi$, and $\\\\pi\\_\\\\text{ref}$ on the hyper response. To enable it, we define $p(\\\\mathcal{H})=1-\\\\sum\\_{y\\\\notin\\\\mathcal{H}}p(y)$ for any distribution $p$. This avoids the need for explicit enumeration and makes the computation tractable.\n\nWe will revise accordingly to make these points easier to understand.\n\n>**Q4: I think you use hyper response because sampling $y\\_1,y\\_2\\\\in\\\\mu$ is intractable. But is this Hyper Response method only able to use in PRO? I think in the population-based formulation of DPO (Line 144), you can also use Hyper Response, is that correct?**\n\n**A7**: We appreciate this insightful question. Hyper response is indeed designed to address the intractability of sampling $y\\_1,y\\_2\\\\in\\\\mu$. According to Theorem 4.1, hyper response can only be constructed from unlabeled/unobserved responses. In contrast, the population-based DPO loss assumes access to preference feedback for every pair, which means that hyper response cannot be directly applied to it.\n\nHowever, as stated in Theorem 4.3, under the specific choices of $\\\\mu$ and $\\\\alpha$, PRO recovers a similar form with DPO, except that it is based on an augmented empirical preference and **involves the hyper-response mechanism**. Actually, it can be treated as an enhanced variant of sample-based DPO, where pseudo preference $p(y\\_1\\succ y\\_2)=1/2$ is introduced for $y\\_1,y\\_2\\\\in\\\\mathcal{Y}\\_\\\\mathcal{H}\\\\setminus\\\\text{supp}(\\\\hat{\\\\mu})$."}, {"review_id": "E1UfAUPL7y", "reviewer": "Reviewer_o7Vr", "summary": "This work identifies the issue of likelihood underdetermination in direct alignment methods for LLMs, which causes decreased absolute likelihoods of example responses and undesired outputs. It revisits and reformulates the loss in direct preference optimization (DPO), revealing the oversimplification of a regularizer as the underlying cause. Based on this, the authors introduce PRO, which resolves the underdetermination issue through an efficient approximation of the complete regularizer and demonstrates superior performance in handling various feedback types.", "questions": "See the section above", "limitations": "See the section above", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengthes:\n1. The paper provides a novel decomposed reformulation of the Direct Preference Optimization (DPO) loss function. This reformulation broadens the applicability of DPO to a wider range of feedback types, and offers new insights into the underlying cause of likelihood underdetermination.\n2. The authors identify that the standard DPO implementation oversimplifies a regularizer, leading to likelihood underdetermination. They demonstrate that reinstating the complete regularizer effectively resolves this issue.\n\nWeaknesses and Questions:\n1. Appendices seem to be missing.\n2.  It is surprising to see the bad performance of DPO. It would be helpful The paper does not provide detailed information on how hyperparameters were chosen. More transparency in the hyperparameter tuning process would enhance the reproducibility and credibility of the results.", "was_revised": true, "final_justification": "This work addresses the issue of likelihood underdetermination within direct alignment methods for Large Language Models (LLMs). This problem leads to a decrease in the absolute likelihood of example responses and results in undesired outputs. By revisiting and reformulating the loss function of Direct Preference Optimization (DPO), the authors identify an oversimplified regularizer as the root cause. Building on this insight, they introduce Preference Ranking Optimization (PRO), which resolves the underdetermination issue by efficiently approximating the complete regularizer, demonstrating superior performance across various feedback types. All my concerns are addressed.", "rebuttal": "Thank you for your review and feedback!\n\nWe would like to clarify that all appendices are included in the **supplementary material**, in line with NeurIPS guidelines, which state that extensive appendices can be provided either in the supplementary material or the main submission. For your convenience, the appendices cover the following sections:\n- A &nbsp; Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15\n- B &nbsp; Proof of Theorems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16\n- C &nbsp; Comparison of PRO Loss and RLHF Objective . . . . . . . . . . . . 25\n- D &nbsp; Implementation Details of PRO Loss . . . . . . . . . . . . . . . . . . . . 28\n- E &nbsp; Performance Degeneration of KTO . . . . . . . . . . . . . . . . . . . . . 29\n- F &nbsp; The Role of $\\\\alpha$ and $\\\\beta$ in PRO Regularizer . . . . . . . . . . . . . . . . . . 29\n- G &nbsp; Additional Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . 29\n- H &nbsp; Additional Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . 31\n\nWe understand how important complete appendices are for a thorough evaluation, and apologize for any confusion regarding their location. We would be grateful if you could further assess our submission in light of the availability of appendices.\n\nBelow, we respond to the remaining concerns raised in the review.\n\n>**Q1: It is surprising to see the bad performance of DPO.**\n\n**A1**: We agree that DPO's underperformance on the Anthropic-HH dataset is noteworthy. We believe this is due to the following differences between our setup and the original DPO paper:\n- **Evaluation Criteria**: Unlike the DPO paper, which assesses helpfulness alone, our evaluation incorporates helpfulness, harmlessness, and conciseness. As noted in [1], the Anthropic-HH dataset is annotated for both helpfulness and harmlessness (with HH being short for them). Moreover, LLM judges are known to favor overly verbose responses; several benchmarks (e.g., AlpacaEval 2) and previous works (e.g., KTO) have explicitly included conciseness as an evaluation dimension. These metrics therefore more closely match the intent of the dataset as well as real-world alignment objectives. \n- **Sequence Length**: Our experiments utilize more modern context windows (prompt: 1024, sequence: 2048) compared to the original DPO setting (256/512 respectively). As reward hacking often manifests as generating overly verbose outputs (please see the right panel of Figure 1), increasing the context window could further exacerbate this phenomenon. Given that modern LLMs typically operate with even longer contexts, DPO’s vulnerability to such issue is likely even greater.\n\nFurthermore, the underwhelming results of DPO we observe are consistent with earlier findings in the KTO paper [2], where Figure 2 shows that SFT alone outperforms SFT+DPO across multiple Pythia model sizes.\n\nWe will revise the manuscript accordingly to reflect these points.\n\n[1] Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022.  \n[2] Model alignment as prospect theoretic optimization, 2024.\n\n>**Q2: The paper does not provide detailed information on how hyperparameters were chosen. More transparency in the hyperparameter tuning process would enhance the reproducibility and credibility of the results.**\n\n**A2**: We appreciate the reviewer’s concern regarding transparency in hyperparameter tuning. To clarify:\n- **General hyperparameters**: These are specified in Appendix G (Line 954) and align with the official KTO repository (commit ID: 306ed27). We did not perform additional tuning beyond these canonical settings.\n- **Method-specific hyperparameters ($\\\\alpha$, $\\\\beta$)**: \n  - **$\\\\beta$ selection**: We conduct a hyperparameter sweep over $\\\\beta$, as described in Lines 324-326. Figure 2 summarizes the results across $\\\\beta\\\\in\\\\{0.1, 0.03, 0.01, 0.003, 0.001\\\\}$ on the Anthropic-HH dataset. For the UltraFeedback dataset, Table 1 reports the best performance across these $\\\\beta$ values for each method, as described in Lines 332-335. For imbalanced binary feedback (from the Anthropic-HH dataset), we adopt the optimal $\\\\beta$ identified in Figure 2, as mentioned in Line 345-346 and 358-359. For scalar feedback, we similarly perform a sweep over the same range of $\\\\beta$; we will add this clarification in the revision.\n  - **$\\\\alpha$ selection**: Details of $\\\\alpha$ settings are provided in Appendix D (Lines 907-909, 917-918), and a reference is also provided in Line 301 of the main paper. Additionally, based on the comment from Reviewer 5FRF, we have added further experiments during the rebuttal phase to examine the performance across a range of $\\\\alpha$ (see our response A1 to Reviewer 5FRF).\n\nWe hope these clarifications sufficiently address the reviewer's concerns. If there are any additional points or specific aspects the reviewer would like us to elaborate on, we would be glad to provide additional clarification."}, {"review_id": "TjPD4nibSa", "reviewer": "Reviewer_iesQ", "summary": "This paper addresses the limitation that Direct Preference Optimization (DPO) oversimplifies the regularization term. To highlight this issue, the authors present a series of theoretical results. In response, they propose Proximalized Preference Optimization (PRO), a variant of DPO designed to resolve likelihood underdetermination and provide a unified framework for aligning language models across diverse types of preference feedback. Theoretical analysis and empirical experiments are conducted to demonstrate the effectiveness of PRO.", "questions": "1. The corollary establishes a bound involving a constant $C$, but it would be helpful if the authors could further clarify the valid range or typical values of $C$, and how it depends on properties of the policy or data distribution.\n2. In Theorem 4.2, $\\alpha_0$ is given as a lower bound on the regularization coefficient $\\alpha$. However, if $\\alpha$ is required to be too large (i.e., $\\alpha_0$ is large), then the regularization term may dominate the PRO loss, potentially hindering learning from the comparison data. Could the authors provide more discussion on how $\\alpha_0$ is determined\n3. If I understand correctly, the existence of an interior solution in Theorems 4.2 and 4.3 seems to address the degeneracy problem of the policy learned via the DPO loss, as highlighted in the [IPO](https://arxiv.org/abs/2310.12036) paper. Could the authors clarify whether this connection is intentional, and if so, provide experimental results or discussion to support this claim?\n4. Theorem 3.1 shows the equivalence between the $\\mathcal L_{eDPO}$ and the standard DPO formulation. While the formal result is clear, it would be helpful if the authors could provide additional intuition behind this equivalence.", "limitations": "The discussion of limitations is included in Appendix C.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper clearly identifies a key limitation of sample-based DPO methods, namely, the oversimplification of the regularizer term, which leads to likelihood underdetermination.\n2. To address this issue, the authors propose a principled variant of DPO, called Proximalized Preference Optimization (PRO). They support their method with theoretical analysis and empirical results demonstrating its effectiveness.\n\nWeaknesses:\n1. Some of the theoretical proofs lack rigor. For example, the proof of Theorem 3.2 could be more formally structured by applying the KKT conditions.\n2. Several theorems are presented without sufficient explanation or intuition, making them difficult to interpret. (see Questions section for specifics)", "was_revised": true, "final_justification": "The authors have addressed all my concerns through new experimental results and a clearer intuition for the proposed formulation. I maintain my score of acceptance.", "rebuttal": "Thanks for your insightful comments and helpful questions!\n\n>**W1: Some of the theoretical proofs lack rigor. For example, the proof of Theorem 3.2 could be more formally structured by applying the KKT conditions.**\n\n**A1**: We would like to clarify why the KKT conditions are not directly applied, and how our derivation still captures their underlying principles.\n\nThe classical KKT conditions are designed for optimization problems with equality and **non-strict inequality** (i.e., $\\\\geq$ or $\\\\leq$) constraints. However, the problem in Theorem 3.2 is defined over the **open simplex** $\\\\Delta=\\\\{\\\\pi\\\\mid\\\\pi(y)>0,\\\\forall y\\\\in\\\\mathcal{Y}\\\\text{ and }\\\\sum\\_{y\\\\in\\\\mathcal{Y}}=1\\\\}$, since the logarithm in the loss requires all $\\\\pi(y)$ strictly positive. This makes the KKT conditions not directly applicable.\n\nBriefly, the KKT conditions consist of:\n- Primal feasibility: The optimum satisfies all original constraints.\n- Stationary & complementary slackness: At the optimum, the gradient of the objective is a linear combination of the gradients of **active** constraints (those satisfied with equality), signifying that any descent direction will violate some constraint.\n- Dual feasibility: Lagrange multipliers for inequality constraints are non-negative.\n\nIn our case, provided that the optimal solution exists (as assumed in Theorem 3.2):\n- The optimum necessarily lies in the interior of $\\\\Delta$ (i.e., the two conditions following Line 677). \n- This in turn means that sufficiently small moves in any direction remain the strict inequality constraints satisfied, and thus these constraints are, in effect, **inactive**. Then, we only need to consider the equality constraint; the stationarity condition reduces to that the gradient of the objective is colinear to the gradient of this equality constraint—precisely as stated in equation (16).\n- The variable $\\\\lambda$ in (16) serves as the Lagrange multiplier associated with this equality constraint and, in line with the KKT framework, is unconstrained.\n\nTherefore, each element above corresponds exactly, in a one-to-one fashion, to a component in the KKT framework. Based on this derivation, we further establish Theorem 3.2 by analyzing $\\\\lambda$, showing that it must be zero. \n\nWe appreciate the reviewer’s attention to the rigor of our proofs and are glad to provide further clarification if needed.\n\n>**Q1: The corollary establishes a bound involving a constant $C$, but it would be helpful if the authors could further clarify the valid range or typical values of $C$, and how it depends on properties of the policy or data distribution.**\n\n**A2**: The constant $C$ depends on the values of $\\\\pi^\\*(y)$ for $y$ where $\\\\hat{\\\\mu}(y)=0$ or $\\\\hat{s}(y)=0$. However, as established in Theorem 3.2, $\\\\pi^\\*$ is determined by an equation system involving the non-linear sigmoid and logarithm functions, which makes a closed-form analysis of $C$ intractable. \n\nTo gain more intuition on what influences $C$, recall that eDPO includes a regularizer to encourage $\\\\pi$ to remain close with $\\\\pi\\_\\\\text{ref}$, alongside an optimizer that adjusts response probabilities upward or downward depending on the preference signal. According to (2), the preference signal is amplified by a factor of $\\\\hat{\\\\mu}(y)/\\\\mu(y)$ at optimum. If the dataset contains much more preferred responses with large amplification factors than dispreferred responses, the increases in probabilities for these preferred responses is likely to outweigh the decreases for dispreferred ones. Since probabilities must sum to one, this results in lower probabilities for unobserved responses—thus, $C$ tends to be smaller. In general, the typical scale of $C$ is inversely related to the total amplified factors of both preferred and dispreferred responses present in the data.\n\nMoreover, It is important to emphasize that **the main purpose of Corollary 3.3 is to show that eDPO enforces an ordering on the probability changes among preferred, unobserved, and dispreferred responses**. This means that the probability of unobserved response cannot be increased or decreased arbitrarily, as it is bounded by the changes to preferred and dispreferred responses. In other words, the corollary ensures that the underdetermination issue is resolved, regardless of the specific value of $C$.\n\n>**Q2: If the required $\\\\alpha$ is too large (i.e., $\\\\alpha_0$ is large), then the regularization term may dominate the PRO loss, potentially hindering learning from the comparison data. Could the authors provide more discussion on how $\\\\alpha_0$ is determined?**\n\n**A3:** We would like to clarify that $\\\\alpha$ and $\\\\beta$ jointly affect the strength of regularization relative to the learning signal. To elaborate, let us rewrite the PRO loss as\n$$\n\\\\frac{\\\\hat{\\\\mathcal{L}}\\_\\\\text{PRO}}{\\\\beta}=-\\\\mathbb{E}\\_{y\\\\sim\\\\hat{\\\\mu}}\\\\big[\\\\hat{s}(y)\\\\cdot\\\\log\\\\pi\\_\\\\theta(y)\\\\big] + \\\\mathbb{E}\\_{y\\_1,y\\_2\\\\dot{\\\\sim}\\\\mu}[f\\_{\\\\alpha,\\\\beta}(\\\\delta)], \\\\text{ where } f\\_{\\\\alpha,\\\\beta}(\\\\delta)=\\\\frac{\\\\alpha}{2\\\\beta}D\\_\\\\text{KL}\\\\big(\\\\mathcal{B}(0.5)||\\\\mathcal{\\\\sigma(\\\\beta\\\\delta)}\\\\big) \\\\text{ and } \\\\delta = \\\\log\\\\frac{\\\\pi(y\\_1)}{\\\\pi\\_\\\\text{ref}(y\\_1)}-\\\\log\\\\frac{\\\\pi(y\\_2)}{\\\\pi\\_\\\\text{ref}(y\\_2)}.\n$$\nHere, $\\\\alpha$ and $\\\\beta$ only appear in $f\\_{\\\\alpha,\\\\beta}$. To better understand their roles, note that\n$$\\\\nabla\\_\\\\delta f\\_{\\\\alpha,\\\\beta}(\\\\delta)=\\\\frac{\\\\alpha}{2}\\\\bigg[\\\\frac{1}{2}\\\\sigma(\\\\beta\\\\delta)-\\\\frac{1}{2}\\\\sigma(-\\\\beta\\\\delta)\\\\bigg]=\\\\frac{\\\\alpha}{2}\\\\bigg[\\\\sigma(\\\\beta\\\\delta)-\\\\frac{1}{2}\\\\bigg].\n$$\nThis reveals that $\\\\alpha$ determines the **maximum gradient magnitude** of the regularizer, while $\\\\beta$ controls **how rapidly the gradient grows** as $\\\\delta$ moves away from zero. Therefore, even if a large $\\\\alpha$ is required, the overall regularization effect can be modulated by adjusting $\\\\beta$.\n\nWe have empirically validated this in our response A1 to Reviewer 5FRF, where additional experiments are conducted by increasing $\\\\alpha$ beyond the one already producing strong performance. The results show that, even for $3\\\\times$ or $9\\\\times$ values of $\\\\alpha$, reducing $\\\\beta$ appropriately maintains competitive performance.\n\nThese results also indicate that, once $\\\\alpha$ exceeds the required threshold, its further increase has minimal effect, as we can always find a suitable $\\\\beta$ to retain strong performance. Thus, in practice, it suffices to choose any $\\\\alpha$ above the certain threshold, without the need for precise tuning. Considering this, we present a constructive method to determine a sufficient $\\\\alpha\\_0$ in our response A3 to Reviewer CDcH. With this analytical $\\\\alpha\\_0$, hyperparameter tuning primarily focused on $\\\\beta$—similar to the standard DPO. For details, we kindly refer the reviewer to that response due to space constraints.\n\n>**Q3: The existence of an interior solution in Theorems 4.2 and 4.3 seems to address the degeneracy problem of the policy learned via the DPO loss, as highlighted in the IPO paper. Could the authors clarify whether this connection is intentional, and if so, provide experimental results or discussion to support this claim?**\n\n**A4**: Thanks for the insightful observation! While our initial derivation was not specifically intended to address the degeneracy issue highlighted in the IPO paper, Theorems 4.2 and 4.3 do guarantee that our proposed method resolves this problem.\n\nTo empirically demonstrate, we conducted the same experiment as presented in the IPO paper: a bandit environment with three actions and the dataset $\\\\mathcal{D}=\\\\{y\\_a\\\\succ y\\_b, y\\_b\\\\succ y\\_c, y\\_a\\\\succ y\\_c\\\\}$. The policy to be optimized is parameterized as $\\\\pi\\_\\\\theta(y\\_i)=\\\\text{softmax}(\\\\theta)\\_i$, with the reference policy being uniform. We trained the policy for 18,000 steps using Adam with learning rate $0.01$ and batch size 9. All methods converged before training completed; the summarized results are as follows:\n- PRO (we only present the result of $\\\\alpha=5$ due to the space constraints):\n|$\\\\beta\\\\qquad\\\\qquad$|$1.0\\\\qquad\\\\qquad$|$0.3\\\\qquad\\\\quad$|$0.1\\\\qquad\\\\quad$|\n|---|---|---|---|\n|$\\\\pi(y\\_a)$|$0.4736$|$0.7701$|$0.9849$|\n|$\\\\pi(y\\_b)$|$0.3173$|$0.1827$|$0.0149$|\n|$\\\\pi(y\\_c)$|$0.2091$|$0.0472$|$0.0002$|\n- IPO:\n|$\\\\beta\\\\qquad\\\\qquad$|$1.0\\\\qquad\\\\qquad$|$0.3\\\\qquad\\\\quad$|$0.1\\\\qquad\\\\quad$|\n|---|---|---|---|\n|$\\\\pi(y\\_a)$|$0.4469$|$0.6950$|$0.9647$|\n|$\\\\pi(y\\_b)$|$0.3224$|$0.2295$|$0.0341$|\n|$\\\\pi(y\\_c)$|$0.2307$|$0.0755$|$0.0012$|\n- DPO:\n|$\\\\beta\\\\qquad\\\\qquad$|$1.0\\\\qquad\\\\qquad$|$0.3\\\\qquad\\\\quad$|$0.1\\\\qquad\\\\quad$|\n|---|---|---|---|\n|$\\\\pi(y\\_a)$|$1.0000$|$1.0000$|$1.0000$|\n|$\\\\pi(y\\_b)$|$0.0000$|$0.0000$|$0.0000$|\n|$\\\\pi(y\\_c)$|$0.0000$|$0.0000$|$0.0000$|\n\nAs shown above, DPO degenerates to assigning all probability mass to response $a$, regardless of the $\\\\beta$ value. In contrast, both IPO and our PRO method maintain meaningful, non-trivial distributions over the actions.\n\nWe will incorporate the above discussion into the revision.\n\n>**Q4: It would be helpful if the authors could provide additional intuition behind the equivalence in Theorem 3.1.**\n\n**A5**: Theorem 3.1 essentially utilizes an interesting property of the log-sigmoid function. Concretely,\n\n\\begin{align}\n\\\\nabla\\_\\\\delta\\\\big[a\\\\log\\\\sigma(\\\\delta)+(1-a)\\\\log\\\\sigma(-\\\\delta)\\\\big]&=a\\\\sigma(-\\\\delta)-(1-a)\\\\sigma(\\\\delta)\\\\\\\\\n&=a-\\\\sigma(\\\\delta)=\\\\nabla\\_\\\\delta\\\\big[a\\\\delta+\\\\log\\\\sigma(-\\\\delta)\\\\big]\\\\\\\\\n&=a-1+\\\\sigma(-\\\\delta)=\\\\nabla\\_\\\\delta\\\\big[(a-1)\\\\delta+\\\\log\\\\sigma(\\\\delta)\\\\big].\n\\end{align}\n\nIn other words, the convex combination of log-sigmoid gradients can be decoupled into two parts: one that depends on the data-relevant signal $a$; another that is independent of $a$, and thus can be re-arranged to serve as a regularizer.\n\nWe will incorporate the above explanation in the revision to better convey the intuition underlying our proposed method."}]

Author rebuttal:

Meta-review: This paper proposes Proximalized Preference Optimization (PRO), a variant of DPO designed to resolve likelihood underdetermination and provide a unified framework for aligning language models across diverse types of preference feedback. Reviewers appreciated the clear decomposed formulation, theoretical grounding, and promising empirical results. Given the overall positive sentiment, I recommend acceptance.
Decision: Accept (poster)