Displacement-Resistant Extensions of DPO with Nonconvex $f$-Divergences
Authors: Idan Pipano
Shoham Sabach
Kavosh Asadi
Mohammad Ghavamzadeh
Venue: ICLR 2026 Poster (2026)
Keywords: Alignment
Direct Preference Optimization
Reinforcement Learning
f-divergences
Primary area: reinforcement learning
TL;DR: We characterize a broad class of functions f enabling DPO with f-divergence, and prove a simple condition on f that ensures displacement resistance.

Abstract: DPO and related algorithms align language models by directly optimizing the RLHF objective: find a policy that maximizes the Bradley-Terry reward while staying close to a reference policy through a KL divergence penalty. Previous work showed that this approach could be further generalized: the original problem remains tractable even if the KL divergence is replaced by a family of $f$-divergence with a convex generating function $f$. Our first contribution is to show that convexity of $f$ is not essential. Instead, we identify a more general condition, referred to as DPO-inducing, that precisely characterizes when the RLHF problem remains tractable. Our next contribution is to establish a second condition on $f$ that is necessary to prevent probability displacement, a known empirical phenomenon in which the probabilities of the winner and the loser responses approach zero. We refer to any $f$ that satisfies this condition as displacement-resistant. We finally focus on a specific DPO-inducing and displacement-resistant $f$, leading to our novel SquaredPO loss. Compared to DPO, this new loss offers stronger theoretical guarantees while performing competitively in practice.

Published as a conference paper at ICLR 2026
DISPLACEMENT-RESITTAYY EXTENSIONS OF DPO WITH NONCONVEX f D-IIVERGENCES
Idan Pipano
Shoham Sabach
Faculty of Data and Decision Sciences Technion Israel Institute of Technology idan. pi pano@campus technicn.cc... ac.
Faculty of Data and Decision Sciences Technion Israel Institute of Technology
ssabach&comme edu
Kavosh Asadi
Mohammad Ghavamzadeh
Meta
Qualcomm AI Research
kavoshemeta.com
ghavamza@qtia i qua comm. com
ABSTRACT
DPO and related algorithms align language models by directly optimizing the RLHF objective: find policy that maximizes the Bradley-Terry reward while staying close to a reference policy through a KL divergence penalty. Previous work showed that this approach could be further generalized: the original prob- lem remains tractable even if the KL divergence is replaced by family of f- divergence with a convex generating function f. Our first contribution is to show that convexity of f is not essential. Instead, we identify a more general condition, referred as DPO-inducing, that precisely characterizes when the RLHF prob- lem remains tractable. Our next contribution is to establish second condition on f that is necessary to prevent probability displacement, known empirical phe- nomenon in which the probabilities of the winner and the loser responses approach zero. We refer to any f that satisfies this condition as displacement-resisane finally focus specific DPO-inducing and displacement-resisaee our novel SQUAREDPO loss. Compared to DPO, this new loss offers stronger theoretical guarantees while performing competitively in practice.
We f. leading
INTRODUCTION
Language models (LMs) have emerged as a promising path towards achieving AGI. As they be- come increasingly integrated into real-world applications, is important to ensure that LMs behave in accordance with human preferences. A major step towards enabling LM alignment was the re- inforcement learning from human feedhack (RLHF) framework (Christiano al. 2017). In this context, the alignment problem is formulated as maximizing a learned Bradley-Terry (BT) reward signal while ensuring that the final LM does deviate too much from reference policy. This is a guardrail achieved by penalizing deviations through a KL-divergence penalty. A breakthrough was due Rafailov et al. (2024) who showed that the RLHF problem could be solved directly in single step by leveraging the fact that (I) the problem has a closed-form solution, and (II) plug- ging the closed-form solution into the BT model removes intractable quantities. leading simple optimization.
More recent work showed that the so-called DPO approach of Rafailov al. (024) can be gener- alized. In particular, Wang al. (2024) showed that it is possible to maintain tractable optimization while moving from the KL divergence to a family of f-divergences. This use of f-divergences,, not to be confused with minimizing the f-divergence between the learned model and desired model (Go et al.) 2023 Han et al.] 2024). is the focus of our work. More precisely, Wang et al. (2024) showed that if f convex, differentiable, and f is invertible with 0 & dom(f"). then the new optimization problem remains directly solvable, akin to solving the original RLHF problem using DPO.

---

Published as a conference paper at ICLR 2026
Our first contribution is to show that the RLHF problem remains tractable for even broader class of functions than those discovered by WWan et al. (022)) In particular, we show that surprisingly one can relax the convexity assumption and that even for some non-convex functions f. one can get f-divergences that maintain tractability. In fact, we discover the full characterization of functions for which the original RLHF problem remains tractable. Referred to DPO-induuing, the new condition is less restrictive than the class of functions discussed by prior work.
The richness of the class of DPO-inducing functions gives us significant flexibility, but also raises a natural question: which specific choice of DPO-inducing f is best equipped to improve align- ment? To answer this question, we note that an empirical pitfall of DPO and related algorithms is probability displacement, phenomenon in which both winner and loser probabilities approach zero during optimization (Fisch et al., 2025). Our second contribution is show that within the class of DPO-inducing functions, there exists subset of functions that provably hedge against probability displacement. We refer to such functions as displacement- resissstt
We finally focus divergence that both DPO-inducing and displacement-ressstaant We show that optimizing this novel loss results in comparative empirical performance while enjoying robustness to over- optimization and empirically mitigating displacement. Together, our results suggest that DPO- inducing f-divergences can be used effectively for preference optimization, and that displacement- resistant f-divergences, in particular, are best suited for aligning LMs.
novel loss, called SQUAREDPO, which arises by using nonconvex f-
PRELIMINARIES
A key step in training Language Models (LMs) is to align them with human preferences. In this task, we leverage preference dataset 'D, often consisting of triplets (a,, Vws 31). Here, x is a prompt, and yw Iit are two responses, where human, or strong LLM, has annotated yu to be better response for the prompt æ than yı The response yo is referred to as the chosen or winner response, and y as the rejected or loser response.
The original technique to solve this task was to perform a two-stage optimization [Christiano et al.) 2017). Namely, first train parameterized reward model re by minimizing the logative-log- likelihood loss:
min E (a. [- log a (ra (a, yw) To (x, yc))
(1)
under the assumption that our preferences follow the Berry (BT) (Bradley & Tenry 1952) model, i.e., that there exists a reward function r such that the probability of yu being preferred over yi given a prompt æ (r (zz yw) r (z, 3)),. where (t) = 1/ (1 + +) is the sigmoid function. Here and throughout, log(-) denotes the natural logarithm (base e). The learned reward model To is then used by the PPO algorithm to optimize the following objective:
max Ez [Eywaa(()e) (ro (r. v)] - SKL [To (- (x)) || Tiet (: [+]=)]]
Here, higher values of 3 > 0 provide more incentive for the final policy to remain close to Trets where Tief, called the reference policy, is the checkpoint from which optimization is initialized (often the SFT checkpoit).
Noting that Problem algorithm, called DPO, as an alternative for the two-stage RLHF optimization described above. Note that the closed-form solution of (2) is given by me (y x) Nee (y x) exp (ro (1, y) /B) /Z (ar), where the so-called partition function Z (x)) does not depend on y. This closed-form solution can be rewritten as:
2 admits closed-form solution, Rafailov et al. (203) derived one-stage
no (y x) पe (y x)
T (x, v) = 33 log
log Z (z)
(3)
Substituting this into ①1) leads to the single-stage loss:
2

---

Published as a conference paper at ICLR 2026
To (Ye x3) xx)
20 (3/t Tree (yr
x+)
min CDO (0) = Bla.y.. we)DD
logo
(Blog
3 log
(4)
Tree (yw
x))
which can be minimized directly.
While simple and effective, DPO cxhibits counterintuitive empirical phenomenon. This so-called probability displacement phenomenon refers to the tendency of DPO to take the probability both the preferred and the dispreferred responses
2024 [Shen al. (Pang al. 2024 Xiao al.
2024 Nvidia et al.] 2024 Wu et al.] 2025 Pal et al. [024: Rafailov et al.] 2024 Fisch et al. [202
D'Oosterlinck et al. 2025 Asadi al. 2025 Razin al. 2025p. While the research on this topic
is active and ongoing, the theoretical understanding remains partial. Our goal is to advance this
understanding by identifying the precise requirements that ensure both tractability and stability of
DPO optimization.
3 DPO WITH NON-CONVEX f --DIVERGENCES
Note that the RLHF objective 2 employs the KL divergence to measure the discrepancy between the learned policy and the reference policy. However, this discrepancy could be measured using more general notion of distance, such as with the family of f-divergences (Wang et al. 2024 Huang
al. 2025) defined next (Csiszár 1972).
Definition 1. Let f : RR+ + T. be function such that f (1) = 0. For two probability distributions p.q E An the I-divergence between p and q is defined
( as (pt/4s).
KL-divergence is an instantiation of f-divergence with fKL (t) logt. This naturally allows choosing various f-s to obtain various alternatives instead of the KL divergence in the RLHF prob- lem. Therefore, we now focus on the following more general RLHF problem:
max EZZD [Byoso((-)z) [ro (x. v)] B (mo (.1x)) 75
|| Tee (: [)]]]]]]]]]]]]]
(5)
However, recall that our desire is to keep the original RLHF problem tractable and easy to optimize, and so the choice of f needs to be made with this consideration in mind. Interestingly, Wang et al. (2024) show that for any convex with invertible f' that satisfies 0 dom (f"), substituting the optimal solution of the generalized RLHF problem into the original BT model [T] yields the following DPO-like loss, referred to as fDDOO:
min CJ-DPO (0) ((n. log (88'
(yhw |x) Teet (yow x))
3f
To (31 xz)
Tief (31 x))
(6)
In the next section, we show that convexity is not necessary, meaning that there exist non-convex for which the problem remains tractable.
3.1 WIICH f-s Can BE USED?
We now provide a characterization that fully specifies the class of functions f for which the RLHF optimization problem is tractable. To this end, we introduce the following definition:
Definition 2. We say a functton f ; R4 + DPO-inducing if substituting any optimal solution of 5 into the BT model (I) yields the f-DPO loss (6).
Our aim is then to characterize which functions f are DPO-inducing and which are not. The com- plete characterization is provided in Theorem | in the Appendix. As the Theorem iis rather tech- nically involved, in Corollary we add slight assumption (existence of the limit Tim (t)) under which the characterization becomes simpler and straightforward to check.
We note that traditionally f f-divergences defined for convex functions f- However, as shall see in in the context of direct preference optimization with f-divergence, convexity not necessary.
3

---

Published as a conference paper at ICLR 2026
Corollary 1. [Proof in Appendix 2
Let f: R - R be continuous function in R+, which is continuously differentiable in ++ Assume that lim - f (t) exists (possibly =oo). Then, f is DPO-inducing if and only if 1iim f (t) -oc.
See Figure I where functions inside and outside the DPO- -PO-inducing region are examples of functions that do and do not satisfy the condition in Corollary [ respectively. An interesting point about the proof of Theorem [ is that it uses the equivalence of the DPO-inducing property to a property we call interior-inducing (see Definition in the Appendix). In short, f is interior- interior-indueing if optimal solutions to 5 assign non-zero probability to all responses.
The importance of this result is two-fold. First, reveals a broader family of functions f that can be used in f-DPO, not just convex functions as discussed by Wang et al. (2024). Second, the result shows that failure to satisfy lim,-so- I" (1) = -c means that the optimization problem will become intractable.
3.2 WHICH f-s Should BE USED?
Having characterized the class of DPO-inducing functions, i.e., the functions / that can be used in S-DPO, now show that some functions in this class are theoretically less favorable. To this end, given a preference dataset DD and prompt from the dataset, we denote by Sg the set of responses that appeared the preference dataset a response for x (either as a winner or loser). Consider the following optimization problem
max E EIDD TT
Bygwao(-1z) (ro (x, - 3 Σ Tel (y |
T
x
Tel (y
x
yES,
which differs from the modified RLHF objective (5) only by the summation in the f-divergence term
being only over S₂ instead of over the entire response space. Note that Problem (7) might admit
optimal solutions that are very different than those of Problem (5) (see Appendix D.2).
We begin with Lemma D which is interesting in its own right and, as we will see in Lemma also plays a key role in understanding the likelihood displacement phenomenon.
Lemma 1 (Proof in Appendix E.3 Let f E. - IR be a DPO-inducing function. Then, substi- tuting any optimal solution to (7) into the BT model (T) yields the same f-DPO loss (6) as obtained from substituting any optimal solution to (5) into the BT model.
Lemma D reveals a counterintuitive and concerning property of f-DPO: while it solves the two RLHF losses (5) and ( it is also true that it solves the two losses 7 and [ One way to see why this result is concerning is that Problem 7 is conceptually problematic, as its regularization term acts only over in- in-sample responses y Sg, while intuitively we want regularize To to remain close to Tier over the entire response space. To gain additional insight, in Appendix [D22] we find the set of optimal solutions this problem in the special case where f is convex. Another aspect, which holds regardless of the convexity of f. is that based on the value of arg min f (t), the optimal solutions to ( could have undesirable properties, as Sonma[ssoww
Lemma 2 (Proof in Appendix E.4p. Let f be DPO-inducing function with a unique global min- imum at e E (0, 1]. Suppose r (x, v) S 455x T (x., 1o) for all y E Sr. Then, any optimal solution to satisfies
T (y x) S c- Ted (v xx)
(8)
for any prompt x and in-sample response y E Sr-
Lemma shows that under mild assumption if we use any for which arg min f(t) 1, optimal solutions to (7) suffer from likelihood displacement. As special case, consider fKr. (4) = i log I, which is the generating function for the KI. divergence and hence corresponds to the original
loss of DPO. Since arg min fKL (t) = - we get a probability decrease in the in in-sample
We denote R |0, oo) and R++ + (0 no).
'To see why the assumption in Lemma 2 is mild, notice that it is equivalent to requiring the existence of a response y é Sz with higher reward than y. As S₂ typically consists of only two responses, its complement covers almost the entire response space, making the condition very likely to hold.
4

---

Published as a conference paper at ICLR 2026
responses by factor of least e=! Although similar result shown in previous work (Asadi et al. 2025), it was only for DPO, whereas here we prove this for general f. The importance of generalizing this statement to general f is that from Lemma 2 arises theoretically grounded way to mitigate the likelihood displacement issue choosing an for the f-divergence that satisfies arg min f(t) 2 1. We refer to any f that satisfies this property displacement ressiaaa function.
To summarize this section, our theory imposes two desiderata for the function of the f f-divergence that nicely translate to easy-to-check mathematical properties: it should be DPO-inducing, i.c., lim;-0+ (4) = -x, and should he displacement-resisttant. i.e., arg min J(t) 2 1. In the next section, we propose one specific that satisfies our two desiderata, discuss the resulting loss, and report empirical evaluations.
f R + R
Displacement-Resisttat
((1) (log t)²
divverggee f(t) = (t (t 1)2
Ours: f(t) 2 (log t)²
Jeffrey: f(t) log t log t
Reverse KL:
f(t) log
Jensen-Shannon:
f(t) = -(t (t + 1) log
++
+ tlogt
2
f(t) = ---a_(1-att
a-divergence:
ala-1)
KL: f(t) f log
Convex
DPO-Inducing
Figure 1: A Venn diagram illustrating a taxonomy of some generating functions f. The diagram includes classical examples of f-s used in /-divergences, well the functions f(( (log 1)² and f(t) (log t)2, which correspond
Monte Carlo approximation of KL proposed by (Schulman (2020). DPO-inducing refers to Definition 2 and displaceeenttresisttaa refers to the condition I
arg min,eR o (t) proposed in $3.2 to mitigate likelihood displacement. The gray
area the intersection these two sets of functions.
A note convexity Notice that as long as f satisfies our two properties, there seems to be no reason to require f to be convex. In fact, recall that a main reason to require f to be convex in f-divergences is that together with Jensen's inequality and the assumption f(1) 0, it guarantees that the divergence is non-negative. However, our result Lemma undermines this logic, as in Problem (7) the summation in the 'divergence' term is As result, Jensen's 's inequality does not apply, hence convexity of f no longer guarantees the non- negativity of this term.
strict subset of the response space.
5

---

Published as a conference paper at ICLR 2026
4 SQUAREDPO
Our proposed method is the instantiation of the f-DPO loss (6 obtained by choosing [SQUARERPO (1) = (log /22 We refer to the resulting algorithm as SQUAREDPO. The function [SQUARERPO is one possibility for function that satísfies the two properties suggested in $ as demonstrated in Figure This particular function is both DPO-inducing and displacement-resisiee thus making it natural choice for preference optimization. Note that we deliberately chose a non- convex f. in order to explore the broader (compared to prior work) class of functions that Corollary [ enables. We note that non-convex losses have proven effective in the past for preference alignment, e.g., LLM--poposse losses presented in Lu et al. (2024). However, we are the first to explore f-DPO (6) with non-convex functions f.
The explicit formula for the SQUAREDPO loss, obtained by substituting (6. given by
SaaiinnggSSuuoa SSQUAREDPO (1) (log l)/L
SquaredPO
E ((a.wwen) log (Bo (Yw, x)) log
no (o Nedt (ye
3o (y), xx) log
To To [x] Tarlsi (x)
where Bu (v, = 3/ (olg) fa and 3 > 0 is as usual the regularization hyperparameter. Com- pared to the original DPO loss 4 the SQUAREDPO loss can be viewed as "DPO with adaptive 8-s". There a connection between our approach and some of the recent approaches (Meng et al.] 2024 Lee al., 2025 in that both adaptive f-s,, but here the adaptive B-s are derived from theory and first principles rather than introduced heuristically. The adaptive 3-s in SQUAREDPO can
2024 Wu et al.
a safeguard against severe likelihood displacement; when mo (ywr x)) decreases, as is often observed in practice, 30 (Vw, z) increases. An increase in the effective reg- ularization coefficient intuitively means stronger regularization, thereby counteracting the decrease in probability.
5 EXPERIMENTS
In this section, validate theoretical findings regarding the ability of SQUAREDPO to mitigate likelihood displacement, demonstrate that it is robust to over-optimization compared to DPO, and that it performs competitively on standard benchmarks.
5.1 EXPERIMENTAL SETUP
Model and Dataset Our reference model Tief. i.e., the model from which we initialize preference optimization, is Meta-L Instruct (AI@ Metal 2024). The preference dataset we use to align Tet is TL.DR (Völke et al.) (017). We use version of this dataset provided by von Werra ctt al. (2020) that is preprocessed for preference optimization. The dataset consists of Reddit posts from various topics, each paired with a preferred and a dispreferred summary.
Evaluation We compare SQUAREDPO with DPO on three axes. First, we want to empirically probe our theoretical findings that SQUAREDPO is more resistant likelihood displacement than DPO. This is done by computing the chosen log-ratios, log (To (yw x) /Teed (yw rr), for all train- ing samples (xr Vur Vt) E DD in different checkpoints throughout training. Second, we measure the performance of each method on the dataset's validation split, reporting win rates as judged by GPT- Lastly, assess SQUAREDPO! ability generalize to out-of-distribution tasks, we compare its performance against DPO on standard evaluation benchmarks.
The evaluation benchmarks we considered are AlpacaEval 2 (Li et al. 2023 Dubois et al. 2025) and MT-Bench (Zheng et al. 2023). AlpacaEval 2 contains 805 single-turn prompts, where GPT-4 judges") compute both win rate (fraction of cases model beats a strong baseline) and length- controlled win rate that corrects for verbosity bias. MT-Bench consists of 80 multi-turn questions across diverse categories, scored on a 1-10 scale by GPT-4.
We use OpenAl's gpt-4o instead of the default judge gpt-4- 106-preview for reduced API costs. Dubois 2025) show that the induced rankings are very robust the choice of the judge.

---

Published as a conference paper at ICLR 2026
5.2 EXPERIMENTAL RESULTS
We align Men--Llama--8B--nnstuuce on the TL:DR| dataset over four training epochs using LoRA (Hu al. 2022). Full hyperparameter and implementation details are provided in Appendix B
Performance on The Validation Set We sample a set of 512 examples from the validation split of the dataset. Inspired by the experimental setting in Huang et al. (2025), for each epoch in {1,2, 4] we use the checkpoint obtained at the end of this epoch to generate responses to each of the 512 samples. We then take two approaches to assess the performance of our method. First, in Figure 2 we use GPT-4 to compute the win rate of SQUAREDPO against DPO, Second, in Table we use GPT-4 to compute win rates of each of the two methods against the reference model, i.c. the model before alignment. In Table I include another baseline, namely xPO (Huang et al. 2025).
In both approaches, it is evident that SQUAREDPO is robust to over-optimization. In Table ] we see that while DPO's win rate against the reference model drops substantially below 50% already in the second epoch, this is not the case for our loss. The gap in performance grows with the number of epochs. The table also shows that SQUAREDPO is more robust to over-optimization than xPO. In Figure 2 we see the same phenomenon from a different angle: when computing the win rate of SQUAREDPO against DPO, SQUAREDPO achieves statistically significant improvements over DPO in regimes prone to over-optimization (i.c., after two epochs or morc).
One can see in Figure 2 and Table that, albeit not in a statistically significant manner, DPO outper- forms SQUAREDPO on the first epoch. In this context, it is worth emphasizing that the hyperparam- eters used for training were standard choices for DPO (e-g. 3 0.01), and we did not tune them for our loss. This suggests that with appropriate hyperparameter tuning, SQUAREDPO may potentially surpass DPO also in the first epoch.
Table 1:
Win Rate on TL:DR's validation set against the base model Meta-Llama-3-1 3-8--Insrttt when trained on TL;DR. These results demonstrate SQUAREDPO's 's robustness to over-optimization compared to DPO and XPO.
Epochs
SQUAREDPO winrate (%)
XPO winrate (%)
DPO winrate (%)
50.8 0.7
51.2 + 0.8
51.8 + 1.0
2
50.6 + 1.1
48.9 + 1.1
45,0 # 1.1
4
51.0 0.7
48.3 + 1.1
34.7 + 1.3
Standard Benchmarks In Table we present benchmark results for the models obtained by train- ing Meta--Lllma---8D--nnstuuutt for one epoch on TL:DR with SQUAREDPO, XPO or DPO. Each AlpacaEvvll number is averaged over 10 seeds with confidence intervals. The MT-Bench score is an average over eight categories; the per-category breakdown is shown in Figure 5 Additional im- plementation details can be found in Appendix The results in Table show that SQUAREDPO is almost on par with DPO on these benchmarks, with DPO slightly outperforming (it is worth empha- sizing that we did not tune hyperparameters for SQUAREDPO). SQUAREDPO on par with XPO on AlpacaEval and is more performant than xPO on MT-Bench.
Likelihoud Displacement Mitigation To gauge SQUAREDPO's s ability to mitigate likelihood dis- placement, i.c. the phenomenon where winner probabilities decrease during preference optimiza-
Table 2: Benchmark results for our SQUAREDPO and two baselines. For AlpacaEval 2, we report both raw win rate (WR) and length-controlled win rate (LC).
AlpacaEval 2
MT-
Method
LC (%)
WR (%)
Score (1-10)
SQUAREDPO
29.2 0.4
24.5 0.3
7.924
XPO
29.3 + 0.6
24.3 # 0.4
7.900
DPO
29.6 0.4
24.8 0.4
7.925
7

---

Published as a conference paper at ICLR 2026
67.5
WR against DPO
60.0
57.5
55.0
50.0
Epoch
Figure 2: Head-to-head win rate of SQUAREDPO against DPO TL:DR's validation split across training epochs when using these methods to finetune Sita-Llama-3-8B-Insse on TL;DR for epochs. Error bars 10 seeds are reported.
SquaredPO
DPO
Chosen Log-ratio
0.20
-10-
Frequency
-15
0.15
0.10
-25
SquaredPO
SquaredPO Median
0.5
DPC Mean
-35
Median
0.00
-30
-20
-1c
10
Chosen Log-ratio
Epochs
Figure Left: histograms chosen log-ratios log (No (T) x)) for all chosen re- sponses in the training set 'D after one epoch of training. With SQUAREDPO, likelihood displace- ment is less severe; probabilities decrease less than under DPO. Right: the evolution of the mean and the median chosen log-ratio over epochs of training.
8

---

Published as a conference paper at ICLR 2026
tion, we begin with the histograms in Figure 3 (left), which visualize the distribution of the values flog (Tp (Yw (2) Teet (Yw z)) x)) (2.w..)))))) dubbed "chosen log-ratios'. This value is negative for winners that suffered from probability displacement. In Figure 3 To is the model obtained after one training epoch. We that SQUAREDPO is able to mitigate the likelihood displacement issue: while winner probabilities still decrease, the magnitude of the decrease is smaller. Notably, the most extreme decreases observed under DPO absent SQUAREDPO.
Beyond analyzing the snapshot after one epoch of training, we also examine how likelihood dis- placement evolves over the course of training. In Figure i the appendix, we provide histograms for the chosen log-ratios of the models obtained from 1, 2.3 and 4 epochs of alignment. These re- sults indicate that the mitigation of extreme displacement by SQUAREDPO grows more pronounced with additional training. At an aggregate level, the right panel of Figure 3 presents the mean and median chosen log-ratios for each epoch, demonstrating that the decrease DPO is by far more radical than in SQUAREDPO.
Lastly, we report an intriguing 'monotonicity' phenomenon we observe in the training dynamics of DPO: we find that in our experimental setting, nearly all (99.63%) of the chosen responses whose probability decreased in the first epoch, continued decreasing monotonically in the three subsequent epochs. That is, for nearly all responses whose probability reduced from the reference checkpoint to the first epoch, the probability was also reduced from the first epoch to the second epoch and so While the literature contains many reports of the likelihood displacement phenomenon, we are, to the best of our knowledge, the first to report this monotonicity phenomenon, which is checked on a per-winner basis.
This phenomenon is substantially reduced under SQUAREDPO, as only 4.21% of chosen responses that decreased in the first epoch continued to decrease monotonically thereafter. This empirical ob- servation is in accordance with our mechanistic understanding of the SQUAREDPO loss ($4): when a winner Yoe suffers from likelihood displacement, its effective regularization coefficient (y x) increases. This strengthens the regularization on that response and intuitively encourages its proba- bility to return toward (y Iz), thereby breaking the downward monotonicity. See Figure 7 in the appendix for the evolution of log-ratios across training for ten individual winners, and Table for additional details on monotonicity in several experimental settings.
6 RELATED WORK
The likelihood displacement phenomenon has been documented and analyzed in growing body of work (Pang et al. 2024, Xiao et al. 2024; Shen et al. 2024 Nvidia et al.] 2024: Wu ct al.) 2025 Pal
et al. 2024 Rafailov et al. 2024 Fisch et al.) 2025 D' Oosterlinck al. [025 Asadi et al. 2025
Razin et al.
2025). The connection between likelihood displacement and performance is discussed,
e.g., in RRazin al. 2025
The use of /-divergeeces for direct preference optimization was recently introduced by Wang et al. 2024), who proposed the S-DPO loss. Our theoretical results in $3.1 are non-trivial generalization of theirs. Perhaps closest to our work is Huang et al.) (2025). who also analyze an instantiation of otiimizationn While their focus is analyzing
f-DPO with formal guarantees and robustness one specific instance in depth, we take a broader perspective and provide theoretical results that hold for general f.
Li et al. (2255) investigate the role of f-divergences within the RLVR framework, focusing on preserving solution diversity during online training. Other approaches, such as f-PO (Han al.] 2024) and f f-DGP (Go et al. 2023). also use f-divergences for alignment, but do so by minimizing divergence to a target model, rather than by replacing the KI. term in the RLHF objective as in work.
There are several works that propose losses that are similar in flavor to ours, however they are introduced heuristically rather than arise from theory. Similar to SQUAREDPO, SimPO (Meng al.) 2024) can also be viewed as DPO with adaptive 8-s; however, in their case, the A-s depend only on the responses' length and are static throughout training. In 3-DPO (Wu et al. 2024 and e- e-DPO (Lee et al.] 2025), the B-s, just as in our loss, depend both on the current model and on the specific prompt and response, however, they both, unlike us, introduce additional hyperparameter.

---

Published as a conference paper at ICLR 2026
Among the works that deal with likelihood displacement, theoretical analysis most closely sembles that of Asadi et al. (2025). They, as well as others, e.g. (Pal et al. 2024; Wu et al. 2025), add an ad-hoc penalty term to the DPO loss to fight likelihood displacement, while we propose one-term loss that arises from theory, and importantly does not add hyperparameter, unlike the
mentioned approaches that have the coefficient of the penalty term (typically X) as additional hyperparameter.
7 CONCLUSION
We characterized the set of DPO-inducing functions, i.e. functions f that can be used in f-DPO, showing that convexity of f is not necessity. We studied the effect of the choice of f on likelihood displacement, which is the phenomenon where during preference optimization both the rejected and the chosen responses decrease in probability, finding that S arg min S (1) is neces- sary condition mitigate the decrease. We believe that the two properties propose, namely displacement-resisttail and DPO-inducing, may provide a general design principle for future meth- ods in direct preference optimization.
With these two properties in mind, we analyze an example a function that satisfies both desiderata, resulting in novel loss call SQUAREDPO. We find that this novel loss not only competitive with DPO on standard benchmarks, but also offers a substantially greater degree of robustness to over-optimization and a clear empirical mitigation of likelihood displacement. Alongside our em- pirical examination of SQUAREDPO and DPO, we report a monotonicity phenomenon of DPO that related to, but different from, likelihood displacement, We show that SQUAREDPO manages to alleviate this behavior of DPO as well.
Limitations First, our experimental evaluation restricted to a single dataset and model; although provide additional displacement experiments with another model and another dataset in Ap- pendix C.2 Second, the theoretical results regarding likelihood displacement focused on optimal solutions, abstracting away from optimizer behavior and training dynamics. Third, the empir- ical side, we compared our method to DPO and XPO, while other DPO-style algorithms do exist. Another limitation that all our experiments were conducted under LoRA finetuning. On the the- oretical side, while our proposed method does mitigate displacement empirically, the displacement- resistance condition we introduce is formally proved be necessary, though not established as sufficient. Identifying a sufficient condition is an important direction for future work. Finally, while our analysis applies to general choices of f. our experiments instantiate only one such function. Exploring other functions f that satisfy our desiderata is promising avenue for future research.
REFERENCES
Al@Meta. Llama 3 model card. 2024.
URL https://ggitbbb.cco/meeete
/ama3/blob/ N/MODEL, CARD md.
Kavosh Asadi, Julien Han, Idan Pipano, Xingzi Xu, Dominique Perrault-Jonean Shoham Sabach, Karim Bouyarmane, and Mohammad Ghavamzadeh. C2-dpo: Constrained controlled direct pref- erence optimization, 2025. URL
17507
Amir Beck. Introduction to Nonlinear Optimization. Society for Industrial and Applied Mathemat- ics, Philadelphia, PA, 2014. doi: 1911177797781111177655 URRh:tts:://epub siam.
brg/doi/abs/ 1137/1 9 9781611973666
Amir Beck. First-Order Methods in Optimization. Society for Industrial and Applied Mathematics, httpp:s:://epb sim. org/
Philadelphia, PA, 2017. doi: 1711133/7..98111119977999
doi/abs/10. 011./////19111119999991
Ralph Allan Bradlcy and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952. ISSN 00063444, 14643510. URL
http:://ww. jstor. arstto..or//ttaaeee2330000
Paul F Christiano, Jan Leike, Tom Brown. Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In I. Guyon, U. Von
10

---

Published as a conference paper at ICLR 2026
Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL httpss//prroeeeengs..euru..s cc.ppperrccppppprrppppppp Files/paper/2017. 21ced/aeccaaa0000e11222222222444
Robert M. Corless, Gaston H. Gonnet, David E. G. Hare, David Jeffrey, and Donald E. Knuth. On the lambert w function. Adv. Comput. Math., 1996.
Imre Csiszár. A class of measures of informativity of observation channels. Periodica Mathematica Hungarica,
191-213, 1972.
Ganqu Cui, Lifan Yuan, Ning Ding. Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: boosting language models with scaled ai feedback. In Proceedings of the 41st International Conference on Machine Learning. ICML'24, JMLR.org, 2024.
Karel D' D'Oosterlinck, Winnie Xu, Chris Develder, Thomas Demeester, Amanpreet Singh, Christo- pher Potts, Douwe Kiela, and Shikib Mehri. Anchored preference optimization and con- trastive revisions: Addressing underspecification in alignment. Transactions of the Associ- doi: 10.1162/tacl..00745 URL
ation for Computational Linguistics, 13:442 460, 2025.
bttps::/aaclanthllogy .ww./orgg//oo.... wwo.orgg/22222 tacl11..c
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. Length-comtrolled
alpacaeval: A simple way to debias automatic evaluators, 2025. UtLlhtttss:/arxiivooo
abs/2404 04475)
Adam Fisch, Jacob Eisenstein, Vicky Zayats, Alekh Agarwal, Ahmad Beirami, Chirag Nagpal, Pete Shaw, and Jonathan Berant. Robust preference optimization through reward model distillation, 2025. htttps://aaxiv..ootttt
/2005. 2t1.//.2....1166666
Dongyoung Go, Tomasz Korbak, Germán Kruszewski, Jos Rozen, Nahyeon Ryu, and Mare Dymet- man. Aligning language models with preferences through f-divergence minimization, 2023. URL
https: /tt:://arxxie org/ abs 2202 08215
J Han, M Jiang, Y Song, and Ermon. M. xu. f-po: Generalizing preference optimization with f-divergence minimization. arXiv preprint arXiv:2410.21662, 2024.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference on Learning Representations, 2022. URL httts::/ net/ forum?
id-nZeVKeeFYff9
Audrey Huang, Wenhao Zhan, Tengyang Xie, Jason D. Lee, Wen Sun, Akshay Krishnamurthy, and Dylan Foster. Correcting the mythos of kl-regularization: Direct alignment without overop- timization via chi-squared preference optimization. In The Thirleenth International Conference on Learning Representattioss ICLR 2025. Singapore, April 24-28, 2025. OpenReview.net, 2025. net forum? 2oorummiddhh000022255
URL
Sangkyu Lee, Janghoon Han, Hosung Song, Stanley Jungkyu Choi, Honglak Lee, and Youngjae Yu. Kl penalty control via perturbation for direct preference optimization, 2025, URL https:
7/arxiv. org.
Long Li, Jiaran Hao, Jason Klein Liu, Zhijian Zhou, Yanting Miao, Wei Pang, Xiaoyu Tan, Wei Chu, Zhe Wang, Shirui Pan, Chao Qu, and Yuan Qi. The choice of divergence: A neglected key to mitigating diversity collapse in reinforcement learning with verifiable reward, 2025. URL https:/ wat.vv.vvvv...////. www..bs.////.ooooooooo...oo....a 2509 07430
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-followinn models. hhtpps: / / thub wtt.:tttotuttoa hatpsalaaaaaaaaaaaaaa _evalleellll_lll 5 2023.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019. URL https://////..... http:///////s/////e//eeeeeeeeee net/ wwr.mmriii. ?
Bkg6Ricqqy Cqy7

---

Published as a conference paper at ICLR 2026
Chris Lu, Samuel Holt, Claudio Fanconi, Alex J. Chan, Jakob Foerster, Mihaela van der Schaar, and Robert Tjarko Lange. rithms with and for large language models. In A. Globerson, L. Mackey, D. Bel- grave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Advances in Neural In- Curran Associates, Inc., formation Processing Systems, volume 37, pp. http://sec.peeee.e/////peeee///e files//aper//222/// 2024. URL htttp::/ppocceednngggg// hegrissneurppps Pappeess----P-rprr-- Conference. pdf
Discovering preference optimization algo-
File/ (1t1e/77396666229b66600
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
Nvidia, :, Bo
Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H. Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper. Bryan Catanzaro, Sharon Clay, Jonathan Cohen, Sirshak Das, Ayush Dattagupta, Olivier Delalleau, Leon Derezynski, Yi Dong, Daniel Egert, Ellie Evans, Alek- sander Ficek, Denys Fridman, Shaona Ghosh, Boris Ginsburg, Igor Gitman, Tomasz Grze- gorzek, Robert Hero, Jining Huang, Vibhu Jawa, Joseph Jennings, Aastha Jhunjhunwala, John Kamalu, Sadaf Khan, Oleksii Kuchaiev, Patrick LeGresley, Hui Li, Jiwei Liu, Zihan Liu, Eileen Long, Ameya Sunil Mahabaleshwarkar, Somshubra Majumdar, James Maki, Miguel Martinez, Maer Rodrigues de Melo, Ivan Moshkov, Deepak Narayanan, Sean Narenthiran, Jesus Navarro, Phong Nguyen, Osvald Nitski, Vahid Noroozi, Guruprasad Nutheti, Christopher Parisien, Jupin- der Parmar, Mostofa Patwary, Krzysztof Pawelec, Wei Ping, Shrimai Prabhumoye, Rajarshi Roy, Trisha Saar, Vasanth Rao Naik Sabavat, Sanjeev Satheesh, Jane Polak Scowcroft, Jason Se- wall, Pavel Shamis, Gerald Shen, Mohammad Shoeybi, Dave Sizer, Misha Smelyanskiy, Felipe Soares, Makesh Narsimhan Sreedhar, Dan Su, Sandeep Subramanian, Shengyang Sun, Shub- ham Toshniwal, Hao Wang, Zhilin Wang. Jiaxuan You, Jiaqi Zeng, Jimmy Zhang, Jing Zhang, Vivienne Zhang, Yian Zhang. and Chen Zhu. Nemotron-4 340b technical report, 2024. URL https: /ttp://aavvvvvvvv org. www./r////.aaooeeeeeoee/eeee abtp///////////// 11704
Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with dpo-positive, 2024. URL https:
// llaxiivorrg/a/o..avvaa.a.......a abs/ /2022 13228
Richard Yuanzhe Pang. Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization, 2024. URL
abs/ /tes/2444.1933
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In
A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 53728 53741. Curran Associates, Inc., cc paper wttpprrsiiisseepeeee///202/ fi 2e7785500aa666666/7/aeee35566666 Paper Conference pdd.
2023. URL
Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From r to q*: Your language model is secretly a q- f-function, 2024.
12358
Noam Razin, Sadhika Malladi, Adithya Bhaskar, Danqi Chen, Sanjeev Arora, and Boris Hanin. Unintentional unalignment: Likelihood displacement in direct preference optimization. In Inter- national Conference on Learning Representations, 2025..
John Schulman. Approximating kl divergence. htmll 2020. Accessed: 2025-09-16.
Yaojie Shen, Xinyao Wang, Yulei Niu, Ying Zhou, Lexin Tang, Libo Zhang, Fan Chen, and Longyin Wen. Aipo: Improving training objective for iterative preference optimization, 2024. URL
https://aaxiis wttp//ggg////a.oo
/249 08845
Michael Völske Martin Potthast, Shahbaz Syed, and Benno Stein. TL:DR: Mining Reddit to
learn automatic summarization. In Lu Wang. Jackie Chi Kit Cheung. Giuseppe Carenini, and Fei Liu (eds.), Proceedings of the Workshop New Frontiers in Summarization, pp. 59- 63, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10. 1453/v1/W17-45000 htt.httts:///aet aathh ogy. 45088
12

---

Published as a conference paper at ICLR 2026
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. Trl: Transformer reinforcement learning. htttss:/ htwpehhhh..o.ouuugegeeeeeeeeeeee com htt..//////cgggggaa//eettett
2020.
Chaoqi Wang, Yibo Jiang, Chenghao Yang. Han Liu, and Yuxin Chen. Beyond reverse KL: gener- alizing direct preference optimization with diverse divergence constraints. In The Twelfth Inter- national Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL
Junkang Wu, Yuexiang Xie, Zhengyi Yang, Jiancan Wu, Jinyang Gao, Bolin Ding, Xiang Wang, and Xiangnan He. $ b/beetS-DPO: Direct preference optimization with dynamic SY,ettS. In The eighth- Annual Conference Neural Information Processing Systems, 2024. URL
https: http::::///peneeeww.....
Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment. In The Thirteenth International Confer- ence on Learning Representations, 2025. URL https::/openzeviert nwtp:oru//fodddeo
a3PmRgAB5T
Teng Xiao, Yige Yuan, Huaisheng Zhu, Mingxiao Li, and Vasant G Honavar. Cal-dpo: Calibrated direct preference optimization for language model alignment. In The Thirty-eighth Annual Con- ference on Neural Information Processing Systems (NeurIPS), 2024.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li. Dayiheng Liu, Fei Huang, Guanting Dong. Haoran Wei, Huan Lin, Jialong Tang. Jialin Wang, Jian Yang. Jianhong Tu, Jianwei Zhang. Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He. Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng. Rui Men, Ruize Gao, Runji Lin. Shijie Wang. Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 technical report. arXiv preprint arXiv:2407. 10671, 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging lim-as-a-jude with mt-bench and chatbot arena. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023. Curran Associates Inc.
13

---

Published as a conference paper at ICLR 2026
1.25
100
f(t)
0.29
-0.25
XPO
SquaredPO
t- 0.37
0.25
0:50
0.75
100
1.25
154
1.5
2.00
Figure 4: The f-divergence generators used by DPO (tlogt), XPO (5 1)² + t tlog)), and SQUAREDPO (1 (log t)²). The location of the global minimum of each function determines its susceptibility to likelihood displacement, with SQUAREDPO (minimum at = 1) being the most resistant to displacement.
A COMPARISON TO RELATED PREFEREENCE--PPTIIIATTO METHODS
In this appendix, discuss in greater depth the connection between results and several related methods mentioned in the related work.
The XPO loss suggested by Huang al. (2025) is instantiation of f-DPO with ft) ()² + f log f This function attains global minimum ((1) 0.56714, where W is the Lambert W function (Corless et al. 1996). Recall that our theory in $ 3.2 suggests that when- ever the ff used probabilities of all in-sample responses are expected to decrease by at least (Lemma D). This sug- gests that if for two DPO-inducing functions fi and f2 we have arg min f1 < arg min /2 S 1, then f-DPO with fi is more prone likelihood displacement than f-DPO with fz. Therefore. when comparing XPO with vanilla DPO, and recalling that the f used by vanilla DPO, f(t) = log t, attains global minimum at e 0.36788, displacement than vanilla DPO (since e < (1)). This in line with the findings of Huang et al. 2025), who analyze setting in which DPO suffers from likelihood displacement while xPO does not (Remark B.1 therein). Regarding our method SQUAREDPO, which is /-DPO with the function f()) 1 (log 2)² that attains minimum
f-DPO attains global minimum
E (0, then, under mild conditions, the
expect xPO to be less prone to probability
1, theory suggests that it is less prone like- lihood displacement than both vanilla DPO and xPO. See Figure 4 for an illustration of the three generating functions mentioned in this paragraph.
B HYPERPARAMETERY AND IMPLEMENTATION DETAILS
General training details Our implementation of the training pipeline is based on the DPO trainer from the TRL repository (yon Werra et al.) 2020). For losses, we use a learning rate of 5e-7, 0.01 (standard choices for DPO in this setting, e.g. (Meng al. 2024)), batch size of 64, a max sequence length of 2048, a linear learning rate scheduler, and the optimizer is AdamW
Loshchilov & Hutter 2019). We LoRA (Hu al. 2022) with rank r 16, scaling factor
= 32, and dropout rate 0.05. Training was performed in bf16 6 precision on 4xNVIDIA A 100-
SXM4-40GB GPUs. While we focus on LoRA fine-tuning for computational reasons, we expect the same empirical trends to hold under full fine-tuning.
In general, the x divergence is an f-divergence with f (t) 1 (t 1)2, however this function is not DPO-inducing and hence Huang et al. (2025) had add the Kl.-like term llg
14

---

preliminary experiments Published as a conference paper at ICLR 2026
DRD
Writing
SquaredPO
Humanibes
Roleplay
STEM
Roestnnng
Extraction
Math
Coding
Figure 5: Category-level MT-Bench results for the models obtained from training Meta-Llama-3- 8B-Instruct for one epoch on TL:DR using SQUAREDPO, xPO or DPO. For each of the eight MT- Bench categories, the reported value is the average score (across prompts in that category) assigned by an LLM-as-a-judee (gpt-4o). Scores are 0-10 scale, where higher is better.
Clipping Recall that to calculate the SQUAREDPO loss for sample (I,, You Bt), we need to cal- culate Bo (y.x) B/ toly)n) for any y E Sz, that is, for you and yr- In the TRL repository, and in general, one works with log-probabilities rather than with raw probabilities. We calculate 3e (y x) by cxp (min (log Tet 2) log Th (y x)) 50)). Apart from the clipping, this is equivalent to the original expression. The threshold for clipping, 50, is safe enough to prevent numerical issues and does not substantially change the learning curves based performed. We found it necessary to clip from above to prevent 33 (y, 2) from diverging to infinity floating-point representation, whereas clipping from below was necessary because Be (y. x)) approaching zero is less problematic.
Evaluation details To calculate win rates for the validation split, and to calculate win rates and length-controller win rates for AlpacaEval 2, we use the alpaca eval repository (Li et al. 2023). To calculate the MT-Bench results, we use the FastChat repository. all settings, we generate responses to prompts by sampling with temperature of 0.7.
C ADDITIONAL EMPIRICAL RESULTS
C.I MT-BENCH RESULTS BY CATEGORY
preusents a radar chart that breaks down the MT-Bench results from Table into MT-Bench's eight categories. SQUAREDPO shows slight gains DPO in coding tasks, performs worse STEM tasks, and otherwise largely on par.
C.2 ADDITIONAL RESULTS ON LIKELIHOOD DISPLACEMENT
In Figure 6 we present histograms of the distribution of the log- log(mo (yo /)/mer ha//mrr(ja (he x))
for all the 92.9K chosen responses in the training split of TL:DR when using this dataset to perform preference optimization on Meta--Lamaa33-SD-Insttee <r)TTe(f x)) is. the more severe the likelihood displacement this winner witnessed. Notice how with DPO, already after one epoch a significant portion of the winners suffered from severe likelihood displacement, while SQUAREDPO manages to control the extent to which probabilities get reduced. As training progresses, under DPO we see the distribution shifting to more negative values, while SQUAREDPO remains relatively stable.
The smaller the term tog(Te (yow
15

---

Published as a conference paper at ICLR 2026
SquaredPO DPO
DPO
0.25
0.20
Frequency
Frequency
0.15
0:15
0.10
0.00
-30
-20
-10
10
0.00
-00
-60
-40
-2c
Chosen Log-ratio
Chosen Log-ratio
(a) After epoch
(b) After 2 epochs
SquzredPO
SquaredPO
0.30
0.3%
0.5-
Frequency
Frequency
0.20
025
0.35
0.1%
0.10
0.05
0.05
122
-100
50
60
-20
140
-20
100
80
20
Chosen Log-ratio
Chosen Log-ratio
(c) After 3 epochs
(d) After 4 epochs
Figure 6: Histograms of chosen log-ratios (y Re/ x)) for all chosen responses in the training set D at different stages of training. With SQUAREDPO, likelihood displacement is consistently less severe compared to DPO, and the difference becomes more pronounced as training progresses.
16

---

Published as a conference paper at ICLR 2026
Table 3: Among the winners whose probability decreased after the first epoch, we report the per- centage that continued decreasing in later epochs. Results are shown for DPO and SQUAREDPO in two settings: Llama TL;DR and Qwen UltraFeedback.
Liama TL;DR
Qwen UltraFeedback
Condition
DPO (%)
SQUAREDPO
DPO (%)
SQUAREDPO (%)
Decrease from + 2
99.99
10.91
93.72
25.56
Monotone decrease +
99.74
9.38
39.99
6.24
Monotone decrease -
99.63
4.21
35.57
1.64
-10
Log-rati -attraaio
Log-ratio
33
-40
-50
Epochs
Epochs
(a) DPO
(b) SQUAREDPO
Figure 7: The evolution of the log-ratios log(To(ywr //) Weee(ye x)) during training for a random subset of 10 samples from the TL;DR training set. SQUAREDPO mitigates the 'monotonicity' phe- nomenon observed in DPO.
In Table we provide additional results regarding the monotonicity phenomenon reported in $5.2 For example, when using the TL;DR dataset to finetune -eta--lama-3-8B I-St-Inst with DPO, 99.99% of the winners whose probability decreased after the first epoch continued decreasing in the second, compared to only 10.91% % under SQUAREDPO.
To diversify our experimental setting in the context of likelihood displacement across models and datasets, we additionally train Srven--05SB-Inttruue Yang et al. (024] on the UltraFeedback dataset (Cui et al. 2024) for four epochs. The empirical monotonicity analysis for this case is reported in this setting, 35.57% of winners who decreased under DPO in the first epoch contin- ued to decrease monotonically until the fourth epoch, whereas under SQUAREDPO this proportion merely 1.64%. Importantly, the trend that SQUAREDPO manages break the downward mono- tonicity of winner probabilities compared to DPO holds consistently across all experimental settings. A mechanistic explanation of this trend is provided in the last paragraph of $5.
Table
To further illustrate the trend in Table 3 we track the log-ratio trajectories of ten randomly selected winners from the TL:DR training set across the four training epochs of Metaa-Llama---8--nnnnnne under both DPO and SQUAREDPO, using the same winners for each method. We visualize this in Figure 7 Notice that under DPO, all ten winners display monotonic decrease in probability, while for SQUAREDPO this is not the case. Moreover, notice the different scales in the y-axes; under SQUAREDPO, the decrease of smaller magnitude.
D THEORETICAL RESULTS OMITTED FROM THE PAPER
D.1 THEORETICAL RESULTS OMITTED FROM SUBSECTION 3.1
We start by defining the notion of an interior- interior-inducing function, briefly mentioned in 3.1
17

---

Published as a conference paper at ICLR 2026
Definition 3. We say function R+ R is interior-inducing for any reward model ra. any reference policy ref and any regularization coefficient 3 every optimal solution Te to the generalized RLHF problem assigns strictly positive probability to all continuations, i.e., wo (y x)) > 0 for all I. y.
The condition Definition is equivalent to requiring that, for each prompt æ, T (- relative interior (Beck) 2017) of the simplex, which motivates terminology. Next, establish that this definition is equivalent to the definition of a DPO-inducing function.
x) is in the
Lemma 3. Let f R4 + R be a continuous function in R4, which is continuously differentiable in R++. Then, f DPO aPO-inducing if and only if it is interior-inducing
Proof. Assume first that not interior-inducing. Then, there exist Tox Tref, and such that an optimal solution to (5) satisfies (yw x) for some (z, Mos y1) E D. Now, recall that for to DPO-inducing, substituting optimal solutions to ( into the BT model ① should yield the needs to express the rewards difference
f-DPO loss. However, to perform this substitution, rs (x, yw) T (x. y1) in terms the optimal Te (yw x), ne (yt z). Since we have Ty (yw x) = this is impossible.
Assume that f is interior-inducing. We need to show that f DPO-inducing, i.e., that substituting any optimal solution of problem (5) into the BT model (I) yields the f-DPO loss (6). Let To be an
optimal solution to 5). Since f is interior-inducing, by definition Ty (yow x)) >> and Te (y1 x)
0 for any (xr Mus y1) D. Now, we define
(T) := (o (ro (z, y)] y [Io (-1z)
Reef (-1=)]
which is the objective function of problem
(5) Using technical result (see Lemma 4 below) implies
(g) Ög(ma)
mxg (yw xx) ixg (y1 x)
which reads as follows
no (Nwr x))
To (31 xr) Teef (y1 x))
TΦ (I, kw) B8
To (x,, 01) 33
Tret (2o xx)
Rearranging the equation gives
To (3ow x)
Te (3x ())
To (x, Vho) T (x,, 35 and substituting this difference into the BT model
35"
Tref
(yow
xr)
Tire (311 .x))
yields the f-DPO loss
(6) as required.
We next state Theorem D which provides full characterization of DPO-inducing functions.
Theorem 1. Let f 1R, R be a continuous functtinn in IR,,, which is continuously differentiable in +
i (0) co, then f is DPO-inducing.
ii Suppose (0) < xc. Then, f is DPO-inducing if and only if the ratio [(t)=((0) is not bounded from below in any right neighborhood of (i.e., Va > 0, VM R, St (0, such that ((1))((0) < M).
The proof for Theorem can found in Appendix Recall that Corollary T we add an assumption under which the condition of the characterization becomes easier check.
D.2 THE OPTIMAL SOLUTIONS OF PROBLEM
In this appendix, provide and briefly discuss the optimal solution(s) Problem 7 under the additional assumption that f is convex. differentiable, and f is invertible. We do not prove it here, but rather rephrase it in simpler notations and prove it in Lemma
18

---

Published as a conference paper at ICLR 2026
First, notice that Problem 7 is separable over different 2-5 and hence provide the optimal
solution Ty separately for each x E D. So let x E D. Denote iz == maxyes, ro y). the condition
Tee (y x.) p'-1 (TO (1, v) ix yES,
Σ
(10)
B
holds, then the set of optimal solutions Ta (: x)) is the set of no (- x) that satisfy
(y x) Tee (y x)) f (o((x))
Vy Sg,
(y x)) 0,
Vy $ Sn, To (x, < fm
Σ
To (y x)) =1 Σ yES,
Ter (y x)) fr-1
(o22(zooee (ralzig)ff)
18
V =
(11)
Otherwise, the unique optimal solution is:
no (y x) = Tret (y x)) pr-1 (ro(xa)tH)
Vy E Sz.
(12)
To (y x)) 00
Vy $ Sz.
where y is solution to Eyes, Fee (y at) M'1 (ola,y))
This wraps up the characterization of the optimal of the weird optimization problem 77. Notice that in both cases, the optimal solutions can assign zero probability to out-of-sample responses. This demonstrates that Problem 7 does not guardrail out-of-sample responses to stay close in probability to Tee. We rephrase and prove this characterization with more convenient notations in Lemma We note that this result appears in Asadi et (2025) for the special case fKI. (t) = log
E PROOFS
E.1 PROOF OF THEOREM 1
Proof. For simplicity of the proof, we define
max 9 (p) = fr-p--BD (p q)
(13)
DEA"
We first prove the first item. Assume that (0) = oo, and will show that any optimal solution p must be in A {p E A" p > 0}. Indeed, any vector p. for which p; = 0 for some i E [n]. we have that (p)
co. This obviously implies that such vectors p cannot be maximizers of g. Therefore, we get that any optimal solution lies in A
We now prove the second item. Assume that (0) 00.
We assume on the way of contradiction that the ratio t [- (t) - (0)) is bounded from below. We will prove that not specific r and q such that p* C1 is an optimal solution.
by constructing an instance of the problem (3) with
To this end, since the ratio - '[f f (0)) is bounded from below, there exist M E IR and
such that for all L E (0, a) we have [- (1) f [0)] > M. Since (- - f (0) is continuous (a, n]. by the Extreme Value Theorem there exists K₁ such that ( -
(0) Z K₁ for all
min {K1, M }. get gf (t) - (0)) > for all E (0, n]. Moreover, let D = max4e[1.21) f' (t), which is finite by the assumption that f is continuously differentiable in R++ and by the Extreme Value Theorem.
t E [a, n2.. Denoting
Now, we are ready to define r and q. We consider problem 13 with 4; = 1/n for all E [n]] Ti max {BD 3m + 1,11, and r; = 0 for all i 2 2. We will show that p* = ej is an optimal solution. To this end, since the problem obviously must have a maximizer, it enough to show that no p with pa can be optimal solution.
19

---

Published as a conference paper at ICLR 2026
We take p An with P1 such that p; for some i [n]. Let p ps (e1 the probability vector obtained by moving all the mass from 2 1.
ei). that is,
Then, from the definition of p', get
( - (p(p) = (p) p:) 3
Σ 1=1
- (mph) rrp1 + Σ
- / (npt)
[=1
3
3
-1P: f (n (p1 + pp))
f (0) + (mp1) + (np:)
n
n
n
TIP: + (np,) - -()) (f (mpi + np.)) - (mpi))
Dividing by pj > we get
(p') 9 (p)
ri 3 (np,) (npi) -f 33 (np1 + np,)) f (np1)
pi
np:
npi
2n 3m B (mpi + mp:)) - (mip))
(14)
nips
where the inequality is because np: E (0, n..
Now, bound the last term, first we notice that if p1 1/n, then p is not optimal. Indeed, this case there is necessarily j # 1 such that Pj > 1/n. Therefore, by considering the value of on the vector where Pj and pl switches, we get higher value than g()) since ri
ry.
Therefore, we can assume e E (np1, np1 + mp:)) such that
1/n. Moreover, by Lagrange's mean value theorem, there exists
(npi + np.) - f (nep1) (c) S max f (1) = LE tE[1,2m]
npi
where the inequality follows from the fact that e E [1. 2n]. Indeed, since 2 1/n, we have 1 np1- Also, np1 np, since pt + pt S 1. Therefore, E [1,
Using this in
14). we get
g (p') (p)) 2 3m BBDD 2 3D 33 + + 8m - 3D = 1>0,
where the second inequality follows from the fact that rı 2 BDD 3m + 1. Therefore,
(p') -p)) > 0,
(15)
which shows that is not optimal, as required.
Assume that (0) and that the ratio t 'f (t) (0)) is not bounded from below, and we will prove that f is interior-inducing.
Let E R . 4. and B 0. We need to show that any optimal solution to
max g (p) == (r"p -0, (p q)}
P
satisfies E A We assume in contradiction that = 0 for some E [n] and we will prove that is not optimal solution.
Let j E [n] be such that p, > 0. and define p' = + (e; - -e). Notice that for every = < Pj+ we
indeed have O E ". Then, for all Pj, have
20

---

Published as a conference paper at ICLR 2026
9 (p') 9 (p) = =r- (p' -p) - Σ
qef
Pi
3
Σ
qef
Pt
98
1=1
=ar" (ei - -ee)
pf
+
+
q.f
Pi
+
qjf
Pi
9i
qj
qi
43
= (rr - r3) - 3q f
(5(
f (0)
Bqj
f
Pj
f
Pj
(16)
qi
q
We now wish to find e such that g (p²) g (p) > 0, which will prove that p is not an optimal solution. Note that
4i
Pi
P
Po
Pi
lim 8 E-+
43
lim E E-2
9:
f
Pi 4j
R.
%
since ʃ is differentiable at R++. Therefore, by the definition of the limit, there exists & > 0 such that for all E (0,8),
s
P1 91
Pr
f
Pi
<
1,
qi
which means that
(f
Ps 9:
Di-e
9i
93
Pj
-1.
(17)
6
Considering the ratio condition with M
P1
(r, -.) - 2 and min (,) /
95
get that there exists f E (0, a) such that f(t) f()) and EO E Pz). Thus,
< M. Defining &o = tq. we then get ep E (0,
(0)
f()
< M
(18)
Therefore, by combining from 16. 17 and 18] we get
Pj s
38o > 0. -1)
9 ( (p) sa (r: - ry) 3Mso + B5o
1j
where the last equality follows from the definition of M. This shows that g (p g (p), which contradicts the optimality of p.
E.2 PROOF OF COROLLARY
Proof. Since lim fl (t) exists and f is continuous, there is L such that
ft) - f = (0) = lim f (t) L. +
lim t t
(19)
Therefore, L = -00 and only if the ratio (() (0) is not bounded from below. This concludes the result immediate consequence of Theorem
E.3 PROOF OF LEMMA
Proof. Since f is DPO-inducing, it satisfies the conditions in Theorem I Therefore, straight- forward generalization of our proof of Theorem I can show that any optimal solution Ta to Prob- lem satisfies mo (y a:) ( for any y Sz. Therefore, for any (x, Bw. 91) E D have
21

---

Published as a conference paper at ICLR 2026
To (Yw 2x) > 0 and (yu ()) 0. This implies, by Technical Lemma the following equality of derivatives:
Ög(TT)
Ög(TT)
(20)
mh (Nw x)) dre (31 x) where we denote by 9 (TO) the objective function of (7). By calculation, this gives us
Jeet (yhe (x) )
To (Var .)) 38
To (y 1:) Tret (y x)
T (w, T (2,, yi 3
(21)
which when substituted into the BT model yields the f-DPO loss
(6), as required.
E.4 PROOF OF LEMMA
Proof. Suppose in contradiction that there exists an optimal solution тe to Problem 7 such that for
some x and for some E Sz, To (y 2) > c. Weef (y Iz). Denote by ÿ E arg y . To (z, an out-of-sample response for x with a maximal reward. We define
c. Tet (y z)
y:
T (= x))
To (ú a:) + To (y xt) Th (u xx),
(To (= xx),
{v.0}-
Since (0, 1]. we immediately obtain that T (- (z) is indeed valid probability distribution. From ( and using the definition of TT z), we have
To (y x)
g (TT) - 9 (mO) = v' (ú 2)) ra (ar, 0) vb (y x)) re (2, 9) 3 (y | ())
Tret (y)
x))
a (y
xr)
mo (ú 2)) re (ar, 0) no (y) xx) re (xx, v) 3 (y (x) f
Jeat (y x))
re (z. 0) (mo (y x) 7/ (y/ x)) rΦ (x. (No (y [x) na (y x))
no (y x))
T (y (2) f (c) S (y x))
Teat (y x²)
Now, since is the unique global minimum of have f (c) f [ and by our assump- )
tion > 0, which implies that
g (mb) - g (T)) > rs (x, () (mo (y x) The (v x)) ro (x. (To (y .) my (y ()) Now, recalling that T (ar, 9) and that Te (y/ .²) > Ta (y) x)) by construction of nig. we get that (I²) - g (Na) in contradic- tion to the optimality of ma.
ms (z, v/) 2 ro (z, y) by the assumption of the Lemma
F TECHNICAL LEMMAS AND THEIR PROOFS
We denote and define the - 1 dimensional simplex by =" {pE 1E
Lemma 4. Let A + be a function and p* be an optimal solution of the problem
max (p)
(22)
DEA"
Suppose that g is differentiable at p". Then, for any i,j E [n] for which P,P > 0, we have 0gip") 0g(p") Op. 0p;
Proof. Suppose in contradiction dg(p") dg(p") dp,
Let v 0j- - e; where e, and e, are the i-th
and j-th standard basis vectors in R Then, we have
(g = 0g (p")
0g (p*)
> 0.
Öp,
Op:
22

---

Published as a conference paper at ICLR 2026
Since g is differentiable at p'. using the definition of the directional derivative of g along the vector V (see Section 1.5,1 in Beck 2014). ) we get that
9 (p'
+ hv) - g (p")
V (p") lim h-0
h
Therefore, for a small enough 0, we have
g (p' + hv) > 9 (p")
Furthermore, since P > 0, there exsits small enough h > 0 such that p" + hv E A", These two facts contradict the optimality of p". Since similar arguments will derive contradiction in the we obtain the desired result.
case where 0g(p") ) > Ups
@g(p")
Bp;
Lemma 5. Let f - - invertible. Let S be strict subset of [n] {1,..., n... For given E RR, 9 A. consider the following problem:
be convex DPO-inducing function, which is differentiable and I' is
Σ
piri - -8 =. (vi/as) iES
DIRX AEA
(23)
Denote f = max and 2 = g. ((r, B)) for any i S. If Eies < L, then the set of optimal solutions to (23) can be described as all vectors p satisfying
= Zi
p: = 0,
Vi $ S, r3 R
(24)
Σ P: 1 Eies 2t.
igS PiEP
Otherwise, the unique optimal solution
23 is:
P: = 9.8
Fitt
Vi E S,
(25)
=0
Vi V. S.
where μ is solution Eies ((r: + m)/B)= 1.
Proof. Since the problem is convex (since f is convex) and Slater's condition holds, the set of optimal solutions coincides with the set KKT points. The Lagrangian in this case is given by
-r. B
Σ qi
Pi yi
+
(A,
+
Em1-1)
(26)
EE
where A R. and μ R. The KKT conditions are
dp. r, - 38 21 + WES.
aL
0L ri + \ + 4 == 0,
vi $ S, (feasibility),
Op:
Pi 0, Σ Pi 1,
A 0,
Vi E [n], (multipliers),
A 0,
Vi E [n],
(complementary slackness).
23

---

Published as a conference paper at ICLR 2026
Since assume that is DPO OPO-inducing, by Lemma we get that it is also interior-inducing and therefore any solution p to (23 satisfies ps > for all i S. Therefore A = 0 for all i E S, and the KKT conditions can be equivalently written as:
Pe = f
tu
Vie S.
+
Vi $ S, (feasibility),
E A"
À 2 0,
Vi $ S, (multipliers),
App: = O,
Vi $ S, (complementary slackness).
We now split the proof into the two cases as mentioned in the lemma. Recall that f max;gs F and 2, qf ((r: f).3).
Assume that Eies 24 < First, the points described 24 are all KKT points with / -f and
>: =0 vi f A A ra Vi Vi f.
It remains show that other KKT points exist. To this end, let p be any KKT point. We will prove that p coincides with one the solutions in (24). By feasibility, it suffices establish that p: z; for all S and p: 0 for all i such that r; < f.
For any & S, we have that r; + A; + = 0. Since X; 2 0, we have that μr S -r; for all & S and therefore also μ -f. If p; 0 for all S, since Eies 2 < obtain contradiction since
Σ EE
-P=
Σ 98-
ri + 1 3
9ef- iE
1
EES
iES
where the first inequality is because f increasing (by convexity of f) and since p S
Hence some p: > f $ S. Complementary slackness implies A; and thus r; = n. Moreover, r Xy for all j & S, since r, + X3 -p. Since X 2 O, we have that ri 2 ry for all j $ S. which means that r, = f. Hence μ -f, which means that p, = 2 desired.
Assume that Cies 2; 1. In this case, prove that the unique KKT point is the point given in 225). Note that by feasibility it is enough to show that p; 0 for all & S.
Suppose, in contradiction, that Pe > for some & $ S. Then, d and hence μ = -rE- Moreover,
12 where the first inequality is by feasibility of p. the second inequality is since f-1 is in- creasing (by convexity of f) and ry < f, and the penultimate inequality is rearrangement
Σ --f' iES
P:
Tt
2 PE Σ r iES
ri 8 2 PE + + (27)
3
the assumption of Case (recall that 2, = q.-"-1 ((r; - f)/8))
24


Reviews:
[{"review_id": "DAydijriuU", "reviewer": "Reviewer_Zbuk", "summary": "This work dives deeper into DPO with f-divergences ($f$-DPO): (1) Proves more relaxed DPO-inducing condition, i.e., a sufficient condition that yields the $f$-DPO loss; (2) Proves a displacement-resistant condition, i.e., a necessary condition for $f$-DPO to avoid probability displacement issue; (3) proposes SQUAREDPO, a special case of $f$-DPO that satisfies the DPO-inducing and displacement-resistant conditions, and is empirically demonstrated to have better performance and mitigated displacement issue compared with vanilla DPO.", "questions": "(1) What is $\\mathbb{R} _ {++}$ in Corollary 1? \n\n(2) Both $\\ln$ and $\\log$ are used in the paper. Do they mean the same? \n\n(3) Is there any sufficient condition to prevent probability displacement? \n\n(4) What metrics are used in Figure 4?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The perspective of looking at $f$-DPO is novel, with non-trivial theoretical results, which also yields a new DPO variant that is theoretically and empirically better than vanilla DPO. Many benchmarks are used in the experiments. There seem to be sufficient details to reproduce the experiments.", "weaknesses": "Figure 4 does not show significant performance difference between DPO and the proposed SQUAREDPO. \n\nThe paper could be better if \n\n(1) Sufficient conditions for displacement-resistancy could be provided. \n\n(2) $f$-DPO with more choices of $f$ could be empirically compared, such as $\\chi^2$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OxYAfdiQ97", "reviewer": "Reviewer_YofB", "summary": "The paper proposes exploring DPO variants that are based on a generalization of f-divergences that does not require convexity of f. They also analyze how these variants such as SquaredPO prevent probability displacement. They derive a set of functions that are DPO-inducing and a set of functions that are displacement resistant. They show that SquaredPO in particular is more robust to over-optimization and mitigates displacement while maintaining comparable performance to DPO.", "questions": "- Could you provide justification as to why the convexity of f may be a concern?\n- Could you provide comparisons to existing methods such as f-DPO or Chi-PO?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "Analysis - The paper provides a thorough analysis of the properties of different objectives and characterizes a wide range of possible objective functions. They also provide direct empirical comparisons of changes in likelihood as well as win rate and benchmark performance. \n\nClarity - The paper provides a clear presentation of ideas with detailed explanations. The theoretical definitions and interpretations walk through the key ideas and the experimental setup is well described and provides support for the claims.", "weaknesses": "Contributions - While the analysis is thorough and claims made are well supported, there is a lack of comparison to other methods that aim to achieve the same goal and it is unclear whether the convexity constraint is an issue. Figure 1 shows that there are multiple convex functions which are DPO-inducing and displacement-resistant, many of which have already been explored and have successfully mitigated over-optimization of displacement. As a result, without further comparison to these existing methods or a strong justification as to why the convexity constraint is limiting, it is unclear whether the paper provides a method or insights that goes beyond existing methods. The key ideas involved are also primarily extensions of existing ideas in f-DPO and the cited paper for Lemma 2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XOS5sc6Rt5", "reviewer": "Reviewer_iniP", "summary": "Likelihood displacement has been know as one of the problems of DPO. This paper tries to solve this likelihood displacement issue by changing the KL divergenge regularization term in RLHF objective to nonconvex f-divergences. It theoretically shows that a convexity is not a necessary condition. From this theoretical finding, this paper suggests square of log function. It is both displacement resistant and DPO inducing. Empirical results on TL;DR and MT-Bench show that SQUAREDPO alleviates displacement while maintaining alignment performance comparable to standard DPO.", "questions": "- How much displacement is mitigated? \nHow it improved the model performance empirically?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Rigorous theoretical analysis.\n- Easy to adapt loss (only regularization term is changed yet the validity is proved)", "weaknesses": "- Experiment is narrow.(model is only llama, the only baseline is naiveDPO)\n- In experiment, the performance gain is marginal or even similar to naive DPO. Specifically, if the same performance for squareDPO be achieved with epoch 4 as DPO with epoch 1(Table1), why we should use SquareDPO?\n- No performance analysis without LORA", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gtWoizxqVc", "reviewer": "Reviewer_Y1Xo", "summary": "This paper studies a generalization of $f$-DPO to non-convex $f$ generator functions, and identifies a simple condition to enforce on the generator function $f$ to be more resistant to the phenomenon of \"likelihood displacement\" affecting DPO, by which the probabilities of chosen responses in the preference dataset tend to 0 as the training progresses. The identified condition is that the minimum of $f(t)$ should occur at $t \\geq 1$. They call such functions \"displacement-resistant\".  The paper proposes a specific generator function that they term SquaredPO and compares it against DPO on TL;DR.", "questions": "- Q1) Your \"displacement-resistant\" condition is a necessary, but is it sufficient to avoid likelihood displacement? If not, isn't the name \"displacement-resistant\" a bit misleading?\n- Q2) Your displacement-resistant is quite simple. Wouldn't it predict that a simple change of variables, say $t' = t - (1- e^{-1})$ for the reverse KL, work just as well?\n- Q3) I think you might want to clarify early on that the usage of $f$-divergences here concerns the generalization of the regularization penalty term only, to distinguish it from other usages in LLM post-training where they have been used as the loss to minimize in a distribution matching objective (https://arxiv.org/abs/2302.08215). Also, not sure if this is relevant, but I also note that there was another contemporary paper that studied the $f$-divergence generalization of the penalty term in the context of RLVR: https://arxiv.org/abs/2509.07430.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "- S1) The paper discusses novel ideas on the realm of using $f$ divergences as penalty terms in KL-regularized RL, specifically in the context of $f$-DPO.\n- S2) It comes with an extensive theoretical analysis of the discussed algorithms", "weaknesses": "- W1) The paper, while clear and well-written, was also a bit terse to read for me. This is highly subjective, and I don't weigh this weakness strongly in my evaluation.\n- W2) The usage of a non-convex $f$ felt quite arbitrary, and there is no comparison with a convex displacement-resistant function. In a sense, the generalization to non-convex $f$ might be interesting, but it's unclear from the content of this paper what can we gain from it.\n- W3) More in general, the comparison of the proposed technique is very limited as it only considers one baseline (DPO) and one dataset (TL;DR) , as noted by the authors in the limitations section.  While the authors claim that they leave further comparisons for future work, I don't think this is an aspect that can be relegated to future work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)