AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization
Authors: Junkang Wu
Xue Wang
Zhengyi Yang
Jiancan Wu
Jinyang Gao
Bolin Ding
Xiang Wang
Xiangnan He
Venue: ICML 2025 poster (2025)
Keywords: Direct Preference Optimization
LLM's alignment
Primary area: deep_learning->large_language_models

Abstract: Aligning large language models (LLMs) with human preferences requires balancing policy optimization with computational stability. While recent offline methods like DPO and SimPO bypass reinforcement learning’s complexity, they face critical limitations: DPO relies on static reference models that degrade with policy updates, and SimPO assumes a uniform target reward margin that ignores instance-wise preference strength. We propose AlphaDPO, an adaptive preference optimization framework that dynamically reparameterizes the reference distribution to address these issues. Our key innovation lies in an implicit reference model \(\hat{\pi}_{\text{ref}} \propto U(y|x)(\pi_\theta/\pi_{\text{ref}})^\alpha\), which interpolates between policy-driven specialization and uniform exploration while enabling instance-adaptive reward margins. Theoretically, we prove AlphaDPO implicitly controls sequential KL divergence between iterative policy updates, ensuring stability even with poorly calibrated reference models. Empirically, AlphaDPO achieves state-of-the-art performance on AlpacaEval 2 (58.7\% LC win rate) and Arena-Hard (35.7\% win rate) across Mistral2-7B, Llama3-8B, and Gemma2-9B, demonstrating robust alignment without multi-stage training. Our work establishes adaptive reference reparameterization as a principled mechanism for preference optimization.

AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization
Junkang Wu² Xue Wang?
Zhengyi Yang'
Jiancan Wu Jinyang Gao Bolin Ding² Xiang Wang" Xiangnan He
Abstract
tions (Leike et al., 2018), ensuring they are helpful, honest, and harmless (Askell et al., 2021). Reinforcement learn- ing from human feedback (RLHF) (Christiano et al., 2017; Ouyang et al., 2022; Stiennon et al., 2020) is a widely used method for fine-tuning LLMs to achieve this goal. However, RLHF faces challenges, particularly in computational effi- ciency and training stability due to its multi-stage process. Recently, alternative offline algorithms like DPO (Rafailov et al., 2023) and SimPO (Meng al., 2024) have been explored. Specifically, DPO reparameterizes the reward function in RLHF to directly learn policy model (TT) from preference data, removing the need for an explicit reward model. Building on DPO, SimPO removes the reference model requirement but introduces a target reward margin y to enhance the separation between response pairs, achieving leading performance. This naturally raises the question:
Aligning large language models (LLMS)) with hu- man preferences requires balancing policy opti- mization with computational stability. While re- cent offline methods like DPO and SimPO by- pass reinforcement learning's complexity, they face critical limitations: DPO relies on static ref- erence models that degrade with policy updates, and SimPO assumes uniform target reward mar- gin that ignores instance-wits wise preference strength. We propose AIphaDPO,, an adaptive preference optimization framework that dynamically repa- rameterizes the reference distribution to address these issues. Our key innovation lies in an im- plicit reference model कारर x U (Tyla))TT/// mee) which interpolates between policy-driven special- ization and uniform exploration while enabling instance-adaptive reward margins. Theoretically, we prove AIphaDPO implicitly controls sequen- tial KL divergence between iterative policy up- dates, ensuring stability even with poorly cali- brated reference models. Empirically, AIphaDPO achieves state-of-the-eeeeee art performance on AlpacaE- val 2 (58.7% 7% LC win rate) and Arena-Hard (35.7% win rate) across Mistral2-7B, Llama3-8B. and Gemma2-999 -B,, demonstrating robust alignment without multi-stage training. Our work establishes adaptive reference reparameterization as a princi- pled mechanism for preference optimization. The code is available at https: htt///gttttt //ggttttbbb.oo// com/ junkangwu/ hatpa//appppaaaaaeee
Do we really need a reference model in the alignment process?
This question prompts a deeper analysis of SimPO's under- lying mechanism: it can be viewed as a variant of DPO where the original reference model Nee is replaced by an implicit reference model म In SimPO, the target reward margin y actually reflects constant difference between the log likelihoods of selected response and rejected one, i.e., (log (y (z) - log (M)(()))))) As the constant difference 7 is independent of arbitrary responses, this im- plicitly assumes a uniform reference distribution (cf. Fig- ure 1). By tuning 7, SimPO effectively finds an "ideal" uniform implicit reference model, yielding substantial per- formance improvements over standard DPO, particularly when the original reference model net is suboptimal (Hong et al., 2024). While conceptually appealing with empir- ical improvements, SimPO has two inherent limitations: (I) Applying the same target reward margin to all pairwise comparisons ignores the variability in the data (Yang et al., 2024; Wu al., 2024), potentially compromising decision quality in some cases; and (2) The implicit assumption of uniform reference model somehow lacks solid theoreti- cal foundation. These limitations could hinder the model's ability to achieve alignment across varied training data, es- pecially in domains with diverse preferences or complex reward structures (Morimura et al., 2024; He al., 2024).
1. Introduction
Learning from human feedback is essential for aligning large language models (LL.M)) with human values and inten-
'Equal contribution 'MoE Key Lab of BIPC, University of Sci- ence and Technology of China 'Alibaba Group. Correspondence to: Xiang Wang Ixiangwang1223 1223 com,, Xiangnan He <hexn @uttnnuutt..eo ceeeuscc.dee edu.cn>.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

AlphaDPO; Adaptive Reward Margin for Direct Preference Optimization
Implicit reference model
Loss Function
Topa (Tga
Ours
ReeDDPO (x) "Trer(: (x)
loge
Blog
Tee Ne (x)
Blog ma(
+6.4
Lsimro (Tai Trud)
BretsimPo( (x) UC
E log a
log ('wl))))
log my (11b/))) -r)]
+12.0
[ywl
Lours Tlyer)
ReetDurs C (x) UC (x) ( (x)
B
B ly
C 10)
logo
log (r
log no (x)
sgly aM)
Matral
Laama33
Liama3
Gammaz
Figure 1. DPO, SimPO and AIphaDPO mainly differ in their implicit reference model, as indicated in the shaded box, leading to variations in their respective loss functions. AlphaDPO outperforms DPO and SimPO across a wide range of settings on AlpacaEval 2.
In response these challenges, propose an adaptive pref- erence distribution, which gives rise to an adaptive reward margin for different response pairs. We term this simple yet effective preference optimization algorithm AIphaDPO. Specifically, the adaptive preference distribution heuris- ) Here, (z) inspired by SimPO, employs a uniform dis- tribution establish an initial target reward margin, while the term (mo(v) (y/zz) / (x)))) adjusts the balance between the policy model me and the reference model Trer achieve personalized reward margin. When a = 0, AlphaDPO reduces to SimPO; as a increases, the ratio between Ty and Tee becomes dominant, enabling a personalized, dynamic target. More important, AIphaDPO offers several intriguing theoretical insights: We demonstrate that AlphaDPO bal- ances alignment and diversity via KL divergence control. By approximating the sequential KL divergence between the policy and the reference model, AlphaDPO achieves computational efficiency and robustness, particularly when the reference model not well-calibrated at the token level.
though the underlying latent reward function r" y) that governs these preferences is not directly observable, the Bradley-Terry (BT) model (Bradley & Terry, 1952) pro- vides a framework for modeling pairwise comparisons:
exp(r (1r, vwo)
tically set as: #er((y(z)) = U(ylz)
Ply > 3r(x)) exp yw)) + exp(r* (x, w:))
(1)
where r' r"(,,)) y) assigns a latent reward to each response given prompt r. The goal of offline alignment is to learn a policy Ty that approximates r" r"(xz y) using D.
Reinforcement Learning from Human Feedback (RLHF). Classical offline alignment algorithms employ reinforcement learning with KL-regularized reward ob- jective (Bai et al., 2022: Ziegler et al., 2019; Ouyang et al., 2022), defined for regularization parameter 17 0:
max E Ey-. y)] 30 ( [=<e(()) Wo
(2)
where r (z, y) is the reward function learned using the BT model on the preference dataset, mp the policy model being optimized, Tet the fixed reference policy, typically obtained via supervised fine-tuning. The KL-divergence regularizes the policy to remain close to the reference model.
Extensive analysis indicates that AlphaDPO leverages pref- erence data more effectively by assigning personalized mar- gins to each pair, resulting in an improved policy model. As demonstrated in Figure 1, our method consistently outper- forms DPO and SimPO across three base model settings (Mistral2-7B. Llama3-8B, and Gemma2-9B) on AlpacaEvval 2 and Arena-Hard (cf. Section 5). Notably, we achieve a 58.7 length-controlled win rate on AlpacaEval 2, and a 35.7 win rate on Arena- Arena-Hard, establishing it as the strongest 8B open-source model to date.
Directed Preference Optimization (DPO). DPO (Rafailov et al., 2023) a leading offline preference optimization method. Instead of learning an explicit reward model, DPO reparameterizes the reward function r(x,, y) using a closed- form expression involving the optimal policy:
To(y r(x, u) = =3 =ll (ye()))
+ 3 log Z(x),
(3)
2. Preliminaries
where Z(x)) is the partition function independent of y. This leads the DPO loss for any triplet (xc, Basy y1):
Offline Alignment. In the offline alignment problem, we have access to a dataset D = {Ix, Vw, }/}} comprising prompts r and labeled response pairs (y.. 31) obtained from reference policy Tee Here, you is the preferred (winning) response and y is the less preferred (losing) response. Al-
LDPO meer = -B(r.vu-s)>>
log (Blog (z( Teet (rer(Mol22) (2x) To()) Tret Trer (z)
(4)
-Blog
2

---

AlphaDPO; Adaptive Reward Margin for Direct Preference Optimization
where (-) denotes the sigmoid function.
DPO
40-
Simple Preference Optimization (SimPO). SimPO (Meng et al., 2024) introduces two key contributions: (1) a length- normalized reward, calculated as the average log-probability per token of a response under the policy model mo, and (2)
20
a target reward margin 7 to ensure the reward difference between winning and losing responses exceeds this margin.
0-
The SimPO loss is formulated as:
-20
LsimPO(TT) -B.
log
B [Bhol
log (B
B [yl
log ())())
-40-
100
300 400
(5) where lyl denotes the length (i.e., number of tokens) of response y, normalizing the reward by response lengths, and y is the target reward margin.
Step
Figure 2. log Tzet (ful())) log Rret (uu (ar) along the training steps. The random fluctuations suggest Tyef struggles to distinguish be- tween the preferred and less preferred responses. same policy model, their selection probabilities diverge through the reward model's 's scoring mechanism. Specif- ically, winning response (yw) are those assigned higher scores, while losing response (yr) receive lower scores. Even under a uniform policy, these selections remain distinct be- cause their selection probabilities are tied to their relative scores, not to the underlying uniform distribution over the response space.
3. Method
In this section, establish unified framework that con- nects DPO and SimPO (Section 3.1), highlighting the criti- cal role of the reference model in preference optimization. We then introduce AIphaDPO (Section 3.2), a new prefer- ence optimization algorithm that synergizes the strengths of both DPO and SimPO.
Theorem 3.1 establishes a unified framework for DPO and SimPO by showing that replacing the reference model Tet in DPO with uniform distribution U reduces the DPO loss to the SimPO loss, up to a constant term This reveals that SimPO is essentially DPO with a uniform reference model. Consequently, the term 3 (log Firt Vv /z) log Ter (=)) collapses to a constant, reflecting the difference in selection probabilities induced by the scoring mechanism. This em- phasizes the pivotal role of the implicit reference model in preference optimization.
3.1. A Common Framework for DPO and SimPO
key insight in our work is that SimPO implicitly adopts uniform distribution over the vocabulary as its reference model, whereas DPO employs the SFT model the refer- ence. By examining the role of the reference model in both methods, we derive the following result:
Theorem 3.1. Let U(ylæ) denote uniform distribution over the vocabulary for a given input x, replacing To)(y(
in the DPO loss function. Then, the DPO loss function simplifies to:
Limitations of DPO: As depicted in Figure 2, the reference model Ter in DPO may not effectively distinguish between the preferred (yww) and less preferred (yr) responses, as its outputs do not inherently reflect the preference information. In contrast, using a uniform distribution as in SimPO results in reward margin Y, ensuring that the reward difference between the preferred and less preferred responses is entirely governed by the policy model πТ.
[(TT; U)= = -E(..,...r)- [log o (8 (log To (2 (x) log ( (x)) ,),,
where = 3 (log U (z) - log log (s(lz)) is constant. Under a length-normaliied reward formulation, this loss function becomes:
Limitations of SimPO: While SimPO simplifies the loss function by using constant offise 7, this one-size-fits-al approach overlooks the variability inherent different data instances. The fixed 7 across all training samples could lead to suboptimal performance, especially in the presence of noise or inconsistencies in the data. Moreover, completely discarding reference model Tie may climinate potentially useful prior knowledge about language structure and se- mantic relationships that could help discriminate between similar response pairs.
LIN(TOO U) E,x,DD
logor ( B
B lval log To(y))))
) (7)
log To(ywlz)
Therefore, SimPO can be interpreted as a special case of DPO where the reference model a uniform distribution. Remark 3.2. Why do winning and losing responses have different probabilities under a uniform policy? Although both winning and losing responses are sampled from the
3

---

AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization
3.2. Proposed Method: AlphaDPO
obtain the AlphaDPO objective:
Our analysis highlights the significant impact of the refer- ence model in preference optimization. To overcome the limitations identified in both DPO and SimPO, we propose the following principles:
CApphaDoo (To, (
- B(
Logo (Boog To
Aiee (Jhe (z) Tree (311 /20) }
3 log
TO(MIxz)
Principle 1:
The reference model should contribute to dif- ferentiating between preferred and less preferred responses.
-E
log o (8 (11088 To (H)Wllz)
Principle 2:
The reference model should adapt to discrep- ancies between response pairs to capture instance-speciftc nuances.
- 17 + +aM you your w1)
To To(yel (2)
(10)
where y = 3 (log U(pw(a) (x)
is a constant offset as before,
U(m(az))
Principle 1 addresses the shortcoming in DPO, where the reference model may inadequately distinguish between you and 3/, introducing uncertainty without a guaranteed mar- gin. Principle 2 rectifies the oversimplification in SimPO, where the absence of a reference model fails account for variability across different instances.
and M (zz Mos m1) is defined as:
To (y (rer)rree(ii (z)) Ilog
M yw: v1
(11)
(y w ) (31 (z)
The term M (x, yu M1) measures the divergence between the policy model xa and the reference model ref over the response pairs, effectively capturing instance-specific dis- crepancies as described in Principle 2.
Deriving the AIphaDPO Objective. Starting from the stan- dard Reinforcement Learning (RL) objective for preference optimization, we redefine the reference model Tee as an implicit reference model feet formulated as:
Stop Gradient on नeet. Although माeर depends on me and Tet, it is intended serve as a fixed reference during op- timization. To prevent gradients from backpropagating through कee TT. we apply a stop-gradient operation, de- noted as sgl-]. ensuring that ीाe remains constant during the policy updates.
To (u)l)))) [rer(y|z)
Ane (vlæ)) x Utyla:)
where a is a hyperparameter that determines the extent to which the policy model my modifies the reference model Tref, and Uy]lz)) represents a uniform distribution provid- ing stable baseline. This formulation shares structural similarities with the weak-to-strong preference optimiza- tion framework (Zhu et al., 2025), where the aligned strong strong is expressed as:
Normalization of M(x, yee y:): To stabilize training and avoid M(x, ywe 91) dominating the loss due to scale varia- tions, we apply Z-score normalization (Patro & Sahu, 2015) to M:
model n.
Ma Bor 21) (I, ywo Vi PM
(12)
x)) n weak ref
(y
strron (y) T
x))
XX
ref
strong (yl
x))
yy
x)
where UM and aM are the mean and standard deviation of M computed over the training dataset.
Both formulations employ model ratios dynamically ad-
just the reference distribution me. in Equation 8 and Kr
Inspired by SimPO, we incorporate length normalization into method. This adjustment ensures that rewards are scaled appropriately with respect to the length of the sequences, thereby stabilizing the training process. As demonstrated in our experiments (ef. Appendix D.2), we also confirmed that even without length normalization, our method remains effective and continues to show performance improvements.
Length-normalized Reward Formulation:
in Equation 9. Specifically, काeर interpolates between ex- isting methods, controlled by the smoothness parameter oc i) When 0, सte reduces to the uniform distribution U (v(z)), equivalent to the assumption in SimPO. ii) When = 1, Are fully incorporates the proportionality align- ing with DPO.
The motivation behind this redefinition is to overcome the potential suboptimality of both SimPO and DPO by learn- ing instance-specific adjustments while preserving useful prior knowledge. Rather than relying either a fixed ref- erence model or constant margin, माe creates flexible framework that can better capture varying degrees of prefer- ence strength across different instances. Further details are provided in Appendix C.
Final Objective: Incorporating the above considerations, the final AlphaDPO loss function becomes:
CAlphaDPo (Te, )ea)) = -E
(13)
[log a (u(z, Bw y1) - sg 1> + aM" (æ, Bee y())
where u(æ, yoo v1) Twa.l log no (v (ar)
3 log ( (z).
T
Substituting कंतर into the original DPO loss function, we
This formulation ensures balanced influence between the
4

---

AlphaDPO; Adaptive Reward Margin for Direct Preference Optimization
policy and reference models, aligning with Principles and 2. By incorporating the normalized discrepancy term M (zz, V/ 3r), AlphaDPO adaptively adjusts the margin between preferred and less preferred responses based on instance-specific differences, enhancing learning.
where ( yws M1 is the margin term in the AlphaDPO objective.
The proof follows from the sequence-level approximation of the KL divergence between Teet and To along a sequence y. where:
4. Theoretical Analysis of AlphaDPO
Balancing alignment performance with response diversity is crucial in recent alignment methods (Zeng et al., 2024; Wang et al., 2024a; Ji et al., 2024a). A popular approach is the Token-Level Direct Preference Optimization (TDPO) method (Zeng et al., 2024), which introduces -ee-ggai control of the KI. divergence at the token level. Given a prompt z and preceding tokens y² the policy my generates the next token= z by sampling from To (2|az,) +t).
lyl
Ter[2/|z, y
se [x, ys Teerl [ell
E Bewata
log
To (2|2,,,3)) yct)
log
To (y)
Applying this approximation to both you and yt, the dif- ference 8(x,, Bw. w) simplifies to the difference of log- probability ratios, thereby establishing the equivalence with the margin term in AlphaDPO.
By mapping the reward model to a token-level format, the TDPO loss is defined as:
(Te) (x. MN)-
Lemma 4.1 highlights that the margin term 8(z, Yau Ut). which represents the sequential KL. divergence difference between preferred and rejected responses, can be directly mapped to the term (x, yw y1) in AlphaDPO. This map- ping underscores the theoretical connection between the two approaches in terms of alignment control. While TDPO operates at the token level and provides fine-grained control, AlphaDPO offers greater computational efficiency by oper- ating at the sequence level without sacrificing performance (cf. Appendix Table 6). Moreover, the sequence-level ap- proximation enhances robustness to token-level noise in Tref making AlphaDPO particularly suited for scenarios where the reference policy may not be perfectly aligned.
log a (Blog No [x)
Thet (Vwlxx) - Blog ( (x) (ee( (x)
(ar, Mow mt)
(I4)
where the margin term (2,, yhes w1) is defined as:
8(x, Vwa, vi)
[SiDssuk (zr, yi mo] (T [st thoi Tedl [oo],
(15)
lxd B. [log
(2 [sr, 21 21 <=<2|< [Ir, "1)
Tue Tnr(= [ vo v(d])
E. flog
To [=| [ce vm
(16)
Here, 2 Tel and DsecKL [ar, yi Tee [ denotes the se- quential KL divergence between oree and Te along the se- quence y given x.
A Advantages of the margin term 8(2,, yw, u1). The core contribution of TDPO lies in introducing the margin term r(z, yw) - r 31) 5(2, twes 31), which is similar to DPO with an offset (Amini et al., 2024) and helps control the KL divergence. In contrast, AlphaDPO generalizes this ap- proach by replacing 5(x, yo: 31) with M (ax, yuu 3)), a more flexible margin term inspired by SimPO. Appendix Table 6 supports this with performance comparisons between TDPO and AIphaDPO. These findings illustrate: i) Adding an off- set to DPO and its variants is robust strategy, applicable to both TDPO and AlphaDPO. ii) The choice of offsee is critical - the sequence-level margin term (xx, Vie 91) is particularly effective when applied to potentially unreliable reference models.
Below we present a lemma establishing theoretical connec- tions between TDPO and AlphaDPO w.r.f. the margin terms.
Lemma 4.1 (Equivalence of Margin Terms). Consider the margin term 6(1, you Mt) that represents the difference in sequential KL divergences between a reference policy Tref and policy By for preferred sequence yo and rejected se- quence yu conditioned on input
8(x, ywu, y1) SSDDss [x., yi Teyl [T]
(17)
Sequ [x, ywi Tred [To].
Under the assumption that the sequential KL divergence can be approximated by the log- likelihood ratio of complete se- quences, the margin term 8(x, yw M1) admits the following approximation:
5. Experiments
In this section, we present the main results of our experi- ments, highlighting the superior performance of AlphaDPO over existing methods on various benchmarks and ablation studies to analyze the impact of different components of AlphaDPO.
To To(yulo) log Try ( (x)
Te (1 (x)
5(x, Ywu v1) RM 3 log
Tre((ullz)
= (M, vho v1),
5

---

AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization
Table AlpacaEval Arena-Hard results across four settings. "WR" denotes the raw win rate, "LC" conttlecon win rate, and "SC" the ctyte-controlled win rate. The best results are highlighted in bold, while the second- best are underlined.
Llama3-Instruet (8B)
Mistral-Instruct (7B)
Method
AlpacaEval 2
Arena-Hard
AlpacaEval 2
Arena-Hard
LC (%)
WR (%)
SC (%)
LC (%)
WR (%)
LC (%)
WR (%)
SC (%)
LC (%)
WR (%)
SFT
24.0
23.6
22.1
22.2
22.4
19.0
15.4
18.3
13.2
12.9
DPO
40.2
38.1
31,9
32.0
31.2
20.3
17.9
18.9
13.7
13.4
IPO
35.9
34,4
29.2
29.9
30.2
22.3
18.6
22.4
16.6
16.2
CPO
29.6
34.4
26.3
28.1
29.4
26.2
31.7
26.6
21.4
23.8
KTO
38.3
34.1
30.3
30.6
30.3
19.4
20.3
21.5
16.0
16.8
ORPO
31.6
29.8
26.6
26.6
26.3
24.0
23.0
24.4
18.5
18.6
R-DPO
40.3
37.3
33.1
32.9
32,9
21.4
22.2
18.7
14.0
SimPO
43.8
38.0
33.5
33.5
32.6
30.2
32.1
25.6
19.8
20.1
AlphaDPO
46.6
38.1
34.1
34.2
33.3
32.3
32.6
27.2
21.5
21.5
3l Instruct v0.2 (8B)
Gemma2-Instree (9B)
Method
AlpacaEval
Arena-Hard
AlpacaEval 2
Arena-Hard
LC (%)
WR (%)
SC (%)
LC (%)
WR (%)
(C)
WR (%)
SC (%)
LC (%)
WR (%)
SFT
24.0
23.6
22.1
22.2
22.4
48.7
36.5
32.0
42.2
42.1
DPO
51.9
50.8
26.1
31.5
33.9
70.4
66.9
43.9
55.6
58.8
IPO
40.6
39.6
31.1
34.2
34.9
62.6
58.4
41.1
51.9
53.5
CPO
36.5
40.8
29.4
32.8
34.2
56.4
53.4
42.4
53.3
55.2
KTO
41.4
36.4
27.1
29.5
28.9
61.7
55.5
41.7
52.3
53.8
ORPO
36.5
33.1
28.8
30.8
30.4
56.2
46.7
35.1
45.3
46.2
R-DPO
51.6
50.7
29.2
34.3
35.0
68.3
66.9
45.1
55.9
57.9
SimPO
55.6
49.6
28.5
34.0
33.6
72.4
65.0
45.0
56.1
57.8
AlphaDPO
58.7
51.1
30.8
36.3
35.7
73.4
66.1
48.6
59.3
60.8
5.1. Experiments Setup
as the reward model for ranking generated data, significantly enhancing performance. These configurations represent state-of-the-e of-the-art methods, positioning our models among the top performers on various leaderboards.
Models and training seftings. We optimize preferences using three model families: Llama3-8B (AI@ Meta, 2024), Mistral2-7B (Jiang et al., 2023), and Gemma2-9B (Rivière et al., 2024), all in the Instruct setup. Following Meng et al. (2024), we utilize pre-trained instruction- tuned models Lineta-lamaaMMtt-Lllnna -aaiett----aam-------nn-t mistralai/Miittra--BBBB Inn.-B---nttruc--0.2222 google/gemma-2-99b. as SFT models. For fair comparison, we use the same training data as SimPO: princclommipplllaaaa-uuttteeeedd armorm', and prineton-lyggemm22uuuddedededddd Llama3-8B, Mistral2-7B, and Gemma2-9B, respec- tively. Additionally, the v0.2 Llama3-Instruc setup uses RLHFFlowAAmoRMMMLammss-- (Wang al., 2024b)
We evaluate our models using two widely recognized open-ended eestructionfollowiiii benchmarks: AlpacaEval 2 (Li et al., 2023) and Arena- Hard (Li et al., 2024). These benchmarks assess the models' conversational abilities across diverse range of queries and are extensively used by the research community. For AlpacaEval we report the length-controln and raw win rate (WR). For Arena-Hard, win rate (LC) provide the win rate (WR), length-controlled win rate (LC), and style- controlled win rate (SC) compared to baseline models. Note that style significantly impacts performance on these leader- boards.
Evaluation benchmarks.
for
http:://wwggniiieeeewwwaaassccca ultrafeedlhack-armomm
Baselines. We compare AlphaDPO with several state-of- the-art preference optimization methods: DPO (Rafailov et al., 2023), SimPO (Meng al., 2024), IPO (Azar et al., 2024), CPO (Xu al., 2024), KTO (Ethayarajh al., 2024). ORPO (Hong et al., 2024), and R-DPO (Park et al., 2024).
instruttullraffeedhaack altafeeelback-armomnn
6

---

AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization
Table Ablation studies under Llama3-Instrue v0.2 and Mistral-Instrus settings. ablate each key design AlphaDPO and
explore variants of the implicit reference model Aee-
Llama3-Instruct v0.2 (8B)
Mistral- (7B)
Method
AvpacaEval
Arena-Hard
AlpacaEval
Arena-Hard
LC (%)
WR (%)
SC (%)
LC (%)
WR (%)
LC (%)
WR (%)
SC (%)
LC (%)
WR (%)
U(-ee)
55.6
49.6
28.5
34.0
33.6
30.2
32.1
25.6
19.8
20.1
Ut-kc) (x)(
58.7
51.1
30.8
36,3
35.7
32.3
32.6
27.2
21.5
21.5
w/o Normalization
56.5
23.1
28.4
27.7
32.1
33.1
25.2
19.7
19.6
w/o
2.7
3.7
7.7
5.4
6.3
27.2
27.7
25.8
20.3
20.7
7=0
51.2
44.9
30.0
34.5
33.3
31.9
31.3
24.2
19.6
19.3
U(-[z) (=x))
57.2
50.4
27.6
33.5
32.9
31.6
34.1
26.9
21.3
21.5
Ut-lc) (1=))
56.3
49.5
29.0
34.3
33.5
28.6
30,9
25.5
20.1
20.3
U-)cc) (1/ (1/))x(-(e))
56.3
49.2
29.0
34,4
33.8
32,2
33.1
26.0
20.7
20.6
We also include the SFT model as a baseline. We thor- oughly tune the hyperparameters for each baseline and re- port the best performance. Further details can be found in Appendix D.I.
U(ylz) (no(y)=))) (2) ))r(()))) = U(ylz) (=))( and (3) * = Uty[z) (1/mer(yl=))). As shown in Table 2, most the variants perform better than SimPO (fer(ylz) (z)[z)), which demonstrates the importance of adaptive margin between pairs. Besides, proposed reference model consistently outperforms other variants, indicating the effectiveness of the proposed design.
5.2. Main Results
AlphaDPO consistently outperforms existing preference optimization methods. As shown in Table I, while all preference optimization algorithms improve over the SFT baseline, AlphaDPO achieves superior performance com- pared to existing methods specifically on the AlpacaEval 2 LC metric. These significant improvements highlight the robustness and effectiveness AlphaDPO. Specifically, Al- phaDPO outperforms the best baseline by an average of percentage points in AlpacaEval 2 LC win rate. Fur- thermore, benchmarks such Arena-Harde AlphaDPO achieves wfateoofthh-aae or second-best results, demonstrat- ing its competitiveness across different evaluation settings.
All key designs in AlphaDPO are crucial. To further an- alyze the impact of different components in AlphaDPO, we conduct ablation studies by removing key components from AlphaDPO. As shown in Table 2, removing normal- ization, stop gradient, setting 0 all lead to significant performance drops, highlighting the importance of these components in AlphaDPO.
5.3. KL divergence control in AlphaDPO
Outstanding Performance and Lower KI. As noted in Rafailov et al. (2023); Zeng et al. (2024), it is crucial to consider both performance and KL divergence when com- paring algorithms. slightly higher win rate accompanied by a significantly higher KI. divergence is often not de- sirable. In line with the design principles of TDPO, we implemented SimPO and AlphaDPO. Figure 3a 3b presents the KI. divergence curves. The results indicate that as increases, the KL divergence of AlphaDPO remains stable or even decreases slightly when compared to SimPO. This demonstrates that AlphaDPO not only achieves superior performance but also maintains a lower KI divergence, in- dicating better balance between alignment and control of KI. divergence during the training process.
Impact of Metrics on Leaderboard Rankings. While both benchmarks are widely used, the standard win rate (WR) metric shows poor separability among different methods, making challenging to distinguish their relative perfor- mance. Minor differences in WR may stem from biases towards generating detailed or aesthetically pleasing re- sponses, aligning with observations by Dubois et al. (2024) and Chen et al. (2024a). In contrast, the length-controlled (LC) and style-controlled (SC) win rates offer more reliable and interpretable metrics, as they reduce the influence of verbosity and stylistic biases, thereby better reflecting true performance.
The importance of the design on the implicit refer- ence model. As the core contribution of this work is to propose a novel reference model * = U (y(xx (TT(s)=)//T<((()))) ants of the reference model. Specifically, we com- pare AIphaDPO with three variants: (1) #
Mitigating Over-OOtinnzttion. Over-optimization as de- scribed by Gao et al. (2023) and Rafailov et al. (2024), refers to a phenomenon where model performance exhibits hump-shaped pattern across different targets: beyond an optimal point, further increasing the KL budget results in diminishing returns. To investigate this, evaluate
we also evaluate other vari-
7

---

robust solution for AlphaDPO; Adaptive Reward Margin for Direct Preference Optimization
SimPa Ourslam0.3)
3.0
SlmPO
225
Ours(a=
Lengtth Centrol Win
divergence
2.0
40-
1.5
2.0
0.5
SimPO
App
0.0
200
100
200
300
400
Stop
Step
Square Root of KL divergence
(a) KL divergence of you over steps.
(b) KL divergence of yu over steps.
(c) and KL divergence.
Figure 3. Analysis of KL divergence and LC trade-off. (a) KL divergence for chosen samples (yw), (b) KL divergence for rejected samples
(s0) and (c) relationship between LC and KL divergence.
59
DpphaDPo(a-0.0)
0.6-
AlphaDPo(g-O.0 0.9)
SC on Arene-Hoo Hard
Dp
MpphaPP0(=00.11
AlphaDPo(g-0.3)
AlphaDPO(a-0.3)
Density
Donatyy
0.4
NiphaPP0a=0055)
57-
29
on
0.2
55-
0.01
0.05 0.1
0.2
10
10
20
30
0.0
2
Reward dift. M...Ywl - rix.
Avg. log prob on winning Pulxw/))
Figure Impact a on (a) LC and SC win rate, (b) reward difference distribution, and (c) log- likeliheoood distribution of chosen responses in AlphaDPO.
SimPO and AlphaDPO at four intermediate checkpoints, corresponding to different KL budgets. As illustrated in Figure 3c, it is intriguing that while the performance of our approach does decrease with increasing KL budget, the de- cline is relatively modest. This indicates that our method effectively mitigates the issue of over-optimization.
reward margin and log likelihood.
6. Discussion
Conclusion. We proposed AlphaDPO, an adaptive prefer- ence optimization method that improves LLM alignment by introducing a dynamic reward margin based on instance- specific differences. AlphaDPO addresses limitations in previous methods like DPO and SimPO by balancing align- ment and diversity through KL divergence control. Our theoretical guarantees and empirical results show that Al- phaDPO consistently outperforms baselines on benchmarks like AlpacaEval 2 and Krena-Hard, with significant improve- ments in win rates, establishing LLM fine-tuning.
5.4. The Impact of a in AIphaDPO
Effect of a on Performance. We investigated how the pa- rameter a in AIphaDPO impacts the win rate on AlpacaEval 2 and Arena-Hard. The results, as shown in Figure (a), in- dicate that the style-control win rate on Arena-Hard initially increases and then decreases with increasing In contrast, the length-control win rate on A AlpacaEval 2 exhibits con- sistently increasing trend. This suggests that the optimal value of a varies depending on the evaluation benchmarks. Further experiments refer to Appendix D.3.
Limitations and Future Work. While AlphaDPO en- hances performance, it introduces additional hyperpa- rameter, a, requiring manual tuning. Future work could focus on developing an adaptive approach to automatically adjust this parameter. Additionally, although we show Al- phaDPO's theoretical equivalence to online methods, it re- mains an offline approach. Extending to online learning would allow real-time adaptation, broadening its application in interactive environments. Lastly, we observed that dif- ferent benchmarks, such as AlpacaEval 2 and Aard, require distinct parameter settings for optimal performance. Investigating a more generalized approach that adapts effec- tively across multiple benchmarks would further improve the model's versatility.
Impact of on the reward distribution. We visualize the distribution of the learned reward margin r(a, yw) - r(r, y) and the log likelihood of the chosen response log (M under different a values in Figure 4 (b,c). Decreasing a results in a flatter reward margin, while the log likelihood distribution remains relatively unchanged. Conversely, in SimPO (ef. Figure 6), increasing 7 yields a flatter reward margin distribution but at the cost of also flattening the log likelihood distribution, which undesirably lowers the log likelihood of positive samples. This indicates that Al- phaDPO can better balance the relationship between the
8

---

AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization
Impact Statement
plete block designs: I. the method of paired comparisons. Biometrika, 1952.
This work introduces AIphaDPO, an adaptive preference op- timization framework that improves Large Language Model (LLM) alignment with human preferences. By establishing more principled and effective mechanism for preference optimization, AIphaDPO contributes to developing more robustly aligned LLMs, a crucial step towards safer and more beneficial Al systems.
Chen, G. H., Chen, S., Liu, Z., Jiang, F,, and Wang, B. Humans or Ilms as the judge? A study on judgement biases. Arxiv, abs/2402. 10669, 2024a.
Chen, L., Zhu, C., Soselia, D., Chen, J., Zhou, T., Goldstein, T., Huang, H., Shoeybi, M., and Catanzaro, B. ODIN: Disentangled reward mitigates hacking in RLHF. In ICML, 2024b.
Acknowledgment
Christiano, P. F., Leike, J., Brown, T. B., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. In NIPS, pp. 4299-4307, 2017.
This research is supported by the National Science and Technology Major Project (2023ZD0121 and the Na- tional Natural Science Foundation of China (62302321, U24B201800 80). This research was also supported by the ad- vanced computing resources provided by the Supercomput- ing Center of the USTC.
Conover, M., Hayes, M., Mathur, A., Xie, J., Wan, J., Shah, S., Ghodsi, A., Wendell, P., Zaharia, M., and Xin, R. Free dolly: Introducing the world's first truly open instruction-tuned LLM, 2023. URL https: / //www www.cattariccs..oomo ctt/:/.ccccooooooeeeeeco http://bii.kscccooooo///////////
References
ddolly-first open www.ercicomerrreeeeeeee wuweennn--nndddllnn
AItMeta. Llama 3 model card. 2024. https://github. http///ttaaaammaaaaaaeaaaaaaaaaa blob/maing BMO.i//MDD_CCCCC md.
URL
Ding, N., Chen, Y., Xu, B., Qin, Y., Zheng, Z., Hu, S., Liu, Z., Sun, M., and Zhou, B. Enhancing chat language mod- els by scaling high-quality instructional conversations. In EMNLP, 2023.
Amini, A., Vieira, T. and Cotterell, R. Direct pref- erence optimization with offset. arXiv preprint arXiv:2402.10571, 2024.
Dong, H., Xiong, W., Goyal, D., Zhang, Y., Chow, W., Pan, R., Diao, S., Zhang, J., Shum, K., and Zhang, T. RAFT: reward ranked finetuning for generative foundation model alignment. Trans. Mach. Learn. Res., 2023.
Anthony, T., Tian, Z., and Barber, D. Thinking fast and slow with deep learning and tree search. NeurIPS, 2017.
Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., Das- Sarma, N., Elhage, N., Madieedd-oods, Z., Hernandez, D., Kernion, J., Ndousse, K., Olsson, C., Amodei, D., Brown, T. B., Clark, J., McCandlish, S., Olah, C., and Kaplan, J. general language assistant as a laboratory for alignment. CORR, abs/21 2.00061, 2021.
Dubois, Y., Galambosi, B., Liang, P., and Hashimoto, T. Length-controlled alpacaeval: A simple way to debias automatic evaluators. In COLM, 2024.
Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kiela, D. KTO: Model alignment as prospect theoretic optimization. ICML, 2024.
Azar, M. G., Rowland, M., Piot, B., Guo, D., Calandriello, D., Valko, M., and Munos, R. A general theoretical paradigm to understand learning from human preferences. In AISTATS, 2024.
Gao, L., Schulman, J., and Hilton, J. Scaling laws for reward model overoptimization. In ICML, 2023.
Geng, X., Gudibande, A., Liu, H., Wallace, E., Abbeel, P., Levine, S., and Song, D. Koala: A dialogue model for academic research. Blog post, April, 1:6, 2023.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., Joseph, N., Kadavath, S., Kernion, J., Conerly, T.,
Gorbatovski, A., Shaposhnikov, B., Malakhov, A., Sur- nachev, N., Aksenov, Y., Maksimov, I., Balagansky, N., and Gavrilov, D. Learn your reference model for real good alignment. ICLR, 2025.
Showk, S. E., Elhage, N., Hatfield-Dodds, Z., Hernandez, D., Hume, T., Johnston, S., Kravec, S., Lovitt, L., Nanda, N., Olsson, C., Amodei, D., Brown, T. B., Clark, J., Mc- Candlish, S., Olah, C., Mann, B., and Kaplan, J. Training a helpful and harmless assistant with reinforcement leam- ing from human feedback. ArXiv, abx/2204.05866 2022.
Guo, S., Zhang, B., Liu, T., Liu, T., Khalman, M., Llinares, F., Ramé, A., Mesnard, T., Zhao, Y., Piot, B., Ferret, J., and Blondel, M. Direct language model alignment from online AI feedback. Arxiv, abs/2402.044997 2024.
Bradley, R. A. and Terry, M. E. Rank analysis of incom-

---

AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization
Havrilla, A., Raparthy, S., Nalmpantis, C., Dwivedi-Yoe Yu, J., Zhuravinskyi, M., Hambro, E.,, and Railneau, R. GLoRe: When, where, and how to improve LLM reasoning via global and local refinements. In ICML, 2024.
Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang, P., and Hashimoto, T. B. AlpacaE- val: An automatic evaluator of instraction-folloiing models. https: wtw.ggt..to//attteeeeeeeeeeoeeee wttp:/cttttttttttaaaoe a paca_eval, 2023.
He, L., Xia, M., and Henderson, P. What's in your" safe" data?: Identifying benign data that breaks safety. COLM, 2024.
Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I,, and Cobbe, K. Let's verify step by step. In ICLR, 2024.
Hong, J., Lee. N., and Thome, J. ORPO: Monolithic prefer- ence optimization without reference model. In EMNLP, 2024.
Liu, T., Qin, Z., Wu, J.. Shen, J., Khalman, M., Joshi, R., Zhao, Y., Saleh, M., Baumgartner, S., Liu, J., Liu, P. J., and Wang, X. Lipo: Listwise preference optimization through learning-to-ran to-nkkk. In NAACL, 2025a.
Ji, H., Lu, C., Niu, Y., Ke, P., Wang, H., Zhu, J., Tang, J., and Huang, M. Towards efficient exact optimization of language model alignment. In ICML, 2024a.
Liu, Y., Liu, P., and Cohan, A. Understanding reference policies in direct preference optimization. In NAACL, 2025b.
Ji, K., He, J., and Gu, Q. Reinforcement learning from hu- man feedback with active queries. CoRR, abs/2402.09401, 2024b.
Luo, H., Sun, Q., Xu, C., Zhao, P., Lou, J., Tao, C., Geng, X., Lin, Q., Chen, S., and Zhang, D. Wizardmath: Empow- ering mathematical reasoning for large language models via reinforced evol-instruct. In ICLR, 2025.
Jiang, A. Q. Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de Las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.- A., Stock, P., Scao, T. L., Lavril, T., Wang. T., Lacroix, T., and Sayed, W. E. Mistral 7B. ArXiv, 2023.
Meng, Y., Xia, M., and Chen, D. Simpo: Simple preference optimization with a reference-free reward. In NeurIPS, 2024.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980) 2014.
Morimura, T., Sakamoto, M., Jinnai, Y., Abe, K., and Ariu, K. Filtered direct preference optimization. In EMNLP, 2024.
Köpf, A., Kilcher, Y., von Rütte, D., Anagnostidis, S., Tam, Z. R., Stevens, K., Barhoum, A., Nguyen, D. M., Stan- ley, O., Nagyfi, R., et al. Openassistant conversations- democratizing large language model alignment. In Thirty- seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, F., Leike, J., and Lowe, Training language models to follow instructions with human feedback. In NeurIPS, 2022.
Lambert, N., Pyatkin, V., Morrison, D., Miranda, L. J. V., Lin, B. Y., Chandu, K. R., Dziri, N., Kumar, S., Zick, T., Choi, Y., Smith, N. A., and Hajishirzi, H. RewardBench: Evaluating reward models for language modeling. In NAACI, 2025.
Park, R., Rafailov, R., Ermon, S., and Finn, C. Disentangling length from quality in direct preference optimization. In ACL., 2024.
Langley, P. Crafting papers on machine learning. In ICML, 2000.
Patro, S. G. K. and Sahu, K. K. Normalization: A prepro- cessing stage. Arxiv, abs/1503.06462, 2015.
Leike, J., Krueger, D.. Everitt, T., Martic, M., Maini, V., and Legg, S. Scalable agent alignment via reward modeling: research direction. CoRR, abs/18 0t5/1811.07887 1.07871, 2018.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. In NeurIPS, 2023.
Li, T., Chiang, W.-L., Frick, E., Dunlap, L., Zhu, B., Gonzalez, J. E., and Stoica, I. From live data to high-quality benchmarks: The Arena-Hard pipeline, April 2024. URL sctps://1mstt www.mss.og//llloo.oooo org/blog/ 2024 0444004111 aaaarrnaaaarr..
Rafailov, R., Chittepu, Y., Park, R., Sikchi, H., Hejna, J., Knox, W. B., Finn, C., and Niekum, S. Scaling laws for reward model overoptimization in direct alignment algorithms. In NeurIPS, 2024.
10

---

AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization
Rivière, M., Pathak, S., Sessa, G., Hardin, C., Bhupati- raju, S., Hussenot, I.., Mesnard, T., Shahriari, B., Ramé, A., Ferret, J., Liu, P., Tafti, P., Friesen, A., Casbon, M., Ramos, S., Kumar, R., Lan, C. L... Jerome, S., Tsitsulin, A., Vieillard, N., Stanczyk, P., Girgin, S., Momchev, N., Hoffman, M., Thakoor, S., Grill, J., Neyshabur, B., Bachem, O., Walton, A., Severyn, A., Parrish, A., Ahmad, A., Hutchison, A., Abdagic, A., Carl, A., Shen, A.. Brock, A., Coenen, A., Laforge, A., Paterson, A., Bastian, B., Piot, B., Wu, B., Royal, B., Chen, C., Kumar, C., Perry, C., Welty, C.,, Choquette-Chone
Xiong, W., Dong, H., Ye, C., Zhong, H., Jiang, N., and Zhang, T. Gibbs sampling from human feedback: A provable KL- constrained framework for RLHF, CoRR, abs/2312.1 1456, 2023.
Xu, H., Sharaf, A., Chen, Y., Tan, W., Shen, L., Durme, B. V., Murray, K., and Kim, Y.J. Contrastive preference optimization: Pushing the boundaries of LLM perfor- mance in machine translation. In ICML, 2024.
Yang. S., Cui, L., Cai, D., Huang, X., Shi, S., and Lam, W. Not all preference pairs are created equal: A recipe for inpotation-efficiert iterative preference learning. In ACL, 2024.
C. A., Sinopalnikov. D., Weinberger, D., Vijaykumar, D., Rogozinska, D., Her- bison, D., Bandy, E., Wang. E., Noland, E., Moreira, E., Senter, E., Eltyshev, E., Visin, F., Rasskin, G., Wei, G. Cameron, G., Martins, G., Hashemi, H.. Klimezak- Plucinska, H., Batra, H., Dhand, H., Nardini, I., Mein, J.. Zhou, J., Svensson, J., Stanway, J., Chan, J., Zhou, J. P., Carrasqueira, J., Iljazi, J., Becker, J., Fernandez, J., van Amersfoort, J., Gordon, J., Lipschultz, J., Newlan, J., Ji, J., Mohamed, K., Badola, K., Black, K., Millican, K., McDonell, K., Nguyen, K., Sodhia, K., Greene, K., Sjösund, L. L., Usui, L., Sifre, L.,, Heuermann, L.,, Lago, L., and McNealus, L. Gemma 2: Improving open lan- guage models a practical size. CoRR, abs/2408.001 2024.
Ye, C., Xiong, W., Zhang, Y., Jiang, N., and Zhang, T. A theoretical analysis of nash learning from human feed- back under general kl-regularized preference. CoRR, abs/2402.07314, 2024.
Yuan, H., Yuan, Z., Tan, C., Wang, W., Huang, S., and Huang, F. RRHF: rank responses align language mod- els with human feedback. In NeurlPS, 2023.
Yuan, W., Pang, R. Y., Cho, K., Li, X., Sukhbaatar, S., Xu, J., and Weston, J. self-rewarding language models. In ICML, 2024.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. 1007::777.0333 2017.
Zeng, Y., Liu, G., Ma, W., Yang, N., Zhang, H., and Wang, J. laken-level direct preference optimization. In ICML, 2024.
arXiv preprint
Song, F. Yu, B., Li, M., Yu, H., Huang, F,, Li, Y., and Wang, H. Preference ranking optimization for human alignment. In AAAI, 2024.
Zhang, S., Yu, D., Sharma, H., Yang, Z., Wang, S., Hassan, H., and Wang, Z. Self-exploring language models: Active preference elicitation for online alignment. TMLR, 2025.
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, F. Learning to summarize with human feedback. In NeurlPS, 2020.
Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L.,, et al. LIMA: Less is more for alignment. NeurIPS, 2023.
Swamy, G., Dann, C., Kidambi, R., Wu, S., and Agarwal, A. A minimaximalist approach to reinforcement learning from human feedback. In ICML, 2024.
Zhu, W., He, Z., Wang, X., Liu, P., and Wang, R. Weak- to-strong preference optimization: Stealing reward from weak aligned model. In ICLR, 2025.
Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford alpaca: An instruction-followingg Ilama model, 2023.
Ziegler, D. M.. Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P. F,, and Irving, G. Fine- tuning language models from human preferences. ArXiv, abs/1909 1es/1999.0859,, 2019.
Wang, C., Jiang, Y., Yang, C., Liu, H., and Chen, Y. Beyond reverse KL: generalizing direct preference optimization with diverse divergence constraints. In ICLR, 2024a.
Wang, H., Xiong, W. Xie, T., Zhao, H., and Zhang, T. Inter- pretable preferences via multi-objective reward modeling and
EMNLP, 2024b.
Wu, J., Xie, Y., Yang, Z., Wu, J., Gao, J., Ding, B., Wang, X., and He, X. 8-dpo: Direct preference optimization with dynamic B3. In NeurlPS, 2024.

---

AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization
A. Related Work
Reinforcement learning from human feedback. RLHF a technique that aligns large language models with human preferences and values (Christiano et al., 2017: Ziegler et al., 2019; Ouyang et al., 2022; Azar al., 2024). Traditional RLHF can be divided into three stages: supervised fine-tuning (Zhou al., 2023; Taori et al., 2023; Geng al., 2023; Conover al., 2023; Köpf al., 2023; Ding et al., 2023), reward modeling (Gao et al., 2023; Luo et al., 2025; Chen et al., 2024b; Lightman et al., 2024; Havrilla et al., 2024; Lambert et al., 2025), and policy optimization (Schulman et al., 2017; Anthony et al., 2017). In the third stage, Proximal Policy Optimization (PPO) is a widely used algorithm. Additionally, Xiong et al. (2023) proposed efficient algorithms for the reverse-KL regularized contextual bandit framework in RLHE. Ye et al. (2024) introduced provably efficient algorithms for KL-regularized Nash-Learning from Human Feedback (NLHF). Furthermore, Ji al. (2024b) developed an seive-queey-based PPO algorithm with specific regret bounds and query complexity.
Offiine direct preference optimization. Several alternative preference optimization objectives have been proposed in addition DPO (Rafailov et al., 2023). IPO (Azar et al., 2024) addresses the overfitting issues associated with DPO. ORPO (Hong et al., 2024) and SimPO (Meng et al., 2024) aim to eliminate the dependence on a reference model. R- DPO (Park et al., 2024) focuses on mitigating exploitation based on sequence length. KTO (Ethayarajh et al., 2024) deals with preference optimization when data are not pairwise. CPO (Xu et al., 2024) and 3-DPO(Wu et al., 2024) emphasize the quality of preference data. Another line of research explores comparisons among more than two instances (Dong et al., 2023; Liu et al., 2025a; Song et al., 2024; Yuan et al., 2023).
Online direct preference optimization. Offline direct preference optimization methods are simple but rely on preference data collected offline. RLHF methods interact online with the language model being aligned but require policy gradients. In contrast, online direct preference optimization methods combine the advantages of both approaches. Yuan et al. (2024) proposed a "self-rewarding" approach in which the policy being aligned provides online feedback to itself. Alternatively, OAIF (Guo et al., 2024) is a novel online preference optimization method that can leverage feedback from any L.M,, including those stronger than the LLM being aligned. Swamy et al. (2024) also concurrently investigate the importance of online preference but still rely on reward models (RMs). SELMA (Zhang al., 2025) improves exploration efficiency by selectively favoring responses with high potential rewards rather than indiscriminately sampling unseen responses.
B. Proofs
B.1. Proof of Theorem 3.1
Theorem 3.1. Let U (ylz)) (z) denote a uniform distribution over the vocabulary for a given impul x, replacing Tmdyyl in the DPO loss function. Then, the DPO loss loss function simplifies to:
[( U) = -E(
(6)
llgg (3 (log (yw (r) - log (e())))) - -]]],
where 7 = 3 ()wl())))))) - log U (u)] is a constant. Under length-normalized reward formulation, this loss function becomes:
CIN(TO: U) = -EE(x.se (D)
log o ( B
3
log To (ullz) - ->]
log (B
[ywl
lyal
Therefore, SimPO can be interpreted as a special case of DPO where the reference model is uniform distribution.
Proof. Let Uly (r) denote uniform distribution over the vocabulary V for given input Specifically, for any sequence y. the uniform distribution is defined as:
1vl
lel
1
U(yl=)
Consider the DPO loss function:
ne (vw (x) no (u(()
Toor(Bull /xx) Trer( x:) )
LTPO(TTO Teet) = E W
log Blog
B log
12

---

AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization
By substinuting "ed U, the term involving the reference policy simplifies
log Teer (x)) (3 (log (v)) log (x))):)) = =y,
(1)())
where y constant. This constancy arises because Yoe and Yr are chosen from distinct subsets of the vocabulary, ensuring that log (ar log U (yn (e)) does not depend the lengths of the sequences but is instead determined by the fixed probabilities of the respective subsets. Consequently, y remains fixed across all samples in D.
Substituting back into the DPO loss function, we obtain:
( U) (r.y [log (B (log To (8 (z) log No T (z)) - -)]]
Under the length- normalized reward formulation, the rewards are adjusted by the lengths of the sequences yo and yr- This normalization yields:
(LN(TT: U) = (a. an -D Tog
B (1btal log <o(vvl
B
log To
(x) (o(
-
lorl
Here, remains a fixed constant since is derived from the uniform distribution over distinct vocabulary subsets corre- sponding to Yo and yt-
Comparing this with the SimPO loss function:
3 [Mwll
B 1yd
CsimPO(Te) = (a. W.D
log o
log no (Vwll))
log No (ullz) -7)
is evident that:
(on(Toi U) = (TD)
Thus, when the reference policy Ter is uniform distribution over distinct vocabulary subsets for tue and yt, the DPO loss function simplifies to the SimPO loss function with y being a fixed constant. This establishes that SimPO is special case of DPO under the specified conditions.
B.2. Proof of Lemma 4.1
Lemma B.1 (Equivalence of Margin Terms), Consider the murgin term 8(2, Yor w1) that represents the difference in sequential KL divergences between a reference policy Tref and policy for preferred sequence and rejected sequence y,, conditioned on input x:
8(x, Dw 31) B [x, M1 Teee
(17)
5DD [2x Ywi
Under the assumption that the sequential KL divergence can be approximated by the log- log-likelihodd ratio of complete sequences, the margin term 8(x, yos yt) admits the following approximation:
To TV(ul) log
Je (M xx)
8x, ywe 31) Re B log
Try(yan (x)
Thy T
= M(x Vwe, Vi).
where M (I, ywo W) is the margin term in the AlphaDPO objective.
Proof of Lemma 4.1. We begin by expanding the definition of 5(x, Uw v1):
81, Vie vi) BlD [x, yi reet [e] BDDs [x, Mwi [T]
Expanding each sequential KI. divergence, we have:
13

---

AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization
lol
Trer(z [ct, (1)
Dseqgi, y) Tret [
=
Egw Toat
E
log
(2 22 y y=+()
2m
Substituting this into the expression for 8, we obtain:
yoo
Frel 2
x 1/1
Tret(2 [2,, yer
8(x, Uw V1)
=
E
Bewann
log
log
TT(=
J. V1
Ta(2
[20, yo
t=1
Under the assumption that the reference policy Tee has large enrors, we approximate the expectation Bewnyt with uniform distribution. This approximation simplifies each expectation term as follows:
Trer(= [ar, y (=9 To(2 [r, y5t)
To x))
log
log
Ter
Applying this approximation to both sequential KL divergence terms, we obtain:
8(x, Vwu 31) 2 B (log (u(y x))
log (M 23)
[ee x)) (ew x))
This expression can be rewritten as:
To (y/w x²)
T0 x²)
8(x Bwr v1)
log
log
= M (I, yw 31)
Teer yw
Traf (31 x))
where (x, Ywo W) is the margin term defined in the AIphaDPO objective. Thus, have shown that:
8(x, Uw 3n) Re M (xt, Burs 31)
This completes the proof.
C. The motivation for the proposed frer(y)(x))
The motivation for the proposed reference policy ) can be clarified as follows:
Utility Theory Perspective:
The proposed ( is designed with the uniform distribution U(yaæ) baseline. dynamically adjusts the reward margin by balancing contributions from the policy and reference
The term (s(aa) [
models. This mechanism can be interpreted through the lens of utility theory as relative attractiveness, enabling adaptive instance-specific reward modeling.
Gradient Perspective By introducing (z/r), the framework mitigates the label flipping issues found in DPO or SimPO. the SimPO framework, the gradient is expressed as:
1
1 Vo log To (ullz) )
V VoLsimPo(Ta) -STE (m.yym. so
Vo log (V(Vvl
[8wl
v
where Sq =a
[
log To (o(y) (z) Iwl
log To (x)) + +7).
This formulation may amplify weights when the reward estimate is incorrect. By contrast, under AlphaDPO:
(11
3 [ul
B
log Tog((()) + aM ywr, (o)..))))))))) un)
80 a
log (vrl=)
Ivwl
14

---

AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization
the additional (M, yur Ver m1 component increases weight when the reward estimate is accurate, ensuring more robust reward signal.
Motivational Core The central goal of the proposed AIphaDPO is to address the unreliability of the reference policy, as outlined in Section 3.1. By integrating the policy model into the reference model design, the quality of the reference model is enhanced, improving fine-tuning performance. Similar concepts have been explored in recent works (Gorbatovski et al., 2025; Liu et al., 2025b).
D. Experiments
D.I. Implementation Details
We observed that the performance of various methods is highly sensitive to model parameters and learning rates. To ensure fair comparison, we conducted a hyperparameter search specified in the respective papers. The specific search ranges are detailed in Table 3. Furthermore, due to recent updates to both Llama3-88 8b and Instruct-7b models, we had to re-implement SimPO as the original results were no longer directly applicable.
Training hyperparameters. For other parameters, we used consistent batch size of 128 across all methods. The learning rate was searched within the range of [3e-7, 5e-7, 8c-7, 1c-6], and all models were trained for a single epoch with a cosine learning rate schedule and 10% warmup phase. Adam was used as the optimizer (Kingma & Ba, 2014). Additionally, the maximum sequence length was set to 2048.
Table Various preference optimization objectives and hyperparameter search range.
Method
Objective
Hyperparameter
DPO (Rafailov al., 2023)
log (3 (glog FarlMal
(a(Nul))
5 log Fefsy Furl)el()
(0.01,0.05,00
IPO (Azar al., 2024)
log xe(Huls) log [a(sil)) TarlDulx)
r [0.01, 1.0)
FarlDela)
CPO (Xu ct al., 2024)
- log a (13 log TO (M (x) B log To (yn (xx)) - A log (B)l))))
1.0, , E [0.01, 0.05,.0.
-Awor (3 log Noluells)
That (thw (z) Beer + Aio Arer B log ma(ucla))
A Am 1.0
KTO (Ethayarajh et al., 2024)
mer(sel)))
where 2xet (..D)) [.KI. (TO [x)] (v)((())))]]]
8 [0.01, 0.1]
log Po (vwlzz) X log (log palio (e)
[pppoull]] log pa(wrla) p-potaila)
ORPO (Hong et al., 2024)
Xe 0.5, 1.0,2.0
where po(v)=) exp [ log (z/la))
R-DPO (Park et al., 2024)
-log (3 (3log log ralswl)) amilual))
B log roipi (alvwl coluil) knrlucl))
E 00055, 0.1, 0.5, 1.0]
3 E [0.01, 0.05, 0.1]
SimPO (Meng et al., 2024)
log (Iww) log (o(wwlx)
8 bill
log no (o(l)z)) -7)
8 [2.0, 4.0,6.0...0 6.0,
7 E 0.3, 0.5, 1.0, 1.0, 1.6
log (u(2, Yors Mi) sg (T aM (I,, Mwr, 9x)])
8 E [2.5, 10.0). 7 0..1. [ee 2, be 2,0.1.0.2
AIphaDPO
where u(x, Nu., Mc) Tual log Te (Mw lucl log mo(yi
Table 4. The hyperparameter values in AlphaDPO used for each training setting.
Setting
8
Learning rate
Mistral-Instruet
2.5
0.15
5e-2
6e-7
Llama3-Instruct
2.5
0.6
0.2
le-6
10.0
0.2
1e-6
Gemma2-Instrre
10.0
0.4
5e-2
8e-7
Hyperparameter in AlphaDPO. Table 4 outlines the hyperparameters used for AlphaDPO under various settings. It's important to note that while our approach involves three key parameters, we have found through experience that B can be reliably set to 10.0 by default. Among these parameters, 7 typically requires more careful tuning. As for a, we have observed consistent performance improvements when set to 5e-2 by default. If you are already familiar with the parameter settings for SimPO, you can focus your search primarily on a or simply adopt the default setting of a 5c
15

---

AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization
32
73.4
52
58.0
73.2
57.5
73.0
72.8
72.6
56.5
29
56.0
72.2
44
55.5
72.0
0.00
0.05
0.10
0.15
0.20
0.00
0.05
0.10
0.15
0.20
0.00
0.05
0.30
0.15
0.20
0.00
0.05
0.10
0.15
0.20
Figure 5. AlphaDPO LC on AlpacaEval with different a values.
Decoding hyperparameters. The decoding hyperparameters used in this study are the same as those employed by SimPO*. We extend our sincere gratitude to the SimPO team for sharing their invaluable insights.
Computation environment. All training experiments presented in this paper were conducted using 8xA 100 GPUs, as per the procedures detailed in the alignment-handbook repository."
D.2. AIphaDPO without length-normalized
In this paper, we consider length- normalized training as stability technique and as a primary contribution of this work. Existing research (Meng al., 2024) has demonstrated that length normalization can indeed enhance model performance, particularly with respect the length control win rate. However, to validate the general applicability of JlphaDPO--ppecfficalll its stability and performance without length normalization-wer conducted experiments across several models: meta- Ilama/Meta- Maatma//Mt--L-amaa---
Llama-3- 3- Instructt mistralai/Miittrl-7BB -inialliiiiisttt----s-n-------- Instruct-v0.2, and google/gemma-2-991 2-9bbiit.
We evaluated DPO, SimPO without length normalization, and AlphaDPO without length normalization. The experimental results, as shown in Table 5, demonstrate that AlphaDPO consistently achieves performance improvements even without the use of length normalization. This indicates the robustness and general effectiveness of AlphaDPO.
Table 5. Performance comparison without length-normalizatioo on AlpacaFval2. "I.C" denotes the length-controlled win rate, and "WR" represents the win rate.
-lama3-Instruce (8B)
Mistral-Instruct (7B)
Llama3-Instruct v0.2 (8B)
Gemma2-Instrus (9B)
Method
LC (%)
WR (%)
LC (%)
WR (%)
LC (%)
WR (%)
LC (%)
(%)
DPO SimPO w/o LN
40.2
38.1
20.3
18.0
51.1
53.3
70.2
66.9
42.4
40.4
30.5
38.2
49.2
52.6
71.2
69.9
AlphaDPO w/o I.N
44.4
42.6
32.0
38.4
51.1
54.0
72.7
70.5
D.3. AlphaDPO with differenct
To analyze the impact of on the model, we adjust its value for four different models. The results are illustrated in Figure 5. When is set to 0, the model degenerates to SimPO. As a increases, performance improves across all models, although the optimal value of a varies among them. This highlights the significance of a.
is noteworthy that within the parameter tuning range [1e-2, 5e-2, 0.1, 0.2], the optimal a values are consistently around 0.1 or even closer 5e-2.
"https://gt thub. http://wwwrrraattooooo////////// Sttpps: //gi thub. http://cw//uuggiiggaaaaaaagggeee
16

---

AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization
3.3 Si Sim 3im00rym2.3) 3)
Aph ph
5mm0
0.
Apnxxboojem..a 0.20
phu3ro
Desa
213
3mP00yy0.55
ApnubPotomm..l
20
Rewerd ott. (xx.22-0k.0)
00
aitring páralo)
22
Figure 6. (a) SimPO: Reward difference distribution under different values. AlphaDPO: Reward difference distribution under different a values. (c) SimPO: Log likelihood distribution on chosen responses under different values. (d) AlphaDPO: Log likelihood distribution on chosen responses under different values.
D.4. Comparison With TDPO
To investigate the relationship between TDPO and AIphaDPO, we conducted the experiments, with the results outlined below.
Table Performance comparison between TDPO and AlphaDPO.
Llama3-Instrut' (8B)
Method
LC (%)
WR (%)
TDPO
52.8
45.9
AIphaDPO w/ 5(2, Y/r, 91)
56.9
50.4
AIphaDPO w/ M (2r, Ywo y1)
58.7
51.1
In its original form, TDPO did not perform well on Llama3-8B. By applying Lemma 4.1, we modified the expression M (a, yw y1) in AIphaDPO to use TDPO's Ö(x, Vu 9t), converting our sentence-level estimations to token-level calculation. This adjustment resulted in noticeable performance improvement, which attribute the nonggh--ormalizaiion, 7 and z-score normalization of 6(, Vus 31). Nevertheless, the modified TDPO still underperformed compared to AlphaDPO. This indicates that, when the Tee is suboptimal, token-level calculations are prone to significant errors.
17


Reviews:
[{"review_id": "35oAqKnLm4", "reviewer": "Reviewer_GxTu", "summary": "This paper proposes a new training algorithm for LLM alignment. First, the authors unify the training objective of the two representative alignment training algorithms, DPO and SimPO, into a single one with a fixed margin. Next, they propose a new training algorithm to mitigate the limitation of each algorithm, by introducing adaptive margin which is constructed by interpolating the fixed original reference model and the training policy model. The effectiveness of this method is first demonstrated with the theoretical results. Also, the empirical results with various state-of-the-art open-source LLMs (e.g., Llama3 or Gemma2) on standard benchmarks (AlpacaEval 2 and Arena-Hard) further support its effectiveness.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Similar idea in previous works.**\n\nWe sincerely thank the reviewer for pointing out relevant prior works. While both our method and previous approaches involve model interpolation, we highlight three key distinctions:   \n- **Adaptive reward margin**: Unlike existing works that focus on regularization strength [1] or data generation [2], AlphaDPO introduces instance-adaptive reward margins by dynamically reparameterizing the reference distribution through the policy-to-reference ratio ($\\pi_θ/π_{ref}$), enabling personalized preference learning that accounts for per-sample preference strength.\n- **Implicit KL control**: Our theoretical analysis reveals that the $\\alpha$-weighted ratio term implicitly controls sequential KL divergence between iterative policy updates, achieving stability without explicit constraints.  \n- **Generalized Preference Optimization**: AlphaDPO generalizes SimPO as special cases ($\\alpha=0$) while enabling smooth transitions between policy specialization ($\\alpha>0$) and uniform exploration ($\\alpha \\to 0$). The empirical superiority across three model families (Table 1), demonstrates the critical advantage of adaptive margins.\n\nWe will add detailed comparisons to these works in the revised manuscript.\n\n---\n\n**Q2: Sensitivity to $\\alpha$**\n\nWe appreciate the reviewer's insightful questions regarding hyperparameter sensitivity. Here we clarify our methodology: \n\n(1) **Primary $\\alpha$ Search Space**: As shown in `Appendix Table 3`, we conducted systematic searches over $\\alpha \\in $ {0.01, 0.05, 0.1, 0.2} based on validation performance. Figure 5 demonstrates that $\\alpha=0.05$ achieves optimal results across most models except Llama3-IT-8B-v0.2 where $\\alpha=0.2$ works best, reflecting architecture-dependent calibration needs.\n\n(2) **Extended Analysis in Figures**: The expanded $\\alpha$ values in Figures 3-4 (including 0.3, 0.5, etc.) were intentionally explored to demonstrate our method's reward distribution characteristics across a broader spectrum, not to claim performance improvements. We acknowledge this caused unintended confusion and will explicitly label these as \"analysis beyond primary search space\" in revisions.\n\n(3) **Baseline Fairness**: All methods including SimPO used identical hyperparameter search budgets. For SimPO's $\\gamma$, we strictly followed the original paper's recommendation space {0.3, 0.5, 1.0, 1.2, 1.4, 1.6} as shown in `Appendix Table 3`, ensuring fair comparison through equivalent tuning efforts.\n\nThis controlled approach ensures our conclusions about AlphaDPO's advantages remain valid despite model-specific $\\alpha$ variations, while the extended analyses provide valuable insights into the method's behavioral patterns.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes, the theoretical results in sections 3 and 4 look valid.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "New ideas and results are key contributions.", "essential_references_not_discussed": "Considered references are sufficient, but it might be nice if the relevant papers are more added. Those works are mentioned in below.", "other_strengths_and_weaknesses": "### Pros\n\n1. **Clarity**. Overall, the writing is clear and easy to follow. In addition, the organization of the main draft is well-established.\n2. **Well-motivated problem and intuitive approach.** Alignment of LLM is important direction and the proposed method seems to be intuitive and effective.\n\n### Cons\n\n- **Similar idea in previous works**: The idea of constructing adaptive reference model through the interpolation between the fixed original reference model $\\pi_{\\text{ref}}$ and the training policy model $\\pi_{\\theta}$ has been explored in previous works [1,2]. While the purpose is quite different, the technical contribution of this work is therefore relatively restricted. The authors should cite these works and add the discussion to clarify the difference and the contribution compared to these works.\n- **Sensitivity to $\\alpha$**: While the proposed method is very sensitivity to the choice of $\\alpha$, the authors never mention  is the specific search space for $\\alpha$ and how they chose this hyper-parameter for the tables. According to Figure 5 in Appendix D, it seems like that the authors chose the different hyper-parameters that yield the best performance for target LLMs among {0, 0.05, 0.1, 0.15, 0.2}; for example, $\\alpha=0.05$ for Mistral IT 7B and $\\alpha=0.2$ for Llama IT 8B v0.2. If it is true,  the choice of $\\alpha$ in Figure 3 and 4 is quite weird, as $\\alpha=0.3$ and $\\alpha=0.01$ are not in the original search space. Also, it’s unclear whether the authors did the same efforts to the considered baselines with the same amount of search space in their hyper-parameters; for example, searching optimal $\\gamma$ in SimPO with same number of hyper-parameter space.\n\n[1] Liu et al., Decoding-time Realignment of Language Models., ICML 2024  \n[2] Kim et al., Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment., ICLR 2025", "other_comments_or_suggestions": "Please respond to the weaknesses above.", "questions_for_authors": "Please respond to the weaknesses above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YeutcA1WqT", "reviewer": "Reviewer_oKhi", "summary": "This paper proposes a novel strategy for LLM alignment designed to address the limitations of SimPO and DPO. The proposed AlphaDPO adaptively sets the reward margin based on the ratio between the preference model and the policy model. The relations to SimPO and TDPO loss have been studied. Extensive experiments demonstrate AlphaDPO's superior performance across multiple baselines and LLM architectures.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: What is the practical utility of the theoretical connection between AlphaDPO and online methods given that online methods themselves lack strong theoretical guarantees?**\n\nWe appreciate the reviewer raising this important point. Although we did not explicitly emphasize the theoretical connection to online methods in our submitted manuscript, we acknowledge that exploring this relationship represents a promising direction for future research. We believe this connection offers two meaningful benefits:\n\n1. **Algorithmic Insights:**  \n   By establishing a theoretical link between AlphaDPO and online methods, we gain a unified view of how adaptive reward margins and the implicit reference model influence policy optimization. Specifically, AlphaDPO's framework demonstrates how sequential KL divergence control naturally arises, thereby providing a clearer understanding of its inherent ability to balance alignment and diversity—even when the reference model is suboptimal. Such theoretical insights were previously unaddressed explicitly in existing online methods.\n\n2. **Empirical Robustness and Practical Guidance:**  \n   Despite the lack of rigorous theoretical guarantees in existing online methods, the theoretical analysis of AlphaDPO indicates that its adaptive mechanism implicitly mitigates typical online optimization pitfalls, such as over-optimization. This robustness is empirically demonstrated through AlphaDPO's stable performance across various KL divergence budgets, as illustrated in Figure 3(c) of our paper.\n\nWe agree with the reviewer's point and will pursue a thorough theoretical investigation of this connection as part of our future work, aiming to further clarify its theoretical foundations and practical implications.\n\n---\n**Q2: Why is AlphaDPO effective when the reference model is not well-calibrated at the token level, given that it operates at the sequence level?**\n\n**Full Details**: A comprehensive theoretical analysis (including Lemma 4.1 and robust optimization derivations) is provided in the `Response to Reviewer bC1g`.  \n\n\nAlphaDPO’s robustness to token-level miscalibration stems from **sequence-level KL divergence approximation** and **adaptive margin design**, which mitigate noise propagation from unreliable token-level signals.  \n\n1. **Theoretical Foundation**  \n   - **Problem Context**: When $\\pi\\_{\\text{ref}}$ is miscalibrated at the token level, token-wise KL terms (e.g., in TDPO) amplify noise.  \n   - **Key Insight**: By approximating the *sequential KL divergence* (Lemma 4.1), AlphaDPO aggregates token-level uncertainties into a sequence-level margin $M(x,y\\_w,y\\_l)$. This reduces sensitivity to token-level errors, as the sequence-level signal is statistically more stable.  \n   - **Robust Optimization**: Our framework explicitly models bounded deviations from $\\pi\\_{\\text{ref}}$ (Section 3.2), ensuring stability even when token-level probabilities are imperfect.  \n\n2. **Empirical Validation**  \n   - **Performance Gain**: As shown in Appendix Table 6, AlphaDPO outperforms TDPO (58.7% vs. 52.8% LC win rate on Llama3-8B), demonstrating superior robustness to reference model noise.  \n   - **Bias Compensation**: Figure 2 (main paper) shows $\\pi\\_{\\text{ref}}$ fails to distinguish $y\\_w$ from $y\\_l$ at the token level. AlphaDPO’s adaptive margin $M$ compensates for this by leveraging sequence-level discrepancies, ensuring stable alignment.  \n\n**Key Takeaways**  \n- **Sequence-Level Robustness**: AlphaDPO avoids token-level noise amplification via sequence-wise KL control, making it less reliant on perfect token calibration.  \n- **Adaptive Margin**: The margin $M(x,y\\_w,y\\_l)$ dynamically adjusts to instance-specific reference model errors, enhancing robustness.  \n- **Empirical Edge**: AlphaDPO’s design consistently outperforms token-level methods (e.g., TDPO) in scenarios with miscalibrated $\\pi\\_{\\text{ref}}$.  \n\n---\n\n**Q3: When does AlphaDPO degrade to DPO?**  \n\nCurrently, the $\\alpha$-DPO algorithm cannot be transformed into DPO merely through parameter adjustments, similar to how SimPO cannot be converted to DPO by altering $\\gamma$. However, I believe this topic presents significant promise, allowing us to propose a more generalized formulation:\n$$\n\\hat{\\pi}\\_{\\text{ref}}(\\cdot|x) \\propto U(\\cdot|x) \\cdot \\pi\\_\\theta^{\\alpha\\_1}(\\cdot|x) \\cdot \\pi\\_{\\text{ref}}^{\\alpha\\_2}(\\cdot|x),\n$$\nwhere $U(\\cdot|x)$ is a uniform distribution. This formulation encompasses:  \n- **DPO**: Set $\\alpha\\_1 = 0$, $\\alpha\\_2 = 1$, recovering the explicit reference model $\\pi\\_{\\text{ref}}$.  \n- **SimPO**: Set $\\alpha\\_1 = \\alpha\\_2 = 0$, yielding a uniform reference model.  \n- **AlphaDPO**: Set $\\alpha\\_1 = \\alpha$, $\\alpha\\_2 = -\\alpha$, enabling adaptive margin control via $\\pi\\_\\theta/\\pi\\_{\\text{ref}}$.", "claims_and_evidence": "Yes, the claims are mostly supported through both theoretical analysis and experimental results.", "methods_and_evaluation_criteria": "Yes, the proposed method aligns well with the LLM preference optimization problem, directly addressing two identified limitations in existing approaches. The evaluation criteria employ standard benchmarks (AlpacaEval 2 and Arena-Hard) and diverse model architectures (Mistral2-7B, Llama3-8B, Gemma2-9B), providing comprehensive evidence of AlphaDPO's effectiveness across different settings.", "theoretical_claims": "Yes, I have checked the proofs provided in this submission, including those in the appendix. I did not find any issues.", "experimental_designs_or_analyses": "Yes, I have examined the experimental designs and analyses in the paper, particularly those in section 5 and appendix D.", "relation_to_broader_scientific_literature": "1. This work explores an essential problem in preference optimization methods—how to effectively utilize the reference model. AlphaDPO proposes a novel interpolation between the current policy model and uniform policy, providing a bridge between DPO and SimPO, offering a more flexible framework.\n2. While AlphaDPO is fundamentally an offline preference optimization technique, its adaptive nature shares conceptual similarities with online RL approaches. The adaptive reference model effectively serves as a dynamic critic, similar to how value functions guide policy updates in online RL.\n3. AlphaDPO provides a theoretically grounded approach to the critical balance between alignment and diversity via KL divergence control.", "essential_references_not_discussed": "To my knowledge, this paper has included sufficient references.", "other_strengths_and_weaknesses": "Strengths:\n- The paper introduces instance-specific margins that advance beyond the fixed approach in SimPO. It establishes connections between existing alignment methods (particularly DPO and SimPO), creating a unified framework that addresses limitations of both approaches.\n- Extensive experiments consistently demonstrate AlphaDPO's superior performance across various LLM architectures and benchmarks. The comprehensive ablation studies effectively isolate the contributions of different components of the approach.\n- The authors provide theoretical analysis on the lower bound and its connections to TDPO.\n- The paper is well-written with clear motivation and is easy to follow.\n\nWeaknesses:\n- While the authors establish a theoretical connection between AlphaDPO and online methods, questions remain about the practical utility of this theoretical framework, given that online methods themselves lack strong theoretical guarantees.\n- The authors claim that AlphaDPO is particularly effective \"when the reference model is not well-calibrated at the token level.\" However, this statement appears contradictory given that AlphaDPO itself operates at the sequence level rather than implementing token-level optimization.\n- From the formulation of the adaptive preference distribution, it’s unclear at what condition it degrade to DPO.", "other_comments_or_suggestions": "None.", "questions_for_authors": "- What is the practical utility of theoretical connection between AlphaDPO and online methods given that online methods themselves lack strong theoretical guarantees?\n- Why does AlphaDPO particularly effective when the reference model is not well-calibrated at the token level?\n- When does AlphaDPO degrade to DPO?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H8Hbr7Jp7s", "reviewer": "Reviewer_ciXp", "summary": "This paper propose AlphaDPO, a new preference optimization framework. The core novelty of this framework is to modify the reference model distribution as the product of a uniform distribution and the ratio between policy model and original reference model, with power factor of alpha. This is effectively equivalent to interpolate between SimPO and DPO. \n\nThe paper perform experiment on AlpacaEval2 and Arena-Hard with 3 representative LLMs, and the empirical results show AlphaDPO has better performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Clarification on DPO's Reference Model Limitation**  \nWe appreciate this insightful question. The necessity for $\\pi\\_{\\text{ref}}$ to distinguish between $y\\_w$ and $y\\_l$ stems from two fundamental aspects of KL-regularized policy optimization:\n1. **Theoretical Foundation of KL-Regularized Objectives**: The RLHF objective (Equation 2) regularizes the policy $\\pi\\_\\theta$ to stay close to $\\pi\\_{\\text{ref}}$ via KL divergence. This regularization implicitly assumes that $\\pi\\_{\\text{ref}}$ provides a meaningful prior for distinguishing high-quality ($y\\_w$) and low-quality ($y\\_l$) responses. If $\\pi\\_{\\text{ref}}$ lacks discriminative power (e.g., assigns similar probabilities to $y\\_w$ and $y\\_l$), the KL term loses its grounding, leading to unstable optimization.\n\n2. **Empirical Evidence**:Recent studies [1,2] demonstrate that $\\pi\\_{\\text{ref}}$ quality significantly impacts DPO performance. DPO's reliance on a static $\\pi\\_{\\text{ref}}$ introduces both theoretical and practical limitations.  \n\n[1] Gorbatovski et al. Learn your reference model for real good alignment. ICLR 2025.   \n[2] Liu et al. Understanding Reference Policies in Direct Preference Optimization. arXiv preprint arXiv:2407.13709.\n\n**Q2: Formalization of $U(y|x)$ and Its Impact**  \nWe thank the reviewer for prompting this clarification. The uniform distribution $U(y|x)$ and its role in $\\gamma$ can be rigorously defined as follows:\n\n1. **Theoretical Framework**:\n   Let $\\mathcal{V}$ denote the vocabulary. The *theoretical* $U(y|x)$ is:\n   $$\n   U(y|x) = \\prod\\_{t=1}^{|y|} \\frac{1}{|\\mathcal{V}|} \\quad \\text{(uniform over all tokens)}\n   $$\n   However, *empirically*, $y\\_w$ and $y\\_l$ are selected via:\n   - **Sampling**: Generate $y\\_1,...,y\\_5 \\sim \\pi\\_{\\text{SFT}}(y|x)$\n   - **Selection**: $y\\_w = \\arg\\max\\_{y\\_i} \\text{score}(y\\_i)$, $y\\_l = \\arg\\min\\_{y\\_i} \\text{score}(y\\_i)$\n   This induces *implicit vocabulary subspaces*:\n   $$\\mathcal{V}\\_{\\text{win}} = \\\\{y \\in \\mathcal{V} \\mid \\text{score}(y) \\geq \\tau\\\\}, \\quad \\mathcal{V}\\_{\\text{lose}} = \\\\{y \\in \\mathcal{V} \\mid \\text{score}(y) \\leq \\tau'\\\\}$$\n\n2. **Effective $U(y|x)$ in Practice**:\n   The *effective* $U(y\\_w|x)$ and $U(y\\_l|x)$ become:\n   $$U(y\\_w|x) = \\prod\\_{t=1}^{|y\\_w|} \\frac{1}{|\\mathcal{V}\\_{\\text{win}}|}, \\quad U(y\\_l|x) = \\prod\\_{t=1}^{|y\\_l|} \\frac{1}{|\\mathcal{V}\\_{\\text{lose}}|}$$  \n   This leads to:$\\gamma = \\beta \\left( \\log U(y\\_w|x) - \\log U(y\\_l|x) \\right)$. The performance impact arises because $\\mathcal{V}\\_{\\text{win}}$ and $\\mathcal{V}\\_{\\text{lose}}$ differ across instances, making SimPO's fixed $\\gamma$ suboptimal.  \n\n**Q3: Alignment of AlphaDPO with DPO at $\\alpha=1$**  \nWe sincerely appreciate the reviewer's careful observation and apologize for the ambiguity in our initial formulation. The implicit reference model $\\hat{\\pi}\\_{\\text{ref}}(y|x)$ is defined as:\n$$\\hat{\\pi}\\_{\\text{ref}}(y|x)\\propto U(y|x) \\left( \\frac{\\pi\\_\\theta(y|x)}{\\pi\\_{\\text{ref}}(y|x)} \\right)^\\alpha,$$  \nwhere $\\alpha$ interpolates between two extremes:\n1. **When $\\alpha = 0$**: $\\hat{\\pi}\\_{\\text{ref}}(y|x)$ reduces to the uniform distribution $U(y|x)$, aligning with SimPO's implicit reference model.  \n2. **When $\\alpha > 0$**: $\\hat{\\pi}\\_{\\text{ref}}(y|x)$ increasingly incorporates the dynamic term $\\frac{\\pi\\_\\theta}{\\pi\\_{\\text{ref}}}$, creating an adaptive reference model. Critically, **AlphaDPO does not strictly reduce to DPO for any finite $\\alpha$**. Instead, it introduces a novel framework that balances exploration (via $U(y|x)$) and exploitation (via $\\frac{\\pi\\_\\theta}{\\pi\\_{\\text{ref}}}$).  \n\nWe will revise the manuscript to eliminate the misleading claim about AlphaDPO reducing to DPO and instead emphasize its unique interpolation mechanism.\n\n**Q4: Statistical Significance of Performance Gains**  \nWe respectfully disagree. Table 1 shows statistically significant improvements across benchmarks:\n\n|Model|AlpacaEval2(LC)|Arena-Hard(LC)|\n|-|-|-|\n|Llama3-8B|+6.4%(43.8→46.6)|+2.1%(33.5→34.2)|\n|Mistral-7B|+7.0% (30.2→32.3) |+8.6%(19.8→21.5)|\n|Llama3-v0.2-8B|+5.6% (55.6→58.7)|+6.8%(34.0→36.3)|\n|Gemma2-9B|+1.3%(72.4→73.4)|+5.7% (56.1→59.3)|\n\nThese gains are consistent and meaningful, particularly given the saturated performance of modern LLMs. To rigorously validate the robustness of our improvements, we report the standard deviations and confidence intervals of current evaluations in (https://anonymous.4open.science/r/AlphaDPO-431F/significant_exp.md).\n\n**Q5: Comparison with Online DPO Methods**  \nAlphaDPO is orthogonal to online preference optimization. While OAIF/IDPO focus on *data collection dynamics*, our work addresses *reference model design* in offline settings. Notably, AlphaDPO's adaptive reference can be integrated into online frameworks by replacing static $\\pi\\_\\text{ref}$ with $\\hat{\\pi}\\_{\\text{ref}}$. We acknowledge this as valuable future work and will explore it in subsequent studies.", "claims_and_evidence": "Most claims are fine. \n\nOne question is in Line147, the limitations of DPO, why is the $\\pi_{ref}$ supposed to distinguish between $y_w$ and $y_l$? Why is this a limitation? Doesn't the $\\pi_{ref}$ just serve as the reference for reward values?", "methods_and_evaluation_criteria": "Strengths: \n- The methods is fine, and the ablation study about other potential design attempts of AlphaDPO, in Table 2, is also convincing.  \n\nWeakness:\n- How the the value of $U(y|x)$ decided? How does the uniform values affect the performance?", "theoretical_claims": "- In Line 208: Can you explicitly show that when $\\alpha=1$, how AlphaDPO aligns with DPO?\n\n\n- The motivation of Principle 1 in Line 171 is not supported. Why should the reference model contribute to differentiating between preferred and less preferred responses?", "experimental_designs_or_analyses": "Strengths:\n\n- The experiment is extensive. Many baseline preference learning methods are compared.", "relation_to_broader_scientific_literature": "This paper fits into the literature of DPO-like preference learning methods. The proposed method is somewhere between the DPO method and SimPO method. The contribution is try to have a tradeoff between those two methods.", "essential_references_not_discussed": "N/a", "other_strengths_and_weaknesses": "Weakness: \n- The proposed method is relatively straightforward, and the performance gain from SimPO is marginal.", "other_comments_or_suggestions": "N/a", "questions_for_authors": "How do the author compare this method with the Online DPO methods such as OAIF (Direct Language Model Alignment from Online AI Feedback) and IDPO (Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2a6VysDLRh", "reviewer": "Reviewer_bC1g", "summary": "This paper introduces AlphaDPO, an adaptive preference optimization framework that improves alignment in large language models (LLMs) by dynamically adjusting the reward margin in preference learning. The key contribution is the introduction of an implicit reference model that interpolates between policy-driven adaptation and uniform exploration, leading to instance-adaptive reward margins. Empirical results demonstrate the superiority of AlphaDPO over previous methods.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for raising this important concern. Below, we provide a rigorous theoretical justification for the approximation in Lemma 4.1 and clarify its empirical validity.\n\n### **Theoretical Justification**  \n\n**1. Problem Formulation with Robustness Constraints**  \nOur objective is to minimize the sequential KL divergence between the optimal reference policy $\\pi\\_{\\text{ref}}^*$ and the policy $\\pi\\_\\theta$, under the uncertainty of $\\pi\\_{\\text{ref}}^*$:  \n$$\n\\min\\_\\theta \\sum\\_{t=1}^T \\mathbb{E}\\_{\\pi\\_{\\text{ref}}^*}\\left[ \\log \\frac{\\pi\\_{\\text{ref}}^*(z|x,y\\_{<t})}{\\pi\\_\\theta(z|x,y\\_{<t})} \\right].\n$$\nSince $\\pi\\_{\\text{ref}}^*$ is unobserved, we assume bounded deviation from an observable reference policy $\\pi\\_{\\text{ref}}$:  \n$\n|\\pi\\_{\\text{ref}}^*(\\cdot) - \\pi\\_{\\text{ref}}(\\cdot)| \\leq C, \\quad C > 0.\n$\nThis leads to a constrained robust optimization problem:  \n$$\n\\min\\_\\theta \\max\\_{\\pi\\_{\\text{ref}}^*} \\sum\\_{t=1}^T \\mathbb{E}\\_{\\pi\\_{\\text{ref}}}\\left[ \\log \\frac{\\pi\\_{\\text{ref}}^*(z|x,y\\_{<t})}{\\pi\\_\\theta(z|x,y\\_{<t})} \\right]\n\\text{s.t.} \\quad |\\pi\\_{\\text{ref}}^*(z|x,y\\_{<t}) - \\pi\\_{\\text{ref}}(z|x,y\\_{<t})| \\leq C. \\nonumber\n$$ \n\n**2. Simplification via Worst-Case Analysis**  \nFor the inner maximization, the worst-case $\\pi\\_{\\text{ref}}^*$ is determined by the sign of the log-ratio:  \n$$\n\\pi\\_{\\text{ref}}^*(z|x,y\\_{<t}) = \n\\begin{cases} \n\\pi\\_{\\text{ref}}(z|x,y\\_{<t}) + C, & \\text{if } \\log \\frac{\\pi\\_{\\text{ref}}(z|x,y\\_{<t})}{\\pi\\_\\theta(z|x,y\\_{<t})} \\geq 0, \\\\\\\\\n\\pi\\_{\\text{ref}}(z|x,y\\_{<t}) - C, & \\text{otherwise}.\n\\end{cases}\n$$ \nSubstituting this into the objective yields:  \n$$\n\\min\\_\\theta \\sum\\_{t=1}^T \\left[ C \\cdot \\log \\frac{\\pi\\_{\\text{ref}}(z|x,y\\_{<t})}{\\pi\\_\\theta(z|x,y\\_{<t})} + \\left|\\pi\\_{\\text{ref}}(z|x,y\\_{<t}) \\log \\frac{\\pi\\_{\\text{ref}}(z|x,y\\_{<t})}{\\pi\\_\\theta(z|x,y\\_{<t})} \\right| \\right].\n$$\n\n**3. Asymptotic Approximation**  \nWhen the deviation bound $C$ is sufficiently large (i.e., $\\pi\\_{\\text{ref}}^*$ is less constrained), the absolute value term dominates, and the objective simplifies to:  \n$$\n\\min\\_\\theta \\sum\\_{t=1}^T \\left[ \\log \\frac{\\pi\\_{\\text{ref}}(z|x,y\\_{<t})}{\\pi\\_\\theta(z|x,y\\_{<t})} \\right].\n$$\nThis corresponds to the sequence-level approximation in Lemma 4.1. The $\\approx$ symbol reflects the asymptotic regime where higher-order terms vanish under large $C$, which aligns with practical scenarios where $\\pi\\_{\\text{ref}}$ is imperfect but provides a reasonable prior.  \n\n**4. Addressing the Uniform Policy Assumption**  \nWe clarify that Lemma 4.1 does *not* assume $\\pi\\_{\\text{ref}}$ is uniform. Instead, it leverages the structure of the KL divergence to show that the margin term $M(x,y\\_w,y\\_l)$ approximates the sequential KL difference under bounded deviations. The uniform policy in SimPO is a special case of our framework (when $\\alpha=0$), but our method generalizes to non-uniform $\\pi\\_{\\text{ref}}$ by adaptively scaling with $\\alpha$.  \n\n---\n\n### **Empirical Validation**  \n\n**Performance**  \nAs demonstrated in `Appendix Table 6`, AlphaDPO achieves significant performance gains over TDPO, with LC win rates increasing from 52.8% (TDPO) to 56.9% (AlphaDPO w/ $\\delta$) and further to 58.7% (AlphaDPO w/ $M$). This progressive improvement highlights the effectiveness of the sequence-level approximation in mitigating token-level noise through variance reduction. By replacing TDPO's token-level margin $\\delta$ with the adaptive sequence-level margin $M$, AlphaDPO enhances robustness while maintaining alignment, underscoring the superiority of sequence-level optimization in handling imperfect reference models.\n\n**Mitigating Reference Model Bias**  \nFigure 2 (main paper) empirically validates that $\\pi\\_{\\text{ref}}$ struggles to distinguish $y\\_w$ from $y\\_l$ ($\\log\\pi\\_{\\text{ref}}(y\\_w|x) - \\log\\pi\\_{\\text{ref}}(y\\_l|x)$ exhibits random fluctuations). AlphaDPO's adaptive margin $M(x,y\\_w,y\\_l)$ explicitly compensates for this bias, ensuring stable optimization even when $\\pi\\_{\\text{ref}}$ is suboptimal.\n\n---\n\n### **Conclusion**  \n\nWhile Lemma 4.1 uses an approximation symbol ($\\approx$), our analysis rigorously justifies its validity under bounded reference model mismatch. The empirical success of AlphaDPO further supports this design choice, demonstrating that sequence-level approximations enhance robustness without sacrificing performance. We acknowledge that deriving a formal error bound remains an open question and will explore this in future work.", "claims_and_evidence": "The empirical claims are valid judged from the experiments results. Lemma 4.1 is problematic.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Lemma 4.1 is problematic. The authors use $\\approx$ in the lemma, which is not rigorous as a mathematical lemma. The proof is not convincing either. The authors claim that the pretrained model $\\pi_{ref}$ is close to uniform policy, which is apparently not true.", "experimental_designs_or_analyses": "Yes, they look good to me.", "relation_to_broader_scientific_literature": "The key contribution of this paper is a new reference policy for DPO loss.  The experiment results show that this method is superior to existing DPO and its variants. However, the theoretical analysis is not sound and rigorous. I suggest the authors fix this issue.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "None.", "other_comments_or_suggestions": "None.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P66v8vagX5", "reviewer": "Reviewer_TEY3", "summary": "This paper proposes AlphaDPO which is a direct preference optimization method with a data-dependent margin. The authors first observe that the DPO objective can be looked at as making the likelihood of the chosen response to be greater than the losing response, with a margin set as the difference between the likelihood of the responses under the reference model. The authors claim that such margin might introduce error in scenarios where the reference model is not calibrated. Therefore, they propose a margin that is the likelihood ratio of the responses under the language model and the reference model. They show how this proposal is theoretically related to SimPO and TDPO. They further empirically compare AlphaDPO with SimPO and several other DPO variants on AlpacaEval with Llama3(8B), Mistral(7B), and Gemma2(9B) models.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Justification of Loss Function**\n\nWhile space constraints limited introductory intuition, we provide multi-faceted justification through:\n1) **Weak-to-Strong Generalization** (Lines 197-200): Similar to weak-to-strong alignment, our adaptive reference models enable policy specialization while preserving exploration.\n2) **Theoretical Link to TDPO** (Sec. 4): Lemma 4.1 shows AlphaDPO's margin term approximates TDPO's KL divergence control, connecting sequence optimization with token-level stability.\n3) **Appendix Analysis**: Utility theory (C.1) and gradient analysis (C.2) demonstrate how our adaptive margin prevents reward hacking while maintaining diversity.\n\nWe will add a roadmap paragraph in the introduction to better signpost these analyses.\n\n**Q2: Theoretical Comparison to SimPO**\n\nWe clarify our theoretical progression:\n1. TDPO's success demonstrates the effectiveness of $r(x,y\\_w)-r(x,y\\_l)-\\delta$ structures for KL control.\n2. AlphaDPO adopts a similar offset structure $r(x,y\\_w)-r(x,y\\_l)-M$ where:\n   - TDPO uses $\\delta=\\beta(D\\_{KL}(y\\_l) - D\\_{KL}(y\\_w))$ with $z\\sim\\pi\\_{ref}$\n   - AlphaDPO uses $M=\\beta(\\log\\frac{\\pi\\_\\theta(y\\_w)}{\\pi\\_{ref}(y\\_w)} - \\log\\frac{\\pi\\_\\theta(y\\_l)}{\\pi\\_{ref}(y\\_l)})$ with uniform sampling\n3. As shown in Appendix D.4, AlphaDPO's $M$ achieves superior performance to TDPO's $\\delta$ (+3.2% AlpacaEval LC), demonstrating the advantage of sequence-level KL approximation over token-level computation.\n\n**Q3: Crude Approximation**  \nDue to space limits, A comprehensive theoretical analysis is provided in the `Response to Reviewer bC1g`.\n\n**Q4: Base Model Validation**  \nTo address potential confounding from pre-aligned models: We conducted additional experiments on Llama3-8B-Base: \n||DPO|SimPO|AlphaDPO|\n|-|-|-|-|\n|truthfulqa_mc2|53.66|60.03|62.89|\n|gsm8k| 52.90|52.84|53.90|\n| mmlu| 62.14 | 62.05|62.43|\n||||\n| MT-Bench|6.5|6.6|6.9|\n||||\n|LC(Alpaca)|14.92|17.97|22.69|\n|WR(Alpaca)|13.02|15.60|20.47|\n\nImprovements remain consistent, confirming AlphaDPO's effectiveness independent of initial alignment.\n\n**Q5: TDPO Comparison**\nCurrent Appendix D.4 shows: AlphaDPO achieves 58.7% LC vs TDPO's 52.8% on Llama3-8B, demonstrating clear superiority.\nThanks for your suggestion, and we will add direct comparisons to Table 1. \n\n**Q6: $\\gamma$ Formulation**\nThe uniform distribution $U(y|x)$ and its role in $\\gamma$ can be rigorously defined as follows:\n1. **Theoretical vs. Empirical $ U(y|x) $**\n   - *Theoretical*: For vocabulary $\\mathcal{V} $, $U(y|x) = \\prod\\_{t=1}^{|y|} \\frac{1}{|\\mathcal{V}|} \\quad \\text{(uniform over tokens)}$\n   - *Empirical*: Responses $ y\\_1, \\dots, y\\_5 \\sim \\pi\\_{\\text{SFT}}(y|x) $ are scored, with $ y\\_w $ and $ y\\_l $ selected via:\n     $y\\_w = \\arg\\max \\text{score}(y\\_i), \\quad y\\_l = \\arg\\min \\text{score}(y\\_i)$\n     This induces implicit subspaces:$\n     \\mathcal{V}\\_{\\text{win}} = \\\\{y\\mid \\text{score}(y)\\geq\\tau\\\\},\\quad\\mathcal{V}\\_{\\text{lose}}=\\\\{y\\mid\\text{score}(y)\\leq \\tau'\\\\}$\n\n2. **Effective $ U(y|x) $ and $\\gamma$**.   \n The *practical* uniform distributions become: $$U(y\\_w|x) = \\prod\\_{t=1}^{|y\\_w|} \\frac{1}{|\\mathcal{V}\\_{\\text{win}}|}, \\quad U(y\\_l|x) = \\prod\\_{t=1}^{|y\\_l|} \\frac{1}{|\\mathcal{V}\\_{\\text{lose}}|}$$\n   Thus, $\\gamma = \\beta (\\log U(y\\_w|x) - \\log U(y\\_l|x))$. Since $ \\mathcal{V}\\_{\\text{win}} $ and $ \\mathcal{V}\\_{\\text{lose}} $ vary per instance, SimPO's fixed $\\gamma$ is suboptimal.\n\n**Q7: Stop Gradient Analysis**\n\nWe highlight the primary distinction:\n\n- **TDPO**: Implements asymmetric gradient flow—enabled for $ y\\_l $, stopped for $ y\\_w $.\n- **AlphaDPO**: Employs symmetric stop-gradient on both terms, using $ \\pi\\_{\\text{ref}} $ as a fixed anchor.\n\nBoth TDPO and AlphaDPO utilize the stop-gradient operation. Examining the detailed impact of single-term versus multi-term stop-gradient is an intriguing avenue for future work.\n\n**Q8: $\\alpha=1$ Misstatement**  \nDue to space constraints, please refer to the response for `Question 3 in Response to Reviewer ciXp.`\n\n**Q9: Experimental Rigor**  \nWe report the standard deviations and confidence intervals of current evaluations in (https://anonymous.4open.science/r/AlphaDPO-431F/significant_exp.md).\n\n**Hyperparameter Fairness**. We strictly followed community standards:\n- All methods underwent equal hyperparameter tuning (`Appendix Table 3`)\n- $\\alpha$ was selected via grid search over [1e-2, 5e-2, 0.1, 0.2] with 5 random seeds\n- DPO received equivalent tuning effort ($\\beta \\in [0.01,0.05,0.1]$)\n\n**KL Analysis**\nWe will enhance Figure 3(c) by:\n- Adding more sampling points (every 50 steps)\n- Including Pareto frontiers for compared methods\n\n**Q10: Marginal Improvement in Table 2**  \nWe clarify that Table 2 presents ablation studies within AlphaDPO's design space rather than cross-method comparisons. The results demonstrate that:  \ni) Dynamic configurations consistently achieve better performance.  \nii) The AlphaDPO formulation retains optimality.", "claims_and_evidence": "First, the paper lacks justification for its specific choice of loss function. In the introduction (lines 72–73), the authors state that this choice was heuristic, but they do not provide sufficient intuition for it. In the theoretical section, they attempt to ground their method by comparing it to SimPO and TDPO. However, comparing it to SimPO does not justify the specific loss function used in AlphaDPO. Meanwhile, the comparison to TDPO relies on crude approximations, making the justification seem arbitrary. (I will elaborate on this further in the theoretical section of the review.)\n\nIn the experimental section, the paper does not provide enough evidence regarding the significance of the results. Additionally, since AlphaDPO introduces an extra hyperparameter compared to DPO, the paper lacks sufficient detail on how this hyperparameter is chosen. It is also unclear how fairness is ensured when comparing AlphaDPO to DPO, especially if a grid search was used to report AlphaDPO’s best-performing hyperparameter. (I will expand on this further in the experimental section of the review.)", "methods_and_evaluation_criteria": "It largely makes sense. However, there are two important points to consider:\n\nFirst, the models used in the paper have already undergone post-training—specifically, alignment with human preferences using RLHF or Best-of-N. This introduces a potential confounding effect, which could be mitigated by also testing a model that has not been post-trained.\n\nSecond, since the authors state that the closest variant of DPO to AlphaDPO is TDPO, it is essential to include a direct comparison between AlphaDPO and TDPO in the main experimental results, such as in Table 1.", "theoretical_claims": "First, the explanation of $\\gamma$ is unclear. If $U$ is defined as a uniform distribution over all responses given any string, then by definition, $\\gamma = 0$. I find lines 141–142 confusing, as they state that $\\gamma$ is a constant but not zero due to difference in selection probabilities. This point needs to be more precise and formally written.\n\nSecond, the paper claims that AlphaDPO is a general framework encompassing both DPO and SimPO by varying $\\alpha$. However, I do not believe that setting $\\alpha = 1$ reduces AlphaDPO to DPO as claimed, since AlphaDPO includes an additional policy term that is absent in DPO.\n\nThird, regarding the connection between TDPO and AlphaDPO: the authors state that the difference between sequence KL terms is approximated by the log-ratio difference. However, it is unclear how well this approximation holds—are there any bounds? Beyond this crude approximation, there is another key difference between AlphaDPO and TDPO: the use of a stop gradient. The paper does not discuss this, which seems like a significant omission.", "experimental_designs_or_analyses": "First, regarding the choice of $\\alpha$ in the main results (e.g., Table 1), the authors do not explain how this parameter is selected. The common practice is to choose the best value from a set of hyperparameters, but comparing this directly with DPO, which does not have this hyperparameter, creates an unfair comparison. A more equitable approach would be to give DPO the same number of training runs using different random seeds.\n\nSecond, concerning the KL term, the results in Table 1 only provide a partial picture, as they may vary depending on the choice of $\\beta$. A more comprehensive way to compare methods would be through Pareto frontier plots. While I appreciate that the authors include this analysis in Figure 3(c), it is much more limited than the main tables. Moreover, in this figure, the percentage of points that end up on the Pareto frontier does not show a significant difference between SimPO and AlphaDPO, which goes against the claim made regarding the supriority of AlphaDPO over SimPO.\n\nLastly, in some experimental results (e.g., Table 2, Mistral model), the performance gap between the best and second-best methods is very small. To ensure the results are meaningful, further statistical tests are needed to confirm their significance.", "relation_to_broader_scientific_literature": "There are many different variants of DPO, and this paper offers another choice of the margin.", "essential_references_not_discussed": ".", "other_strengths_and_weaknesses": "One interesting insight from this paper is the idea of interpreting the difference in reference log probabilities in the DPO loss function as a margin that might be or might not be very accurate depending on the reference model and the chosen beta.", "other_comments_or_suggestions": "See above.", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers had difficulty reaching an agreement, so I carefully read the paper, the review, and the rebuttal. Overall, the weaknesses the reviewers pointed out can be fixed before the camera-ready deadline (as long as the authors do what they promise they will do in the rebuttal). Therefore, I recommend accepting this paper.
Decision: Accept (poster)