Earlier Tokens Contribute More: Learning Direct Preference Optimization From Temporal Decay Perspective
Authors: Ruichen Shao
Bei Li
Gangao Liu
Yang Chen
ZhouXiang
Jingang Wang
Xunliang Cai
Peng Li
Venue: ICLR 2025 Poster (2025)
Keywords: preference optimization
RLHF
DPO
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Direct Preference Optimization (DPO) has gained attention as an efficient alternative to reinforcement learning from human feedback (RLHF) for aligning large language models (LLMs) with human preferences. Despite its advantages, DPO suffers from a length bias, generating responses longer than those from the reference model. Existing solutions like SimPO and SamPO address this issue but uniformly treat the contribution of rewards across sequences, overlooking temporal dynamics. To this end, we propose an enhanced preference optimization method that incorporates a temporal decay factor controlled by a gamma parameter. This dynamic weighting mechanism adjusts the influence of each reward based on its position in the sequence, prioritizing earlier tokens that are more critical for alignment. By adaptively focusing on more relevant feedback, our approach mitigates overfitting to less pertinent data and remains responsive to evolving human preferences. Experimental results on several benchmarks show that our approach consistently outperforms vanilla DPO by 5.9-8.8 points on AlpacaEval 2 and 3.3-9.7 points on Arena-Hard across different model architectures and sizes. 
Furthermore, additional experiments on mathematical and reasoning benchmarks (MMLU, GSM8K, and MATH) confirm that our method enhances performance without compromising general capabilities. Our codebase would be available at \url{https://github.com/LotuSrc/D2PO}.

Published as a conference paper at ICLR 2025
EARLIER TOKENS CONTRIBUTE MORE: LEARNING DIRECT PREFERENCE OPTIMIZATION FROM TEMPO-
RAL DECAY PERSPECTIVE
Ruichen Shao 1x
Bei Li
Gangao Liu
Yang Chen'
Xiang Zhou'
Jingang Wang
Xunliang Cai
Peng Li2.3
"Miiuunn Inc. Institute of Software, Chinese Academy of Sciences
3 University Chinese Academy of Sciences
(shaoruichen, 1 libei17, chenyangl 08, wangjingang02) (me com [1 Liugangaa2023, Lipeng) (apeengOisscie sisass ac. cn
ABSTRACT
Direct Preference Optimization (DPO) has gained attention as an efficient alter- native to reinforcement learning from human feedback (RLHF) for aligning large language models (LLMs) with human preferences. Despite its advantages, DPO suffers from a length bias, generating responses longer than those from the ref- erence model. Existing solutions like SimPO and SamPO address this issue but uniformly treat the contribution of rewards across sequences, overlooking tem- poral dynamics. To this end, we propose enhanced preference optimization method that incorporates a temporal decay factor controlled by gamma param- eter. This dynamic weighting mechanism adjusts the influence of each reward based on its position in the sequence, prioritizing earlier tokens that are more critical for alignment. By adaptively focusing on more relevant feedback, our ap- proach mitigates overfitting to less pertinent data and remains responsive to evoly- ing human preferences. Experimental results on several benchmarks show that our approach consistently outperforms vanilla DPO by 5.9-8.8 points on AlpacaEval 2 and 3.3-9.7 points on Arena-Hard across different model architectures and sizes. Furthermore, additional experiments mathematical and reasoning benchmarks (MMLU, GSM8K, and MATH) confirm that our method enhances performance without compromising general capabilities. Our codebase would be available https://gitee com Lot.Lr//SS..
INTRODUCTION
Direct Preference Optimization (DPO) (Rafailov al., 2023) has recently emerged as a highly efficient alternative for aligning large language models (LLMs) with human preferences (Askell al., 2021; Ouyang et al., 2022). Unlike reinforcement learning from human feedback (RLHF), which involves training reward model followed by iterative policy updates, DPO reframes the problem as a binary classification task directly over human preference data. Compared to supervised fine-tuning, DPO enables the model not only to learn what is good but also to be aware of what bad. This formulation allows DPO to optimize preference alignment in a single-stage training process, bypassing the complexities of reinforcement learning, such policy sampling extensive hyperparameter tuning. By leveraging an analytical mapping between reward functions and optimal policies, DPO fine-tunes LLMS efficiently and stably, offering superior performance in tasks like sentiment control, summarization, and dialogue generation while reducing computational overhead.
Despite its advantages, DPO suffers from length bias problem, which is caused by the unbalanced length preference due to the non-uniform length distribution of chosen and rejected responses. This leads to generated responses tending to be longer than those of the reference model if the majority of chosen responses are longer than the rejected ones. To address this, SimPO (Meng et al., 2024)
Equual Contributions.
Cooresponning Author.

---

Published as a conference paper at ICLR 2025
introduces more streamlined framework by eliminating the need for reference model. Instead of relying on a pre-trained reference model for comparison, SimPO uses the average log probability of generated sequence the implicit reward signal. This innovation reduces computational complexity and memory usage, making SimPO a more efficient alternative to DPO. However, our experiments have revealed that SimPO suffers from unexpected performance issues when applied to data not generated through self- self-sampling. Similarly, SamPO (Lu et al., 2024) addresses DPO's ' length bias by limiting reward computation to the shorter time- series range between chosen and rejected responses, thereby the refining preference optimization.
Both of these studies, however, treat the contri- bution of each reward across the entire sequence as uniform. We posit that this uniform treatment cannot fully capture the nuances of preference op- timization. Specifically, the temporal dynamics within a sequence may influence the importance of certain tokens or segments over others. To validate this conjecture, we plot the KL divergence between the instruct models and their DPO variants on three widely used open-source models, where the results are shown in Figure 1. We notice that the KI. diver- gence remains larger at earlier tokens but gradually decreases along the positions, which indicates car- lier tokens' distributions are more likely affected by DPO. This observation aligns with the findings of previous studies (Lin al., 2024; Yang et al., 2023) that alignment is more critical for earlier to- kens. This also consistent with the nature of next-token prediction, where an accurate prefix allows subsequent tokens to be generated on a more reliable foundation, thereby improving the overall qual- ity (Edunov et al., 2018). In other words, the uncertainty of carlier tokens is much lower, and the calibration for more recent tokens is higher than that for earlier ones (Wang al., 2020).
3.0
Llamas-Instrrer (8B)
Divergeace
2.5
Gemma2 Instruct (9B)
Mistal-NoMo-Tssuucc (125)
2.0
1.5
1.0
0.5
200
600
1,000 1,200 1,400
Response Position
Figure 1: Visualization of KL divergence of instruct models and their DPO variants. The results include three widely used open-source LLMs: Llama3, Gemma2. and Mistral-NeMo. Observation here indicates earlier tokens con- tribute more during alignment.
Building on this observation, we propose an enhanced version of DPO, namely temporal decay based DPO (short for D²PO that integrates temporal decay factor, controlled by a gamma pa- rameter, to further refine the influence of preference data during training. Our method introduces a dynamic weighting mechanism that modulates the contribution of each reward based on its temporal relevance, allowing the model to prioritize earlier feedback over more recent tokens. To this end, when the coefficient is slightly less than 1, it gradually reduces the influence of more recent rewards, which are inherently dependent on past rewards. Surprisingly, this temporal decay strategy has also been validated in the text-to-image synthesis task (Yang et al., 2024), where they emphasize the ear- lier steps in the reverse chain of the diffusion process. Our work complements theirs by showcasing the effectiveness of this approach in an autoregressive context, particularly in standard RLHF tasks. We provide theoretical analysis on how earlier tokens can contribute more significantly from teken-level Markov Decision Process (MDP) perspective. Further discussion on the differences can be found in Appendix G. By incorporating this adaptive temporal decay mechanism, D'PO not only facilitates earlier tokens to contribute more but also maintains the computational efficiency that makes DPO such a com- pelling approach for preference optimization. Experimental results on several widely used bench- marks, including AlpacaEval2, Arena-Hard and MT-bench, demonstrate the effectiveness on both off-policy and on-policy configurations. For example, in on-policy setups, D'PO outperforms DPO by up 5.9-8.8 performance gains in terms of win rate on AlpacaEEva12 and 3.3-9.7 points on Arena- Hard, respectively. Similarly, in off policy setups, our method also demonstrates performance im- provements. As bonus of this decay mechanism which helps in reducing the overestimation of rewards caused by length bias in preference pairs, our method could be casily extended to reference- free mode, and it also can beat SimPO (Meng et al., 2024) by large margin. Specifically, our best reference- free DPPO model can achieve 62.4 LC win rate on AlpacaEval 2 and 63.6 win rate on Arena Hard, which is competitive with the reference-based model. We also conducted additional experiments mathematical and reasoning benchmarks, such as MMLU, GSMSK, and Math, in- dicating that our method enhances performance without compromising general capabilities.
2

---

Published as a conference paper at ICLR 2025
2
RELATED WORK
2.1 REINFORCEMENT LEARNING FROM HUMAN FEEDBACK(RLHF)
The classical RLHF pipeline (Christiano al., 2017; Ziegler al., 2019; Ouyang al., 2022) consists of two distinct stages: The reward modeling phase and the RL. phase.
Reward Modeling Phase. The reward modeling is a binary classification task. Given a prompt, the comparison pair (y1, 32) is collected by querying the supervised fine-tuning (SFT) model. Then, the preference Vo > yt is labeled by human which is used to train a reward model. Typically, Bradley- Terry model (Bradley & Terry, 1952) which quantifies the likelihood one action being preferred over another is usually used to modeling the preference relations:
exp (I,
p (3/1 1/2 x))
exp (r (z, v1)) + exp (r (z, (r (z, (x.
Reinforcement Learning Phase. With the reward model in place, the second phase involves opti- mizing policy through reinforcement learning, such as proximal policy optimization (PPO) (Schul- man et al., 2017), aiming maximize the learned reward while ensuring the policy remains close to predefined reference policy (Korbak et al., 2022). This optimization is crucial for preventing model drift and maintaining alignment with human judgments, which is typically formulated as:
max Ez- D.ywaa (1z) [ro(x, a)] BEEDD [KL (TO( (- [ree x))]
(2)
9
2.2 DIRECT ALIGNMENT ALGORITTMS (DAAS)
RLHF has become cornerstone in the training of LLMs, facilitating their alignment with human preferences. However, the classical RLHF framework (Ouyang et al., 2022) is characterized by a two-stage training process, which includes reward modeling, and reinforcement learning. This complexity introduces several challenges and limitations, including reward over-optimization (Gao et al., 2022: Dubois et al., 2023; Wang et al., 2024a), training instability (Wu et al., 2023) and efficiency (Wang al., 2024b). Nowadays, DAAS have emerged promising alternative, which can be divided into two major categories based on whether to consider a reference model.
Reference-basee Methods. The reference-based methods in DAAS utilize pre-existing model, often a supervised fine-tuned (SFT) model, as reference point during the optimization process. This reference model serves baseline to which the updated model is compared, ensuring that updates do not deviate excessively from the initial, presumably safe and aligned, model configu- ration. DPO (Rafailov et al., 2023) is the most popular reference-based alignment algorithm and after its appearance, more researchers attempt to modify objective function for better performance. KTO (Ethayarajh et al., 2024) distinguishes itself by its capability train from non-paired prefer- ence data, providing a unique angle on optimization. IPO (Azar et al., 2024) learns directly from preferences without relying on the Bradley-Tene model assumption that assumes that pairwise pref- erences can be substituted with pointwise rewards. R-DPO (Park et al., 2024) is an enhanced deriva- tive of DPO, fortified with an additional regularization term designed mitigate the tendency to exploit length biases, thus ensuring more balanced and diverse response generation.
Reference-free Methods. In contrast to reference-basee methods that depend on a pre-existing model for guidance, reference-free methods forgo the need for such reference. They directly optimize the model parameters in response to human feedback, which can enhance the flexibility of the optimization process. However, this freedom also presents challenges in controlling the extent of updates. CPO (Xu et al., 2024) leverages sequence likelihood in conjunction with an SFT objective. ORPO (Hong et al., 2024) is a novel alignment method that integrates odds ratio-basee penalty into the supervised fine-ture tinettunig process. SimPO (Meng al., 2024) uses an average log probability as an implicit reward and introduces a target reward margin to enhance performance without relying on reference model.
reward signal and is trained
3

---

Published as a conference paper at ICLR 2025
1.0
91
2/
(a) DPO
(b) SimPO
0.6
0.4
Mwe
Y/
0.2
0.0
(c) SamPO
(d) DºPO (Ours)
Figure 2: Illustration of coefficients DPO, SimPO, SamPO, and our Do across various posi- tions. Each box represents coefficient, and the opacity denotes the magnitude, with darker colors indicating higher values. (a) For DPO, the coefficients are uniform across different positions. (b) For SimPO, the coefficients of the chosen Vh and the rejected and [ol], respectively. (c) In SamPO, the coefficients are selected based on the minimum length of yywl and |y (d) Our method introduces a factor to implement coefficient decay, specifically sequence defined by 7¹ (e.g.. 1, 7, ??...., ?")) Here, we use ? 0.9 for a clear visualization.
normlaized by their lengths |yol
METHODOLOGY
3.1
DIRECT PREFERENCE OPTIMIZATION (DPO).
Direct Preference Optimization (DPO) is a pivotal advancement the field of offline preference- based training for language models. Traditional RLHF involves complex, multi-stage process that includes training a reward model to align with human preferences and subsequently optimizing a policy model maximize this reward while staying close the original model's distribution. DPO simplifies this process by reparameterizing the reward function directly in terms of the policy model, eliminating the need for an explicit reward model:
(o(y)x) + (yel(ylz)
r(z, 3) Boo
B log Z(x),
(3)
where Tref denotes the policy model and reference model, respectively. 7(x) is the partition function, and 33 is a hyperparameter to control the deviation from the reference model. Substituting this reward into the Bradley-Terry (BT) ranking objective yields the DPO loss:
To (yw
x) x)
-Blog To (y: x)
Neef (y) xx) )
LDPO (To; Tref) = (x. log
BBlog
(4)
Tref (yw.
response between the current policy model and supervised fine-tuned (SFT) model. This reward is then incorporated into the Bradley-Terry ranking objective to directly optimize the policy model for pref- erence data. The effectiveness of DPO lies in its ability to simplify the preference optimization process, making it more accessible and efficient for practical applications.
DPO operates by formulating an implicit reward using the log ratio of the likelihood
3.2 TEMPORAL DECAY MATTERS IN PREFERENCE LEARNING.
Motivation. Preference learning plays a pivotal role in op- timizing LLMs, especially when leveraging user feedback to align model outputs with human preferences. While methods like DPO (Rafailov et al., 2023) and its successors (Meng al., 2024; Lu al., 2024) have demonstrated significant potential in this domain, they exhibit critical oversight: the uniform treatment
1.0
Illma---rrttict (8R)
Gemmaz Instruct (91)
Probability
Mistral NeMo Inserct (1 (12B))
0.9
0.8
tokens across the sequence. As illus- trated in Figure 2, DPO, SimPO, and SamPO assign identical coefficients to all tokens within the chosen response Yu and the rejected response y/- We argue that optimizing each to- ken equally, without considering their positional importance temporal dependence, is suboptimal for most scenarios.
0
200
400
600
800
1,000
Response Position
Figure 3: Probability against posi-
tions 1000 samples.
4

---

Published as a conference paper at ICLR 2025
Our observations indicate that earlier tokens receive greater optimization during the preference learning process compared to later ones (see Figure ). This suggests that the benefits derived from the alignment phase over SFT are predominantly due to the optimization of initial tokens. Ad- ditionally, when plotting the prediction probability across different response positions in Figure 3, find that more recent tokens have higher probabilities than earlier tokens. This indicates that the model becomes increasingly confident in predicting tokens as it progresses through the sequence, likely due to the accumulating contextual information from previous tokens. However, since the accuracy of these later tokens is already high- reahheig up to 0.9, further improvements are more likely to come from enhancing the accuracy of the earlier tokens. Therefore, natural approach is to focus on improving the accuracy of the prefix: the more accurate the earlier tokens are, the better the overall quality of the sequence will be.
Temporal Decay Mechanism. Inspired by the success of Yang et al. (2024), where earlier steps crucial in the reverse chain of the diffusion denoising process, we propose temporal decay mechanism to highlight the importance of earlier tokens in LLM scenarios. Considering the original DPO loss formulation, the most direct way to prioritize earlier tokens to incorporate a position- dependent coefficient that decays over time. Multiple decay mechanisms can achieve this, including linear, polynomial, step, and cosine decay functions. After evaluating these options, we chose expo- nential decay due to its simplicity and effectiveness.
Exponential decay applies a coefficient that decreases exponentially with the token position, repre- sented as y', where 7 is the decay rate (0 < 7 1) and t is the token's position. This approach provides a smooth and gradual reduction the influence of later tokens, ensuring that earlier tokens have more significant impact on the loss calculation. To this end, we adapt this concept the DPO loss function which defined as:
To 7 Blg
To (y' ref (y
X, w
T
Te (yī (yf
X, Y
CD=Po (To: Tret) = log a
X, y
0= ref
x, y
0=
(5)
In this formulation, the exponential decay factor 7 adjusts the contribution of each token based on its position in the sequence. As is shown in Figure 2 (d), coefficients of each token in D'PO grad- ually decrease along the position (the color from dark to light), placing greater emphasis tokens. This modification aligns the optimization process with the observed pattern of optimization in preference learning, where initial tokens benefit more from the alignment phase. Similar to the derivation of the vanilla DPO, we provide the detailed derivation of D PO in the Appendix E.
carlier
3.3 REFERENCE-FREE IS CONSISTENT WITH ON-POLICY SETUPS.
Reference-based methods often incorporate KL diver- gence constraint to prevent the policy model from deviat- ing significantly from its initial state, which adds compu- tational and memory overhead. In the context of DPO, this Tarlyn (ar) Density
0.12
0.10
GouPolicy Off-Policy
0.08
0.06
0.04
adaptive margin term log Ew(V.. xx) in the pairwise loss function. This term quantifies the refer- ence model's preference difference between less-preferred (n1) and preferred (yw) responses. We note that the DPO loss can be viewed as specific case of contrastive loss, where log Te (y) measures the relevance between the prompt æ and the response y. The adaptive margin ensures that loss values remain moderate, contributing to training stability. However, if the reference model assigns similar probabilities to both yor and yr (i.e., the margin approaches zero), the impact of the reference model diminishes, suggesting that it can be easily excluded.
constraint appears
0.02
100
-50
0
50
100
Ref murgin. [ne(yola)) Fwrly (x)
Figure 4: Reference margin of DPO.
To validate this, we analyze the margin distributions in the UltraFeedback dataset under off-policy and on-policy settings. In the off-policy setting, we use original responses, while in the on-policy setting, responses are regenerated by the same model. As illustrated in Figure 4, the on-policy dataset exhibits smaller variance in margins and average closer zero compared to the off- policy dataset. This indicates a higher proportion of semi-hard samples in the on on-policy data. From this perspective, can discard the KI. divergence constraint under on-policy settings and easily
5

---

Published as a conference paper at ICLR 2025
derive the reference-free free version loss function:
The
23 Σ t=0
LDPo (TT) = log
log No (y' y
x log no (yf x, xxt)))
(6)
4 THEORETICAL ANALYSIS
In this section, we analyze the influence of the discount factor on the performance of our method compared to DPO, Both DPO and our method can be considered token-level MDP that satisfies the Bellman equation. Here, we define the suboptimality as the performance difference between the optimal policy x' and a given policy x under specific discount factors, which has been widely discussed in offlin RI. (Rashidinejad et al., 2021; Jin et al., 2021).
4.1 SUBOPTIMALITY DECOMPOSITION
Definition (Suboptimality). The suboptimality of policy a with respect to the optimal policy n², starting from initial state under discount factor y, is defined as:
SubOpt(T, 8; 7) V (s) V (s);
where (s) = TET EE ('r(s, 8o is the expected return of policy under discount
factor 7 and H is the finite horizon. To analyze the influence of the discount factor 7, we consider the suboptimality of our method evaluated with evaluation discount factor % 1.0. We decom- pose the suboptimality into three terms that separately capture the differences due to the discount factors and the policy discrepancies. In this way,
rewrite the suboptimality as below:
SubOpt(T,, 8; Ye) = Vir (s)
V (s) 7/
V (s)
V (s)
V. (s) (T)( + [V (s)
A2
Ag
This decomposition allows us to separately analyze the impact the discount factors and the policy differences.
4.2 SUBOPTIMALITY ANALYSIS
Theorem ISSuboptimality Upper Bound). Let K" denote the optimal policy, and π be the policy under discount factor 7 E [0, 1). Assume that rewards are bounded such that |r(s, a)l S for all states and actions a, and consider finite horizon H. Then. the suboptimality of π compared to n² when evaluated with an evaluation discount factor T 1.0 satisfies the following upper bound:
1 1-2"
2(1 112
SubOpt(T, 8;76) S 2(H
R + (1 7) Ex-ar"
[TV(T (a [[][s[]]]]]]]]]]]]]))))))])]]]]] R
(9)
The complete proof is included in Appendix F. This upper bound reveals that the suboptimality depends on both the discount factor 7 and the mismatch between π and π.. Specifically, the first
term H 1 T-7
decreases 7 increases, while the second term ( increases. highlighting
a trade-off in the choice of 7. As both terms vary monotonically with the discount factor y but in opposing directions, there exists an optimal value 7" within the interval (0, 1) that balances these effects to minimize the overall suboptimality.
5
EXPERIMENTS
5.1 EXPERIMENTAL SETUPS
Due to page limitations, briefly describe the model setting, training data, and hyperparameters in the following section. Expanded details on evaluation benchmarks and baselines are available in Appendix A.

---

Published as a conference paper at ICLR 2025
Table 1: We report AlpacaEval 2 (Li al., 2023) (denoted by AE2), Arena-Hard (Li et al., 2024) (denoted by AH), and MT-Bench (Zheng et al., 2023) (denoted by MB) results under three settings using standard provided samples. Note that LC and WR denote length-controlled and raw win rate, respectively. We used off-the-shelf models as the SET model. And our judge model is GPT-4-Turbo.
Llama3-Instruc (8B)
Gemma2-Instruch (9B)
Mistral-NeMMo-Isttuur (12B)
Method
AF2
AH
MB
AE2
AH
MB
AE2
MB
(%)
(%)
(%)
G4-
(%)
(%)
(%)
(%)
(%)
G4-T
SFT
39.1
40.1
27.6
7.5
37.6
47.2
44.1
8.3
44.6
47.7
46.5
8.1
DPO
37.4
40.3
27.7
7.7
38.8
48.8
42.5
8.1
44.4
49.3
48.5
8.3
KTO
33.3
38.1
21.0
7.5
39.1
50.0
43.8
8.3
37.4
48.7
35.8
8.2
IPO
42.2
45.7
31.9
7.6
41.0
50.0
48.2
8.0
39.8
48,9
39.8
8.2
SamPO
40.7
43.1
26.1
7.5
39.9
50.1
46.9
8.2
43.5
49.5
50.1
8.1
DºPO (ours)
43.5
43.0
37.0
7.7
45.5
51.0
50.2
8.3
51.3
54.4
51.8
8.4
ORPO
10.6
15.3
6.8
6.3
11.3
21.6
10.2
7.1
9.6
17.0
9.8
6.9
SimPO
0.3*
0.8"
1,4"
1.6"
38.8
50.0
31.6
8.0
46.8
53.3
46.6
8.0
Table 2: Following the setting in Meng et al. (2024), we used the on-policy data to obtain the chosen and rejected and applied stronger reward model. denotes our reference-free version.
Instruct (8B)
Cemmm22-nntruct (9B)
Mistral-NeMooInsttr (12B)
Method
AE2
AH
MB
AE2
AH
MB
AE2
AH
MB
WR (%)
LC (%)
WR (%)
G4-T
WR (%)
(C)))
WR (%)
G+-T
WR (%)
LC (%)
WR (%)
G4-T
SFT
40.1
27.6
7.5
37.6
47.2
44.1
8.3
44.6
47.7
46.5
DPO
46.2
47.6
42.4
7.8
47.0
53.4
56.7
8.4
53.5
53.3
59.0
8.4
KTO
42.4
44.8
32.1
7.7
48.3
53.4
57.1
8.3
48.9
51.9
53.2
8.4
IPO
42.9
46.0
34.5
7.9
50.9
50.0
59.7
8.3
53.6
54.4
59.7
8.4
SamPO
44.4
47.2
35.8
8.0
45.8
55.2
55.2
8.2
51.1
53.0
58.3
8.3
D'PO (ours)
47.4
53.5
47.3
7.8
57.2
59.7
66.4
8.3
57.3
62.1
62.3
8.6
ORPO
37.8
39.3
25.5
7.7
41.9
51.1
45.3
8.2
43.8
47.5
46.0
8.2
SimPO
44.4
50.3
41.9
7.8
54.5
58.4
65.0
8.3
51.3
55.0
61.9
8.3
DOP (ours)
48.0
53,9
46.1
7.7
56.7
60.8
65.7
8.3
58.3
62.4
63.6
8.3
Model Setting, We conducted preference optimization experiments using three model families: Llama3-8B (AI@ Meta, 2024), Gemma2-9B (Team al., 2024) and Mistral- (Jiang et al., 2023). Here, we mainly focused on building our systems upon the instruct models. Thus, we utilized pre- trained instruction tuned models (c.g.. Mieta-llam//Mtta-LL 2BB-Instruct the SFT models.'
-------nttttttt, google/gemma-22 2-9b- 2-9b-
it, and Wiidia/Mistral-NeMM.
Training Data Our experiments were carried out using the UltraFeedback dataset. Specifically, We categorize the preference data into types: 1) off-policy data (original response pairs from the UltraFeedback dataset), and 2) on-policy data generated using the SFT models. Similar to SimPO (Meng et al., 2024), for each prompt æ, we generated responses using the SFT model with sampling temperature of 0.8. To validate these responses, we employed RLHFlow/ArmoR Llama3-8B-ve (Wang al., 2024c) to assign scores to each response, allowing us select the highest-scoring response yw and the lowest-scoring one as
Hyperparameters For all models, we set the maximum response length to 2,048 tokens and used batch size of 128. Optimization was performed using the AdamW optimizer (Kingma & Ba, 2015) with a learning rate of 5e and a cosine learning rate schedule featuring 10% warmup period. In preference optimization methods, including DPO and its variants such our method DºPO and SamPO, we set B to 0.1 to ensure a fair comparison.
'The exact nature of the instruction-tuning (whether it includes SFT or the complete RLHF pipeline) of these models is not fully disclosed. For simplicity, we refer to these SFT models.

---

Published as a conference paper at ICLR 2025
Llama33 Instract (8B)
Mistral Instrrnstruuc (12B)
Instruct
Gemma2 Instruct (913)
56.5 03.0
47.0 57.5 5 68.0
LC Win (%)
Win Rate (%)
Soure (%)
8,2
47.5
26.0 36.5
7.9
38.0
0.97 0.98 0.99
1.00 1.01 1.02
0.97 0.98 0.99
1,00 1.01 1.02
0.97 0.98 0.99 1.00 1.01 1.02
7 (AlpacaEvval2)
7 (Arema Hard)
(MT-Bench)
Figure 5: Performance against different choices of three open-source source models on three benchmarks.
5.2 EXPERIMENTAL RESULTS
In our experiments, provide comprehensive comparison of our proposed method against DPO and its variants on both off-policy and on-policy data respectively, along with the haselines intro- duced in Section 5.1. The baselines are categorized into two broad paradigms: reference-based and reference-free free (SimPO and ORPO). Notably, as discussed in Section 3.3, our method can be seam- lessly integrated into the reference- paradigm using on-policy data. We ensure fair comparisons by maintaining consistency in the codebase and experimental settings across all methods evaluated.
Off-policy Setups. Table 1 clearly demonstrates that our method delivers significant improve- ments in win rates across all configurations. Specifically, when applied to the Llama3, our method outperforms DPO by margins of 6. and 2.9% in standard and length-controlled evaluation sce- narios, respectively. Similarly, for the Mistral-Nehe -eMo model, our method surpasses DPO by mar- gins of 6.9% and 5.1 1.1% in standard and length-controlled scenarios, respectively. We observed that reference- feerennceeeeeeeeeeeee methods exhibited instability when applied to off-policy data, often leading to a degra- dation in model performance. This issue is particularly evident with SimPO, where previous work observed similar findings (Lu et al., 2024). This phenomenon highlights the challenges associated with reference-free methods in preference optimization on off- off-policy data.
On-policy Setups. As shown in Table 2, our proposed method, along with all baselines, achieves better results compared to off-policy settings. Notably, our method consistently demonstrates im- provements across different setups. Due to the reward model's length preference when selecting on-policy data, models trained on this data are more prone to verbosity. A critical observation in standard evaluations is the inherent bias favoring models that generate longer responses, which tend to achieve higher win rates. However, our method not only achieves superior win rates but also produces significantly shorter responses, showcasing its efficiency in generating concise and rele- vant outputs. Additionally, when the reference model is omitted, our method outperforms SimPO by 2.4-7.4 in LC win rate and 0.7-4.2 in win on AlpacaEval 2 and Arena-Hard, respectively. These findings further underscore the robustness and effectiveness of our approach. This superiority in both reference-bast and reference-fie contexts emphasizes the versatility and reliability of our method in preference optimization.
6 ANALYSES
Plays an Important Role, The temporal decay is one of the main contributions of this work, and we would like to show how the 7 affects the performance. We conducted ablation studies on three open-source models for robust conclusions. Through results as shown in Figure 5, we see that nearly all variants with y lower than 1.0 consistently outperform DPO Also, 7 = 0.98 achieves the highest performance across three benchmarks for these strong open-source models. This indicates that our method is robust to the choice of y, reducing the need for extensive hyperparameter tuning.
Larger than 1 is Harmful. As highlighted in the previous section, we prioritize earlier feedback over more recent feedback, aligning with the next-token prediction paradigm. We conducted an experiment where the decay factor y was set to slightly greater than 1.0 to observe the effects. The
PDPO is special case of ours where 7 equals 1.0.
8

---

Published as a conference paper at ICLR 2025
Table 3: Results on OpenLLM Benchmark, including reasoning and mathematical testsets. Note
that Hella. denotes Hellaswag, Truth. denotes TruthfulQA and Wino. denotes Winogrande.
MMLU
GSM8K
Math
IFEval
ARC-C
Hella.
Truth.
Wino.
Method
0-shot
O-shot
O-shot
O-shot
25-shot
10-shot
O-shot
5-shot
(a)
Llama3- (8B)
61.7
7.9
68.6
62.0
78.8
51.6
75.5
DPO
56.7
70.5
7.8
65.1
65.1
79.9
56.4
74.5
SimPO
55.2
57.5
5.3
60.8
67.6
78.8
63.8
74.3
D*PO (ours)
61,4
72.0
8.5
65.6
65.8
79.0
57.6
75.1
(b)
Srrmma2-Instuuee (9B)
SFT
72.8
87.4
19.4
71.9
71.8
81.7
60.2
77.9
DPO
72.2
88.5
19.4
60.1
69.9
71.5
57.7
72.7
SimPO
72.4
88.2
19.0
71.5
68.3
66.5
58.9
73.7
DDPO (ours)
72.7
21.2
71.2
71.4
81.0
61.3
76.0
results could also be observed in Figure 5. When "y exceeds 1.0, rewards linked later tokens in the sequence receive larger coefficients than those for earlier tokens. However, this adjustment was detrimental to preference optimization, resulting in performance that lagged behind the standard DPO on both the AlpacaEvval 2 and Arena-Hard benchmarks. This finding demonstrates the crucial role of earlier tokens in the alignment process and indicates that overemphasizing later tokens can degrade model performance.
Evaluations on OpenLLM Benchmark. To verify whether the improvements of DºPO on the aforementioned RLHF benchmarks, such as Alpaca Eval2, Arena Hard, and MT-bench, come at the expense of general language modeling ability, we conducted comprehensive evaluation of downstream tasks the Open LLM leaderboard Specifically, we employed zero-shot evalua- tions on MMLU (Hendrycks et al., 2021), GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021), IFEval (Zhou al., 2023), and TruthfulQA (Lin al., 2022). Additionally, we performed few-shot evaluations on ARC-C (Clark et al., 2018), Hellaswag (Zellers et al., 2019), and Wino- grande (Levesque al., 2012) according the official settings in the Open LLM leaderboard. The results are summarized in Table 3, and we observe that:
In the Llama3-8B configuration, our D2PO method significantly outperforms both DPO and SimPO, particularly on the MMLU and MATH benchmarks. Notably, D'POO exhibits less perfor- mance degradation on GSM8K compared to SimPO, despite both methods effectively controlling output length. PO achieves substantial performance gains on the MATH dataset, surpassing the Instruct model by 0.55 points, while the other two methods show noticeable decline.
In the Gemma2- 9B configuration, we observe similar pattern, with DOPO demonstrating a sig- nificant performance advantage on the MATH benchmark. These results suggest that PO effec- tively enhances reasoning and mathematical problem-solving abilities in LLMs across different models. Furthermore, these additional evaluations on specialized datasets confirm that DºPO maintains its effectiveness across various contexts and task types.
Comparisons of Various Decay Strategies We have proven the importance of temporal decay. Following the classic Markov Decision Process, we use exponential decay as our default decay strategy. Meanwhile, We also consider several variants of decay strategies, including Head decay, Linear decay and Power-Law decay. The detailed decay mechanism are summarized in Table 4. We observe 1-0 decay and Linear Deacy show inferior results the tenporal decay, and even underper- forms with the vanilla DPO. While though Power-Law method also shows promising results, but it cannot properly control the response length competitive results with exponential decay.
Lengthy Debias. Previous studies (Park et al., 2024; Lu et al., 2024; Meng et al., 2024) have demonstrated that DPO is susceptible length exploitation, it tends amplify verbosity biases present in the preference datasets. This propensity can lead to suboptimal outcomes where the
Open LIM leaderboard is created by huggingface to provide standardized evaluation setup for I.IMs, which includes several popular benchmarks encompassing wide range of capabilities multiple domains.

---

Published as a conference paper at ICLR 2025
Table 4: Comparison of different decay mechanisms in terms of performance and response length.
A
AH
MB
Decay Strategy
Rewards
WR (%)
LC (%)
Len.
WR (%)
Len.
G4-T
Exponential
fog Polye x.ycc) Prey (rllxce))
57.2
59.7
1950
66.4
724
8,3
Head
B Blog po(yilx.Vet) Pref (r
48.6
54.7
1762
57.4
680
8.2
1=0
7T Σ
Linear
log po(sel)
48.3
1713
59.4
661
Power-Law
B log polyelxyycc) Prof (iiloa))))))
56.8
57.7
2011
71.2
823
8.5
87
model's decisions are disproportionately influenced by the length of the responses rather than their quality relevance. To investigate the relationship between the length bias of training data and the output length of the model, we visualized the DPO and DdPO loss of 1000 random samples based on the length gap between the chosen and rejected responses. For simplicity, verbosity- biased data refers to pairs in which the chosen response must be longer than the reject response and brevity- biased data refers to the opposite type of data.
From Figure 6, we can see that during the DPO train- ing process, the loss of verbosity-biased data is large, while the loss of brevity-biased data is small. Conse- quently, DPO prioritizes the optimization verbosity- biased data, increasing likelihood of longer chosen sponses and decreasing likelihood shorter ones. This kind of imbalance loss can easily cause model verbosity. Meanwhile, DOPO reduce the loss imbalance between verbosity-biasse data and brevity-biased data, thereby controlling the output length of the model.
25-
DÝPO
2.9
DPO
Loss
1.5
L.a
0.5
0.0
2000
Mal
Figure 6: Loss vs. length diff.
further validate our results, we conducted human evaluations the AlpacaEval2 and Arena- Hard datasets using the Gemma2-9B model. We enlisted four evaluators, with each person evaluating 50 samples for each benchmark. For each instruction. we randomized the order of the outputs from DPO and DOPO to prevent bias. The evalu- ators assessed the responses based on three criteria: accuracy. completeness, and relevance, determining which response better for each sample. If both sponses were equally correct or incorrect, the result was considered a tie. As shown in Table 5, our comparison between D'PO and DPO indicates that D'PO achieved a significantly higher win rate than DPO, with an overall win rate of 67% in Arena Hard and 69% in AlpacaEval 2 (calculated as (win + tie/2) / total).
Human Evaluations
Table 5: Human evaluation results on two benchmarks.
Benchmark
Win Tie Lose
AlpacaEEval 2 116 36 48
Arena-Hard
CONCLUSIONS
In this work, we revisited the loss objectives of DPO and its variants, introducing a temporal decay mechanism governed by parameter 7. Motivated by the observation that earlier tokens contribute more significantly during preference optimization, our dynamic weighting scheme prioritizes these initial tokens, aligning naturally with the next-token prediction paradigm. Extensive experiments demonstrate that approach consistently outperforms vanilla DPO, achieving notable improve- ments across diverse benchmarks and model architectures. By enabling DPO to focus more on short-term rewards while retaining its simplicity and stability, method offers compelling lution for preference- basedd fine-tuning of large-scale models. Furthermore, we showed that our method can be extended reference-free, on-policy setting, outperforming existing approaches.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H. Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan M. Cohen, Sirshak Das, Ayush Dattagupta, Olivier Delalleau, Leon Derczynski, Yi Dong, Daniel Egert, Ellie Evans, Aleksander Ficek, Denys Fridman, Shaona Ghosh, Boris Ginsburg, Igor Gitman, Tomasz Grzegorzek, Robert Hero, Jining Huang. Vibhu Jawa, Joseph Jennings, Aastha Jhunjhunwala, John Kamalu, Sadaf Khan, Oleksii Kuchaiev, Patrick LeGresley, Hui Li, Jiwei Liu, Zihan Liu, Eileen Long, Ameya Sunil Maha- baleshwarkar, Somshubra Majumdar, James Maki, Miguel Martinez, Maer Rodrigues de Melo, Ivan Moshkov, Deepak Narayanan, Sean Narenthiran, Jesus Navarro, Phong Nguyen, Osvald Nit- ski, Vahid Noroozi, Guruprasad Nutheti, Christopher Parisien, Jupinder Parmar, Mostofa Patwary, Krzysztof Pawelec, Wei Ping. Shrimai Prabhumoye, Rajarshi Roy, Trisha Saar, Vasanth Rao Naik Sabavat, Sanjeev Satheesh, Jane Polak Scoweroft, Jason Sewall, Pavel Shamis, Gerald Shen, Mo- hammad Shoeybi, Dave Sizer, Misha Smelyanskiy, Felipe Soares, Makesh Narsimhan Sreedhar, Dan Su, Sandeep Subramanian, Shengyang Sun, Shubham Toshniwal, Hao Wang, Zhilin Wang, Jiaxuan You, Jiaqi Zeng, Jimmy Zhang, Jing Zhang. Vivienne Zhang, Yian Zhang, and Chen Zhu. Nemotron-4 340b technical report. CoRR, 0bs/2400.111744 2024.
AetMeta. Llama 3 model card. 2024.
URL https: http//ggtttttttttt com/meta 1lama/
1lama3/blob/ http//maannmmmmm// CARD md.
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Benjamin Mann, Nova DasSarma, Nelson Elhage. Zac Hatfield-Dodds, Danny Hernandez, John Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Christopher Olah, and Jared Kaplan. general language assistant as a laboratory for alignment. ArXiv, abs/2 12.00861, 2021.
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Rémi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learn- ing from human preferences. In International Conference on Artificial Intelligence and Statistics, 2-4 May 2024, Palau de Congressos, Valencia, Spain, volume 238, pp. 4447-4455, PMLR, 2024.
Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39:324, 1952.
Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic. Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Proc. of NeurlPS, pp. 4299-4307, 2017.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try ARC, the A12 reasoning chal- lenge. ArXiv, abs/1803.005577 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek. Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, abs/2110. 14168, 2021.
Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacafarm: simulation framework for methods that learn from human feedback. In Proc. of NeurlPS, 2023.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori Hashimoto. Length-comtrroled Al- pacaEval: A simple way to debias automatic evaluators. ArXiv, abs/2404.04475. 2024.
Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. Understanding taecstrraslttion at scale. In Ellen Riloff, David Chiang, Julia Hockenmaier. and Jun'ichi Tsujii (eds.), Proc. of EMNLP, pp. 489-500, 2018.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Model align- ment as prospect theoretic optimization. In Proc. of ICML, 2024.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In Proc. of ICLR, 2022.
11

---

Published as a conference paper at ICLR 2025
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In Proc. of ICLR, 2021.
Jiwoo Hong, Noah Lee, and James Thome. ORPO:
monolithic preference optimization without ref- erence model. In Yaser Al- Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proc. of EMNLP, pp. 11170-11189 Association for Computational Linguistics, 2024.
Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lu- cile Saulnier, L'clio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7B. ArXiv, abs/2310.0682. 2023.
Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline ri? In Proc. of ICML, pp. 5084 5096, 2021.
Sham M. Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Proc. of ICML, pp. 267-274, 2002.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), Proc. of ICLR, 2015.
Tomasz Korbak, Ethan Perez, and Christopher L. Buckley. with penalties better viewed as bayesian inference. In Proc. of EMNLP, 2022.
Hector J. Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Gerhard Brewka, Thomas Fiter, and Sheila A. Mcllraith (eds.), Principles of Knowledge Rep- resentation and Reasoning: Proceedings of the Thirteenth International Conference, KR 2012, Rome, Italy, June 10-14, 2012, AAAI Press, 2012.
Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. From live data to high-quality benchmarks: The Arena-Hard pipeline, April 2024. URL https:/ / /tsss/msss///////////ssss////sss htrg/bllggg //log/2200 2024 04 ---aarena- hard/.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. AlpacaEval: An automatic evaluator of instruction-follooinng 2023.
models. https:/ /ttps:///ittiii http://gtbb....tttttsss//so/eett www./totoouu/oooooo/oo.oooootts. http://ttttullaba/aaa a httpatttsuuaaaaaaaaaaaaeeeeeee
Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Raghavi Chandu, Chandra Bhagavatula, and Yejin Choi. The unlocking spell on base llms: Rethinking alignment via in-context learning. In Proc. of ICLR, 2024.
Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Proc. of ACI, pp. 3214-32522 2022.
Junru Lu, Jiazheng Li, Siyu An, Meng Zhao, Yulan He, Di Yin, and Xing Sun. Eliminating biased length reliance of direct preference optimization via down-sampled KI. divergence. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proc. of EMNLP, pp. 1047-1067, 2024.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-free reward. In Proc. of NeurlPS, 2024.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. Training language models to follow instructions with human feedback. In Proc. of NewrIPS, 2022.
Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization. In Lun-Wet Ku, Andre Martins, and Vivek Srikumar (eds.), Proc. of ACL Findings, pp. 4998-5017, 2024.
12

---

Published as a conference paper at ICLR 2025
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Proc. of NeurlPS, 2023.
Rafael Rafailov, Jocy Hejna, Ryan Park. and Chelsea Finn. From to q Your language model is secretly a q- q-function. In Proc. of COLM, 2024.
Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and Stuart Russell. Bridging offline reinforcement learning and imitation learning: A tale of pessimism. In Proc. of NeurlPS, pp- 11702-11716, 2021.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1 0030770,.647,... 2017.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Fer- ret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Char- line Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piotr Stanczyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Jean- BasttBn Grill, Behnam Neyshabur, Olivier Bachem, Alanna Walton, Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchi- son, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Perry, Chris Welty, Christopher A. Choquette-Choo, Danila Sinopalnikov, David Wein- berger, Dimple Vijaykumar, Dominika Rogozińskka, Dustin Herbison, Elisa Bandy, Emma Wang. Eric Noland, Erica Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Klimezak -luuciikkk. Harleen Batra, Harsh Dhand, Ivan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Peng Zhou, Joana Carrasqueira, Joana Iljazi, Jocelyn Becker, Joe Fernandez, Joost van Amersfoort, Josh Gordon, Josh Lipschultz, Josh Newlan, Ju yeong Ji, Kareem Mo- hamed, Kartikeya Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen, Kiranbir Sodhia, Kish Greene, Lars Lowe Sjoesund, Lauren Usui, Laurent Sifre, Lena Heuermann, Leti- cia Lago. Lilly McNealus, Livio Baldini Soares, Logan Kilpatrick, Lucas Dixon, Luciano Mar- tins, Machel Reid, Manvinder Singh, Mark Iverson, Martin Görner, Mat Velloso, Mateo Wirth, Matt Davidow, Matt Miller, Matthew Rahtz, Matthew Watson, Meg Risdal, Mehran Kazemi, Michael Moynihan, Ming Zhang, Minsuk Kahng, Minwoo Park, Mofi Rahman, Mohit Khat- wani, Natalie Dao, Nenshad Bardoliwalla, Nesh Devanathan, Neta Dumai, Nilay Chauhan, Os- Wahltinez, Pankil Botarda, Parker Barnes, Paul Barham, Paul Michel, Pengchong Jin, Petko Georgiev, Phil Culliton, Pradeep Kuppala, Ramona Comanescu, Ramona Merhej, Reena Jana, Reza Ardeshir Rokni, Rishabh Agarwal, Ryan Mullins. Samanch Saadat, Sara Mc Carthy, Sarah Perrin, Sébastien M. R. Arnold, Sebastian Krause, Shengyang Dai, Shruti Garg, Shruti Sheth, Sue Ronstrom, Susan Chan, Timothy Jordan, Ting Yu, Tom Eccles, Tom Hennigan, Tomas Ko- cisky, Tulsee Doshi, Vihan Jain, Vikas Yadav, Vilobh Meshram, Vishal Dharmadhikari, Warren Barkley, Wei Wei, Wenming Ye, Woohyun Han, Woosuk Kwon, Xiang Xu, Zhe Shen, Zhitao Gong, Zichuan Wei, Vietor Cotruta, Phoebe Kirk, Anand Rao, Minh Giang, Ludovic Peran, Tris Warkentin, Eli Collins. Joelle Barral, Zoubin Ghahramani, Raia Hadsell, D. Sculley, Jeanine Banks, Anca Dragan, Slav Petrov, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Flena Buchatskaya, Sehastian Borgeaud, Noah Fiedel, Armand Joulin, Kathleen Kenealy, Robert Dadashi, and Alek Andreev. Gemma 2: Improving open language models at 00118.
practical size, 2024. URL https: htattss//axxxi.oot/ org www.oorgg/bbb/eeeeee
Chenglong Wang. Hang Zhou, Kaiyan Chang, Bei Li, Yongyu Mu, Tong Xiao. Tongran Liu, and JingBo Zhu. Hybrid alignment training for large language models. In Proc. of ACL Findings, pp. 11389-11403, 2024a.
Chenglong Wang, Hang Zhou, Yimin Hu, Yifu Huo, Bei Li, Tongran Liu, Tong Xiao, and Jingbo Zhu. ESRL: efficient sampling-basee reinforcement learning for sequence generation. In Proc. of AAAI, 2024b.
Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of ofaerre-f--xpeeee of-experss. In Proc. of EMNLP Findings, pp. 10582-10592, 2024c.
13

---

Published as a conference paper at ICLR 2025
Shuo Wang. Zhaopeng Tu, Shuming Shi, and Yang Liu. On the inference calibration of neural machine translation. In Proc. of ACL. pp. 3070-3079, 2020.
Tianhao Wu, Banghua Zhu, Ruoyu Zhang, Zhaojin Wen, Kannan Ramchandran, and Jiantao Jiao. Pairwise proximal policy optimization: Harnessing relative feedback for Ilm alignment. ArXiv, abs/2310.00212, 2023.
Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of LLM performance in machine translation. In Proc. of ICML, 2024.
Shentao Yang, Shujian Zhang, Congying Xia, Yihao Feng, Caiming Xiong, and Mingyuan Zhou. Preference-grounder token-level guidance for language model fine-tuning. In Proc. of NeurlPS, 2023.
Shentao Yang, Tianqi Chen, and Mingyuan Zhou. A dense reward view on aligning text-to-image diffiusio with preference. In Proc. of ICML, 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? In Anna Korhonen, David R. Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pp. 4791-4800. Association for Computational Linguistics, 2019.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging LLM-as-aajudge with MT-Bench and Chatbot Arena. In Proc. of NeurIPS Datasets and Benchmarks Track, 2023.
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhanghhi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proc. of ACL (Volume 3: System Demonstrations), 2024.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Luaa, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. CoRR, abs/2311.079 1,09111, 2023.
Brian D. Ziebart. Modeling Purposeful Adaptive Behavior with the Principle of Maximum Causal Entropy. PhD thesis, Carnegie Mellon University, USA, 2010.
Brian D. Ziebart, Andrew L. Maas, J. Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforcement learning. In Proc. of AAAI, pp. 1433-1438, 2008
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint Xxx:v:9909 190000893, 2019.
14

---

Published as a conference paper at ICLR 2025
A EXPERIMENTAL SETUPS
Evaluation Benchmarks. We primarily evaluated our models using three widely used open-ended instruction-following benchmarks: MT-Bench (Zheng et al., 2023), AlpacaEval (Li et al., 2023)*, and Arena-Hard v0.1 (Li et al., 2024). These benchmarks assess the models' versatile conversa- tional capabilities across diverse of queries and are widely adopted by the research com- munity. Concretely, AlpacaEval comprises 805 questions from 5 datasets, while MT-Bench encompasses 8 categories with 80 questions. Arena-Hard, enhanced version of MT-Bench', includes 500 rigorously defined technical problem-solving queries. For AlpacaEEval 2, we used alpaca_eval.gpt 4.turbo_fr as the annotator which has higher human agreement and re- port both the raw win rate (WR) and the length-controleed win rate (LC) (Dubois et al., 2024), with the LC metric designed be robust against model verbosity. For Arena-Hard, we reported the WR against the baseline model. For MT-Bench, we report the average M'T-Bench score, using GPT-4-Turbo-202 049-09 as the judge mode!".
Baselines. We selected several advanced preference optimization baselines, including: IPO (Azar al., 2024) is theoretically grounded method that avoids DPO's ss assumption that pairwise prefer- ences can be replaced with pointwise rewards. KTO (Ethayarajh et al., 2024) learns from non-paired preference data. ORPO (Hong al., 2024) introduces reference-model-iee rectly contrast winning and losing responses with the policy model, jointly training with the SFT objective. SimPO (Meng al., 2024) and SamPO (Lu al., 2024) are both designed to address the issue of model verbosity by applying length normalization. We meticulously tune the hyperparam- eters for each baseline and report the best performance.
odds ratio term di-
AlpacaEval Annotator Choice AlpacaEval 2 provides various evaluation templates and in the official readme recommends using awpaca_eval.....maaaaaaaa www.aaptaallla...eeeeeeeee
4.turb well as The former is the default annotator in AlpacaEval 2 with a human agreement rate of 65.7% and much cheaper price. In all of our evaluations, we used the latter as the annotator which has a higher agreement rate of 68.1 % with human annotation data.
B EXPERIMENTAL DETAILS
Considering DPO is special case of DºPO when 7 1.0, to compare the effects of different decay coefficients, we conduct experiments with / fixed at 0.1. In addition, we follow the optimal hyperparameters claimed in SimPO and our code is built on LlamaFactory (Zheng al., 2024). Across all DAAs run, the models were trained on 32 A A100 with a global batch size of 128 (4 gradient accumulation steps). The hyperparameter search range of all methods are displayed in the Table 6.
C MORE ANALYSES
When to Decay In our default setting, we apply decay from the beginning of the prompt rather than from the first generated tokens. Here, we investigate the implications of these two approaches. Theoretically, during the loss computation, both the chosen token Yu and the rejected token share the same prompt prefix, which results in distinct initial scaling coefficients for the reward at the first generated position. For illustration, if the prompt length is I, then in our default setting, the reward for the first generated token is scaled by y while in the alternate setting, the scaling factor would be 1. Through results in Figure 7, we can see that both two settings achieve better results than DPO, and our default setting is much better than the other one. This indicates that proper scaling factor is very important during preference optimization.
Effect of Different Judge Models Here, we mainly evaluated our generated results via GPT-4-Turbo 0409, while previous work mainly used GPT-4 wwwviewweile instead. Re- sults in Table 7 show that DOPO delivers consistent performance gains in both two judge models.
'https: / tatttttssslaaa ll.. gi thub. io/a ttt..io//ppaacaeeeeee//
Adler al. (2024) discussed the existence of incorrect reference answers in MT-Bench, therefore cor- rected version of MT-Bench was used.
"GPT-4-Turbee 2024-04 provides more accurate reference answers and judgments compared to
GPT-4.
15

---

Published as a conference paper at ICLR 2025
66.4
Decay from
62.1
Decay from an
1.C Win Rate (S6)
Docay from prompt
59.7
59.3
Win (%)
Docay from pibinjs
60.5
62.3
55.3
33.5
17.3
52.3
42.8
Gemma2
Mistral-NeMa
IJama3
Mistral-NeMo
(a) AlpacaEval 2
(b) Arena- Arena-Hard
Figure 7: Visualization of the LC Win Rate (%) three models under different decay mechanisms (answer-based and prompt-based decay) in the AlpacaEval and Arena-Hard benchmarks.
Table 6: Various preference optimization objectives and hyperparameter search range.
Method
Objective
Hyperparameter
DPO (Rafailov et al., 2023)
log (3 log Treliiale
log Malauln)
g e [0.01,0.1]
IPO (Azar al., 2024)
log Forliuu i2]
Tafurle) log mulluiss
[0.01, 0.1, 1.0
Aua
(elisii)) (9 log TeelSwin))
Zur) + Air log relagle) Trelaaike)
= A. 1.0
KTO (Ethuyurajh et al., 2024)
where
Bir [AKL (ITe (To(vll))]iaa((())))
8 [0.01,0.05, 0.1]
SamPO 2024)
log a
8 log vols
glog (elle))
rolsi mo5t)
B [0.01,0.1]
(ycly(( where Te (Tin(T.... Ti).y' v' Uniform(T..
1--0
Ing (m log (lng polvals)
log paiscle)
ORPO (Hong et al., 2024)
A [0.1, 1.0,
where po(p) = exp log ma(la)))))
SimPO (Meng al., 2024)
log [
log mo(ml)))))> 1ii log mo(oil=))>)
8 [2.5, 10]
7 [0.3, 0.5, 1.0]
D²PO
to6 xx51) m..r laos")
E 7 (n
3 [0.1]
log
Σ
B' log
y [0.95, 0.97, 0.98
Table 7:
Three benchmarks results with on-policy setups, using spt-4-1106-previeet the judge denotes our reference-free version.
model. i
Llama3-Instruct (8B) Gemma2-Inntrue (9B) Mistral-NeMMo-Insruuc (12B)
Method
AE2
AH
AE2
AH
AE2
AH
WR(%) (%)
LC (C))
WR
WR (%)
LC (%)
WR (%)
WR (S) (C(%)
WR (%)
SFT
31.6
31.7
19.7
37.7
48.2
39.9
40.8
44.2
39.7
DPO
41.7
42.9
31.2
46.4
53.1
47.4
53.4
52.6
47.4
KTO
35.7
37.7
25.2
46.5
52.3
49.2
45.9
49.7
45.4
IPO
40.1
43.2
25.2
49.1
48.3
49.5
51.7
52.9
51.8
SamPO
39.4
41.9
28.7
46.9
56.7
50.4
49.8
52.5
50.4
D'PO (ours)
44.5
50.1
34.1
59.3
62.3
58.4
55.0
60.6
50.6
ORPO
31.5
32.5
20.9
39.8
48.2
41.3
40.1
41.2
SimPO
44.5
49.1
33.1
55.1
59.4
56.5
50.7
53.8
51.0
DOPO" (ours)
45.0
51.9
33.0
58.0
61.5
56.9
56.8
59.9
49.4
Full evaluation results We present the full evaluation of AlpacaEval 2, Arena-Hard and MT- Bench in Table 9 and Table 10. The former is an off policy setup, while the latter is an on-policy setup. For on-policy setups. found that DPO achieve better performance when beta 0.01 and reported this result for fair comparison. Specifically, s." indicates that the model suffered a collapse during training.
16

---

Published as a conference paper at ICLR 2025
Table 9: Full results on benchmarks under off-policy setups.
AlpacaEval 2
Arena Hard
MT-Bench
Method
Win Rate (%)
LC Win Rate (%)
Len.
Win Rate (%)
Len.
G4-Turbo
Llama3-Instrus (8B)
SFT
39.05
40.13
1971
27.6
581
7.5
DPO
37.38
40.28
1880
27.7
546
7.7
KTO
33.29
38.06
1765
21.0
525
7.5
IPO
42.16
45.66
1845
31.9
542
7.6
SamPO
40.68
43.11
1891
26.1
550
7.5
D'PO (y=0.95)
45,90
44.78
2113
40.0
636
8.0
DºPO (7=0.97)
43.46
43.04
1994
37.0
602
7.7
DPPO (y=0.98)
42.73
44.10
1954
35.1
578
7.9
DPPO (7=0.99)
41.74
44.03
1912
30.4
560
8.0
DPPO (y=1.01)
38,50
40.21
1928
26.1
569
7.5
DºPO (g=1.02)
37.69
38.63
1955
26.8
572
7.4
ORPO
10.62
15.32
1386
6.8
764
6,3
SimPO
0.25
0.80
27
1.4
15
1.6
Demma2-Instrure (9B)
SFT
37.58
47.23
1.566
44.1
608
8.3
DPO
38.81
48.83
1546
42.5
595
8.1
KTO
39.07
50.00
1530
43.8
540
8.3
IPO
41.04
50.03
1630
48.2
8.1
SamPO
39.86
50.06
1574
46.9
596
8.2
DPPO (1=0.95)
48.07
50.05
1929
53.4
657
8.5
D`PO (n=0.97)
45.34
49.70
1824
50.7
636
8.3
D'PO (1=0.98)
45.46
50.99
1746
50.2
625
8.3
D'PO (7=0.99)
42.10
50.05
1636
50.2
8,4
D'PO (y=1.01)
38.57
47.45
1577
43.7
612
8.3
DPPO (y=1.02)
ORPO
11.30
21.55
1182
10.2
641
7.1
SimPO
38.76
50.00
1508
31.6
475
8.0
Nertraa--eMo--nnsturu (12B)
SFT
44.60
47.71
1879
46.5
575
8.1
DPO
44.41
49.25
1821
48.5
569
8.3
KTO
37.39
48.68
1620
35.8
8.2
IPO
39.75
48.85
1634
39.8
506
8.2
SamPO
43.54
49.47
1784
50.1
562
8.1
D'PO (n=0.95)
52.17
52.42
2017
54.2
590
8.3
D'PO (7=0.97)
51.30
54.43
1879
51.8
562
8.4
D'PO (7=0.98)
49.57
55.43
1778
47.8
534
8.3
DPPO (7=0.99)
46.96
53.86
L770
45.9
538
8.0
DPPO (1=1.01)
43.65
47.60
1829
46.8
564
8.1
D'PO (7=1.02)
43.84
47.91
1840
45.6
572
8.0
ORPO SimPO
9.64
17.00
1185
9.8
640
6.9
46.77
53.28
1704
46.6
500
8.0
Arena-Hard Style Control Evaluation We have conducted evaluations using the Arena- Hard benchmark, focusing on style control ca- pabilities based Gemma2-9B. Through re- sults in Table 8, we can see that D'PO con- sistently achieves superior performance, when style control is a key factor, highlighting our method's effectiveness in style-controlled settings.
Table 8: Style Control evaluation the Arena- Hard benchmark.
Model
AH AH (Style-Control)
DºPO
66.4
67.2
DPO (6 0.1)
56.7
57.2
DPO (8 0.01)
65.2
66.4
SimPO
65.0
66.3
17

---

Published as a conference paper at ICLR 2025
Table 10: Full results on benchmarks under on-policy setups. denotes our reference-free version.
AlpacaEval2
Arena Hard
MT-Bench
Method
Win Rate (%)
LC Win Rate (%)
Len.
Win Rate (%)
Len.
G4-Turbe
Llama3-Instruct (8B)
SFT
39.05
40.13
1971
27.6
581
7.5
DPO (8=0.01)
48.26
49.93
1937
45,2
568
7.8
KTO
42.36
44.77
1901
32.1
545
7.7
IPO
42.92
45.99
1889
34.5
553
7.9
SamPO
44.35
47.17
1890
35.8
DPPO (y=0.95)
48.13
51.53
1832
42.5
578
7.7
DOPO (y=0.97)
46.15
50.52
1739
41.6
DºPO (y=0.98)
47.39
53.54
1705
47.3
518
7.8
DPPO (y=0.99)
48.01
52.97
1739
44.0
514
7.8
DPO (f=0.1)
47.60
1971
42.4
DPPO (y=1.01)
37.25
38.32
1948
28.1
578
7.6
DO (y=1.02)
37.75
39.30
1942
26.2
566
ORPO
37.75
39.29
1934
25.5
615
7.7
SimPO
44.41
50.34
1704
41.9
477
D'POt (x=0.98)
48.01
53.87
1726
46.1
526
7.7
Gemma2-Instruce (9B)
SFT
37.58
47.23
1566
44.1
608
8.3
DPO (8=0.011
54.53
57.05
1948
65.2
768
8.3
KTO
48.26
53.39
1775
57.1
705
8.3
IPO
50.86
50.00
2129
59.7
7.59
8.3
SamPO
45.78
55.21
1662
55.2
D'PO (y=0.95)
58.39
59,03
2034
65.5
739
8.5
D²PO (y=0.97)
56.83
59.25
1949
64.8
715
8.3
D'PO (y=0.98)
57.20
59.71
1950
66.4
724
8.3
DOPO (y=0.99)
53.98
57.38
1843
63.9
693
8.2
DPO (3=0.1)
47.02
53.43
1737
56.7
D'PO (y=1.01)
38.70
48.06
1592
43.7
610
8.2
P2 (y=1.02)
ORPO
41.93
51.14
1647
45.3
641
8.2
SimPO
54.47
58.42
65.0
Po (7=0.98)
56.71
60.76
1894
65.7
687
8.3
-eeiral--eoo--nnniiii (12B)
SET
44.60
47.71
1879
46.5
575
DPO (8=0.01)
58.76
57.29
2160
63.6
659
8.3
KTO
48.26
53.39
1775
57.1
705
8.3
IPO
50.86
50.00
2129
59.7
759
8.3
SamPO
45.78
55.21
1662
55.2
DOPO (y=0.95)
59.25
57.85
2167
60.7
665
8.6
D2 (y=0.97)
56.77
58.65
1969
57.5
586
8.6
D'PO (y=0.98)
57.34
62.07
1853
62.3
546
8.6
D2PO (y=0.99)
$4.29
58.83
1816
59.3
532
8.4
DPO (3=0.1)
53.48
53.32
59.0
D2PO (y=1.01)
45.34
48.06
1908
44.2
581
8.0
D' (y=1.02)
ORPO
41.93
51.14
1647
45.3
641
8.2
SimPO
54.47
58.42
1871
65.0
744
8.3
D'Pot (y=0.98)
56.71
60.76
1894
65.7
687
8.3
18

---

Published as a conference paper at ICLR 2025
Stronger Instruct Model Evaluation To verify the robustness properties of our method, ducted experiments under on on-policy setups based on a stronger Instruct model, Gemma2-Instnn Instructt (27B). Considering the limited computing resources, SimPO. Table 11 shows the evaluation results on three benchmarks, demonstrating that our method maintains certain advantage over stronger models.
only compared DọPO with DPO and
Table 11: Gemma2-Instren (27B) results under on-policy setups
A ApacaEval2
Arena Hard
MT--ench
Method
Win Rate (%)
L.C Win Rate (%)
Len.
Win Rate (%)
Len.
G4-Turbo
SFT
39.68
48. 13
1633
60.7
648
8.5
DPO (8=0.1)
48.45
53.66
1786
71.2
715
8.5
DPO (8=0.01)
57.39
59.05
1990
78.6
811
8.6
SimPO
54.29
61.30
1748
75.3
682
8.5
DOPO (y=0.988
57.52
60.52
1893
79.8
708
8.6
D CASE ANALYSIS
We list two concrete cases from AlpacaEval2 to compare D'PO with DPO. Regarding the first case in Figure the DPO model's output overthinks and mistakenly assumes that the stopover time is already included in the total travel time in the second step of its analysis, resulting in the omission of this time in the final calculation. In contrast, the model trained with method succinctly lists the times and correctly sums them to obtain the total travel time. Figure shows the second case: given the question, "What is the largest ocean the world?", the DPO model provides response without explanation. In contrast, the D²PO model produces well-structured answer with detailed explanations, making the answer more credible.
Figure A case examples of AlpacaEval2, generated by Hemmma2-Instuse (9B).
Input What is the largest ocean in the world? The largest ocean in the world is the Pacific Ocean.
Gemma2- 2-btit DPO
Gemma2 1amma2-9has 299b-it DOPO
The Pacific Ocean is the largest ocean the world.
Here's why:
Surface Area: It covers over 30% of the Earth's surface, roughly 63.8 million square miles (165.25 million square kilometers).
Volume:
It holds about half of the Earth's s total
water.
Depth: The Pacific Ocean boasts the deepest point on Earth, the Challenger Deep in the Mariana Trench, which plunges over 36, 000 feet (10, meters) below
level.
let me know you'd d like to know more about. the Pacific Ocean or other oceans!
19

---

Published as a conference paper at ICLR 2025
Figure A case examples of AlpacaEval2, generated by Mistral-Nemo-nstrucc (12B).
Input
Q: if it takes hours to fly Singapore, stop over for an hour and a half and then fly 14 hours to London then what is the total travel time if the plane is delayed in Singapore?
Mistral-Nemo DPO
To calculate the total travel time including the delay in Singapore, we need to add up all the components of the journey:
1. Flight Singapore: 8 hours
2. Stopover in Singapore: 1.5 hours (which is
already included in the total time as it's part of the journey, not delay)
the delay is 4. Flight from singapore to London: 14 hours
3. Delay in Singapore: Let's
"p' hours.
The total travel time including the delay would ber
Total travel time Flight to Singapore + Delay in singapore + Flight to London
Total travel time = 8 hours + D hours 14 hours Without knowing the exact duration of the delay (D), we can express the total travel time as: Total travel time 22 hours +
For example, is the delay in singapore is hours, the total travel time would be:
Total travel time - 22 hours + hours 24 hours So, the total travel time including the delay is 22 hours plus the duration of the delay (D).
Mistral-Nemt D*PO
calculate the total travel time including the delay in Singapore, let's s break down the journey and add the delay:
Total initial travel time without delay:
Flight to Singapore: hours
Stopover singapore: 1.5 hours
Flight from Singapore to London: 14 hours
rotal without delay 8 1.5 14 23.5 hours
Now, let's assume the plane is delayed in Singapore by 'D' hours. The delay will only affect the add "D" to the stopover
stopover time, duration:
Adjusted stopover time with delay = 1.5 hours D hours
Now, the total travel time with the delay is:
Total with delay
hours (flight to Singapore) (1.5 + +D) hours (stopover with delay) + hours (flight to London)
Total with delay = 8 + (1.5 D) 14 Total with delay 23.5
hours
Without knowing the speci fic curation
delay (D), we can express the total travel time with the delay as:
Total travel time with delay = 23.5 hours Delay in Singapore hours)
20

---

Published as a conference paper at ICLR 2025
E DERIVATION OF D'PO
In the reinforcement learning scenario, two fundamental concepts are the state-value function and the action-value function Q. The former represents the expected cumulative reward from taking action a in state S, meanwhile the latter represents the reward under state s. We extend the rela- tion between the Q-function and the V-function under KL divergence constraint, as proposed in Rafailov et al. (2024), by incorporating the temporal decay mechanism:
[r (st, at) log Tzef (ar se) + +V=* (SL+1), if St+1 is not terminal Q' (st, r (st, ar) 1og Trel (a. 5z)
(I0)
if S terminal
where ? E (0, 1] represents the discount factor. the assumption of DPO and its subsequent variants, y rypically
which indicates long-term returns do not need to decay. However, in auto-regressive scenarios such as language models, the longer the context provided, the lower the uncertainty of the model's predictions (see Figure 3). Therefore, the tokens the beginning of the response make greater contributions to the total return. Based on this assumption, we can get the formulation of the reward over trajectory t (51, 01, -1-10 ST
T-1 Σ (rrn(s ar) = Σ (s) ar) 7 log Tre (a,, st) - -2 7 V
T-1
(11)
1=
Noting that, the general maximum entropy RL. setting (Ziebart al., 2008; Ziebart, 2010), the optimal policy is given by Boltzmann probability distribution as:
(a: s:) (x)"
(12)
By taking the logarithm of the Eq. (12), we can simplified Eq. (11);
T-1
Ta(ae,82)
Σ (r((8, at) (s0, an) log Tre 80) + Σ (r
(13)
Tref (at, 81)
T-1
no To 8t) Tref (tt, 8t)
=V*(ss) +
Σ (
Blog
(14)
We can then plug Eq. (14) into the Bradley-Tore Terry ranking objective, which yields final loss formation:
T, Σ 7'r(s), at) - Σ ('r(s))a))
CpPo (TO) 1e-llgg log
(15)
To (y' Tret (y' y,y
x, y
T
no (yi
x, yitt
= log o
y Bog
log
(16)
(yf x, x(t)
t=
This is similar to the standard DPO objective, except for an additional temporay decay term 7. We empirically set y < 1 to focus more on short term return rather than long term return.
21

---

Published as a conference paper at ICLR 2025
F THEORETICAL ANALYSIS
In this section, we provide the detailed derivation of the upper bound for
(1, Vire (s) Vi (s)
V % (8) V (8)
+
V
(8)
Vi
(8)
+
V (8) V ($)
(17)
A2
As
F.1 THE UPPER BOUND OF A, AND A3
Noting that A and As both capture the difference in the expected returns of the same policy when evaluated under different 7, we can analyze the upper bound of these two items together.
The term A is given by with 7- 1.0:
A
VT Te
(s)
VØ
(s)
H
#-1
E
rs,, az)
Σ
at)
1=
H-1
Ex
Σ
(1 - 11--")rr 7))r(s), az)
Assuming the rewards are bounded, i.., (r(s, a)l s R, we have:
1- n
A < Σ (1-ン))= (H
1 R.
(18)
Similarly, we can obtain the upper bound of Ag:
H-1
1 1- 7 11
A
S Σ (1 (1-ン))-
(H
1 - R.
(19)
F.2 THE UPPER BOUND OF Ae
Lemma Performance Difference Lemma with finite horizon H (Kakade & Langford, 2002)
V (s) (8) 1 - Eandr [EneA(T"(C s) - #[a ()) (s,
(20)
where πt" represents optimal policy and π represents policy.
Based on the assumption that the rewards are bounded, i.c., Ir(s, a)l S R, we have:
1 H
Q" <
R
R
(21)
1
t=0
Finally, we can get
A V (s) (s)
(22)
(1 - H2
(1 - (1-7)2 Eervar
(a 8) a(a () R
(23)
2(1 722
Es-eT* (als)
(24)
(1 7)
22

---

Published as a conference paper at ICLR 2025
F.3 SUBOPTIMAL ANALYSIS
Adding the bounds on A1, A2, and Ag. we obtain:
SubOpt(TT, (:7 S 2(H 1 R 2(1 - -
1 1-7
(1 (1-7)2 Band [TV(T" (cls)] R (25)
Since both terms vary monotonically with the temporary decay factor y but in opposing directions, this implies the existence of optimal trade-off value, denoted y", within the interval (0, 1).
G COMPARISON WITH STRONGLY RELATED WORK
We noticed that Yang al. (2024) also emphasized the importance of focusing the contribution of earlier steps, but within the reverse chain a diffusion denoising process, rather than in an autoregressive LLM scenario. They employed y parameter control the contribution of earlier steps, ensuring the quality of generation. We would like to highlight how our approach differs from previous studies in the following perspectives:
Different Perspectives and Tasks:
While our work and the referenced prior work both in- volve preference optimization, they are derived from fundamentally different perspectives and are applied different downstream tasks. The prior work focuses on text-to-image tasks, which involve fixed-length generation through a non-autoregressive diffusion pro- cess. contrast, our research centered on LLMs in an autoregressive context, where sequence generation dynamics are inherently different.
Motivation Differences: As illustrated in Figures and 3, our motivation diverges signif- icantly from prior work. Our temporal decay mechanism designed address specific challenges in LLMs, such length bias and the need for alignment with human prefer- ences across varying sequence lengths.
Flexible Decay Mechanism: Our approach is not limited to exponential decay. As shown in Table 4, we explore multiple decay strategies, demonstrating the flexibility and adaptability of our method different scenarios and tasks.
Theoretical Insights: We have provided theoretical analysis based on the token-level MDP, suggesting the existence of an optimal gamma value for enhancing preference opti- mization. This theoretical foundation supports the practical effectiveness of our approach.
Extension and Complementarity: Our work serves as both extension and complement to the referenced study. While the prior work has not validated its approach on standard RLHF benchmarks, our method has been tested and shown to be effective in these contexts, as detailed in our experimental results.
23


Reviews:
[{"review_id": "R1WW9Ia0Ov", "reviewer": "Reviewer_cJJC", "summary": "This paper introduces D2PO (Decay-based Direct Preference Optimization), which aims to mitigate the length bias of DPO through a temporal decay parameter $\\gamma$. This follows the principle that the quality of the overall sequence generated can be improved when the earlier tokens are more accurate. The authors show that D2PO outperforms baselines on AlpacaEval2, Arena-Hard and MT-Bench.", "questions": "1. In what kind of tasks or settings (e.g. types of datasets) do you think the assumption that earlier tokens are more important may not hold?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors motivate the problem clearly with the length bias problem of DPO with Figure 1, and showing that earlier tokens are more crucial in alignment settings. Figure 3 motivates the need for temporal decay instead of treating all tokens across a sequence uniformly.\n1. Overall it is a complete paper which is well-written and easy to follow. As far as I can tell, the experimental results are also complete with thorough ablations.\n1. The comparison between D2PO and the baselines (DPO, SimPO and SamPO) is very clear and I like the details in Figure 2 as well as Table 5 in the Appendix, in addition to the quantitative results.", "weaknesses": "1. Lack of Novelty: the derivation in 3.3 seems like a straightforward extension of Rafailov et, al. (2024) but using discounted return instead of return. \n1. Lack of Theoretical Insight: Intuitively it makes sense that earlier tokens can be more important, and the authors do a good job providing experiments to support this. However, there is no formal motivation (e.g. some theoretical insights under the token MDP setting).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZTsgtFR6E3", "reviewer": "Reviewer_ozx7", "summary": "The paper \"Earlier Tokens Contribute More: Learning Direct Preference Optimization from Temporal Decay Perspective\" introduces D2PO, a variant of Direct Preference Optimization (DPO) that uses a temporal decay factor to prioritize earlier tokens in model responses. By applying decay to later tokens, D2PO mitigates verbosity bias and aligns responses more closely with human preferences. Empirical results on benchmarks like AlpacaEval 2 and Arena-Hard demonstrate significant performance gains over standard DPO.", "questions": "1. Would D2PO’s performance remain consistent on datasets that prioritize logical reasoning or mathematical tasks, where alignment needs may differ from instruction-following datasets?\n2. Is there a specific range of gamma values that you would recommend based on your findings, or does it vary significantly across models?\n3. Have you considered exploring an adaptive or task-specific gamma parameter to reduce sensitivity and improve D2PO’s generalizability?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Innovative temporal decay mechanism that dynamically weights token importance, effectively addressing alignment biases.\n2. Demonstrated gains over similar methods in benchmarks like AlpacaEval 2 and Arena-Hard, highlighting the model's capacity to generate concise and relevant responses.\n3. Flexible design, capable of operating in both reference-based and reference-free settings.", "weaknesses": "1. Reliance on Instruction-Following Datasets and GPT-4 as the Judge Model: The results in this paper are primarily validated on instruction-following datasets, using GPT-4 as the judge model. These choices raise concerns about generalizability, as instruction-following datasets can introduce noise that may obscure certain aspects of model alignment. Additionally, relying solely on GPT-4’s judgments may bias evaluation outcomes. To ensure robust assessment, further evaluations on alternative, specialized datasets—such as math or logical reasoning datasets—would help verify D2PO's effectiveness in different contexts and task types. This additional testing would strengthen the validity of D2PO's benefits and clarify its performance across a broader range of alignment tasks.\n\n2. Sensitivity to the Gamma Parameter: The performance of D2PO heavily relies on the gamma (γ) parameter, which controls the rate of decay across tokens. While gamma tuning allows flexibility, it may also necessitate extensive fine-tuning to achieve optimal performance across varied tasks. This dependency on precise gamma adjustment could make D2PO less practical in scenarios where rapid adaptation across diverse datasets is required. An exploration of adaptive or task-specific gamma values could improve generalizability and reduce the need for manual tuning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PnnXnhvR4P", "reviewer": "Reviewer_FaCH", "summary": "The paper introduces a temporal decay mechanism that prioritizes earlier tokens in Direct Preference Optimization (DPO) to improve the alignment of large language models with human preferences. By dynamically adjusting the weights of tokens based on their position, the authors address the length bias seen in existing models, leading to enhanced learning from human feedback. Empirical results demonstrate that this approach outperforms traditional methods like SimPO and SamPO across benchmarks including AlpacaEval and Arena-Hard by large margin.", "questions": "According to some emprical observation, the gain in autobenchmark could come from length as well as better style control [3]. Have you tried testing on IFEval or gsm8k like tasks?\n\n[3] \"https://lmsys.org/blog/2024-08-28-style-control/\"", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Presents significant advancements in preference optimization by linking temporal decay concepts to value learning within RL frameworks. \n\nThe authors conduct extensive, controlled experiments comparing their proposed method with a broad spectrum of established baselines, including SimPO, DPO, and KTO.", "weaknesses": "A main bottleneck of the paper is the reliance on automated benchmarks, which can be susceptible to manipulation or \"hacking.\", see [1]. This raises concerns about the validity of the results, suggesting that human evaluation is neccasary.\n\nAdditionally, the empirical results indicate that incorporating length control significantly increases the win rate of the proposed method. This finding suggests that it worth to further compare their approach with works [2] that also focus on length control and its impact on model performance.\n\n[1] \"Cheating automatic llm benchmarks: Null models achieve high win rates.\"\n[2] \"Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t1kgwDGVR9", "reviewer": "Reviewer_tbQd", "summary": "Most existing DPO variants assign rewards uniformly across the entire sequence, overlooking temporal dynamics in text generation.\nFor this purpose, this paper proposes D$^2$PO, an enhanced variant of DPO that incorporates a temporal decay factor, so as to prioritize earlier tokens that can be more critical for RLHF and focus on most relevant feedback.\nExperiment results verified the gain of the proposed method over vanilla DPO.", "questions": "1. L78-79: why is the more recent feedback potentially outdated?\n2. In the derivation of D$^2$PO, how did you cancel the partition function? Note that this is not an issue with the original DPO because it assume a bandit reward so that the partition function depends only on the prompt, which is the same for both chosen and rejected responses. \n3. L211-212: what is the definition of \"accuracy\"?\n4. In eq. (11), should $p_\\theta$ be $\\pi_\\theta$?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The overall direction of token-level reward/credit assignment for RLHF in LMs is promising.\n2. Experimental results and ablation studies are comprehensive and generally strong.", "weaknesses": "1. A major concern on this paper is about its contribution/novelty, as it omits several highly relevant related works. For example, the training loss eq. (5) in this paper is (almost) identical to eqs. (8) and (21) in [1]. Furthermore, the idea that earlier tokens can be more important for LMs' generation/alignment/reward optimization (e.g., L21-22, L69-71) has been discussed in [2] (first paragraph of section 3 and appendix F.3). At the minimum level, the authors ought to have a adequate citation and discussion with these prior works. Otherwise, the contribution of this paper is unjustified.\n\n[1] Yang, Shentao, Tianqi Chen, and Mingyuan Zhou. \"A Dense Reward View on Aligning Text-to-Image Diffusion with Preference.\" Forty-first International Conference on Machine Learning.\n\n[2] Yang, Shentao, et al. \"Preference-grounded token-level guidance for language model fine-tuning.\" Advances in Neural Information Processing Systems 36 (2023).\n\n2. Missing evaluation on the OpenLLM benchmark. Does D$^2$PO's improved RLHF come at the cost of general LM ability such as MMLU and/or GSM8K?\n\n3. L253-254: missing citation for maximum entropy RL, such as [3, 4].\n\n[3] Ziebart, Brian D., et al. \"Maximum entropy inverse reinforcement learning.\" AAAI. Vol. 8. 2008.\n\n[4] Ziebart, Brian D. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Carnegie Mellon University, 2010.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)