Earlier Tokens Contribute More: Learning Direct Preference Optimization From Temporal Decay Perspective

Authors: Ruichen Shao, Bei Li, Gangao Liu, Yang Chen, ZhouXiang, Jingang Wang, Xunliang Cai, Peng Li
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: preference optimization, RLHF, DPO

Abstract:
Direct Preference Optimization (DPO) has gained attention as an efficient alternative to reinforcement learning from human feedback (RLHF) for aligning large language models (LLMs) with human preferences. Despite its advantages, DPO suffers from a length bias, generating responses longer than those from the reference model. Existing solutions like SimPO and SamPO address this issue but uniformly treat the contribution of rewards across sequences, overlooking temporal dynamics. To this end, we propose an enhanced preference optimization method that incorporates a temporal decay factor controlled by a gamma parameter. This dynamic weighting mechanism adjusts the influence of each reward based on its position in the sequence, prioritizing earlier tokens that are more critical for alignment. By adaptively focusing on more relevant feedback, our approach mitigates overfitting to less pertinent data and remains responsive to evolving human preferences. Experimental results on several benchmarks show that our approach consistently outperforms vanilla DPO by 5.9-8.8 points on AlpacaEval 2 and 3.3-9.7 points on Arena-Hard across different model architectures and sizes. 
Furthermore, additional experiments on mathematical and reasoning benchmarks (MMLU, GSM8K, and MATH) confirm that our method enhances performance without compromising general capabilities. Our codebase would be available at \url{https://github.com/LotuSrc/D2PO}.

Decision: accept (poster)
Reviewer scores: 8, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1729963379118

thread_type:
official_review

<thread_object>
creator: reviewer_tbqd

<content>
summary:
Most existing DPO variants assign rewards uniformly across the entire sequence, overlooking temporal dynamics in text generation.
For this purpose, this paper proposes D$^2$PO, an enhanced variant of DPO that incorporates a temporal decay factor, so as to prioritize earlier tokens that can be more critical for RLHF and focus on most relevant feedback.
Experiment results verified the gain of the proposed method over vanilla DPO.

strengths:
1. The overall direction of token-level reward/credit assignment for RLHF in LMs is promising.
2. Experimental results and ablation studies are comprehensive and generally strong.

weaknesses:
1. A major concern on this paper is about its contribution/novelty, as it omits several highly relevant related works. For example, the training loss eq. (5) in this paper is (almost) identical to eqs. (8) and (21) in [1]. Furthermore, the idea that earlier tokens can be more important for LMs' generation/alignment/reward optimization (e.g., L21-22, L69-71) has been discussed in [2] (first paragraph of section 3 and appendix F.3). At the minimum level, the authors ought to have a adequate citation and discussion with these prior works. Otherwise, the contribution of this paper is unjustified.

[1] Yang, Shentao, Tianqi Chen, and Mingyuan Zhou. "A Dense Reward View on Aligning Text-to-Image Diffusion with Preference." Forty-first International Conference on Machine Learning.

[2] Yang, Shentao, et al. "Preference-grounded token-level guidance for language model fine-tuning." Advances in Neural Information Processing Systems 36 (2023).

2. Missing evaluation on the OpenLLM benchmark. Does D$^2$PO's improved RLHF come at the cost of general LM ability such as MMLU and/or GSM8K?

3. L253-254: missing citation for maximum entropy RL, such as [3, 4].

[3] Ziebart, Brian D., et al. "Maximum entropy inverse reinforcement learning." AAAI. Vol. 8. 2008.

[4] Ziebart, Brian D. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Carnegie Mellon University, 2010.

questions:
1. L78-79: why is the more recent feedback potentially outdated?
2. In the derivation of D$^2$PO, how did you cancel the partition function? Note that this is not an issue with the original DPO because it assume a bandit reward so that the partition function depends only on the prompt, which is the same for both chosen and rejected responses. 
3. L211-212: what is the definition of "accuracy"?
4. In eq. (11), should $p_\theta$ be $\pi_\theta$?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730608479130

thread_type:
official_review

<thread_object>
creator: reviewer_fach

<content>
summary:
The paper introduces a temporal decay mechanism that prioritizes earlier tokens in Direct Preference Optimization (DPO) to improve the alignment of large language models with human preferences. By dynamically adjusting the weights of tokens based on their position, the authors address the length bias seen in existing models, leading to enhanced learning from human feedback. Empirical results demonstrate that this approach outperforms traditional methods like SimPO and SamPO across benchmarks including AlpacaEval and Arena-Hard by large margin.

strengths:
Presents significant advancements in preference optimization by linking temporal decay concepts to value learning within RL frameworks. 

The authors conduct extensive, controlled experiments comparing their proposed method with a broad spectrum of established baselines, including SimPO, DPO, and KTO.

weaknesses:
A main bottleneck of the paper is the reliance on automated benchmarks, which can be susceptible to manipulation or "hacking.", see [1]. This raises concerns about the validity of the results, suggesting that human evaluation is neccasary.

Additionally, the empirical results indicate that incorporating length control significantly increases the win rate of the proposed method. This finding suggests that it worth to further compare their approach with works [2] that also focus on length control and its impact on model performance.

[1] "Cheating automatic llm benchmarks: Null models achieve high win rates."
[2] "Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge"

questions:
According to some emprical observation, the gain in autobenchmark could come from length as well as better style control [3]. Have you tried testing on IFEval or gsm8k like tasks?

[3] "https://lmsys.org/blog/2024-08-28-style-control/"
</content>
</thread_object>

</thread>
<thread>
creation date:
1730759086462

thread_type:
official_review

<thread_object>
creator: reviewer_ozx7

<content>
summary:
The paper "Earlier Tokens Contribute More: Learning Direct Preference Optimization from Temporal Decay Perspective" introduces D2PO, a variant of Direct Preference Optimization (DPO) that uses a temporal decay factor to prioritize earlier tokens in model responses. By applying decay to later tokens, D2PO mitigates verbosity bias and aligns responses more closely with human preferences. Empirical results on benchmarks like AlpacaEval 2 and Arena-Hard demonstrate significant performance gains over standard DPO.

strengths:
1. Innovative temporal decay mechanism that dynamically weights token importance, effectively addressing alignment biases.
2. Demonstrated gains over similar methods in benchmarks like AlpacaEval 2 and Arena-Hard, highlighting the model's capacity to generate concise and relevant responses.
3. Flexible design, capable of operating in both reference-based and reference-free settings.

weaknesses:
1. Reliance on Instruction-Following Datasets and GPT-4 as the Judge Model: The results in this paper are primarily validated on instruction-following datasets, using GPT-4 as the judge model. These choices raise concerns about generalizability, as instruction-following datasets can introduce noise that may obscure certain aspects of model alignment. Additionally, relying solely on GPT-4’s judgments may bias evaluation outcomes. To ensure robust assessment, further evaluations on alternative, specialized datasets—such as math or logical reasoning datasets—would help verify D2PO's effectiveness in different contexts and task types. This additional testing would strengthen the validity of D2PO's benefits and clarify its performance across a broader range of alignment tasks.

2. Sensitivity to the Gamma Parameter: The performance of D2PO heavily relies on the gamma (γ) parameter, which controls the rate of decay across tokens. While gamma tuning allows flexibility, it may also necessitate extensive fine-tuning to achieve optimal performance across varied tasks. This dependency on precise gamma adjustment could make D2PO less practical in scenarios where rapid adaptation across diverse datasets is required. An exploration of adaptive or task-specific gamma values could improve generalizability and reduce the need for manual tuning.

questions:
1. Would D2PO’s performance remain consistent on datasets that prioritize logical reasoning or mathematical tasks, where alignment needs may differ from instruction-following datasets?
2. Is there a specific range of gamma values that you would recommend based on your findings, or does it vary significantly across models?
3. Have you considered exploring an adaptive or task-specific gamma parameter to reduce sensitivity and improve D2PO’s generalizability?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731405679624

thread_type:
official_review

<thread_object>
creator: reviewer_cjjc

<content>
summary:
This paper introduces D2PO (Decay-based Direct Preference Optimization), which aims to mitigate the length bias of DPO through a temporal decay parameter $\gamma$. This follows the principle that the quality of the overall sequence generated can be improved when the earlier tokens are more accurate. The authors show that D2PO outperforms baselines on AlpacaEval2, Arena-Hard and MT-Bench.

strengths:
1. The authors motivate the problem clearly with the length bias problem of DPO with Figure 1, and showing that earlier tokens are more crucial in alignment settings. Figure 3 motivates the need for temporal decay instead of treating all tokens across a sequence uniformly.
1. Overall it is a complete paper which is well-written and easy to follow. As far as I can tell, the experimental results are also complete with thorough ablations.
1. The comparison between D2PO and the baselines (DPO, SimPO and SamPO) is very clear and I like the details in Figure 2 as well as Table 5 in the Appendix, in addition to the quantitative results.

weaknesses:
1. Lack of Novelty: the derivation in 3.3 seems like a straightforward extension of Rafailov et, al. (2024) but using discounted return instead of return. 
1. Lack of Theoretical Insight: Intuitively it makes sense that earlier tokens can be more important, and the authors do a good job providing experiments to support this. However, there is no formal motivation (e.g. some theoretical insights under the token MDP setting).

questions:
1. In what kind of tasks or settings (e.g. types of datasets) do you think the assumption that earlier tokens are more important may not hold?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731919104304

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response for Additional Experiments

comment:
To further substantiate our claims, we conducted additional experiments on three widely used benchmarks focused on logical reasoning and mathematical problem-solving: MMLU, GSM8K, and Math. These benchmarks are specifically designed to assess the reasoning abilities of LLMs in diverse and complex tasks. Additionally, we report results on the OpenLLM Benchmark to reinforce our assertion that our improvements do not come at the cost of general language modeling ability. Due to time constraints, our experiments were primarily conducted on the Llama3-8B and Gemma2-9B models.

| Method (Llama3-8B)    | MMLU  | GSM8K | Math  | IFEval prompt-strict | ARC-C (25) | Hellaswag(10) | TruthfulQA(0)| Winogrande(5)|
|-----------------------|-------|-------|-------|----------------------|------------|---------------|--------------|--------------|
| Instruct              | 61.66 | 78.47 | 7.91  | 68.58                | 61.95      | 78.80         | 51.62        | 75.53        |
| D²PO (gamma0.98)      | 61.38 | 71.95 | 8.46  | 65.62                | 65.78      | 79.03         | 57.57        | 75.14        |
| DPO                   | 56.66 | 70.51 | 7.77  | 65.06                | 65.10      | 79.99         | 56.38        | 74.51        |
| SimPO                 | 55.22 | 57.54 | 5.27  | 60.81                | 67.58      | 78.82         | 63.83        | 74.27        |


| Method (Gemma2-9B)    | MMLU  | GSM8K | Math  | IFEval prompt-strict | ARC-C (25) | Hellaswag(10) | TruthfulQA(0) | Winogrande(5)|
|-----------------------|-------|-------|-------|----------------------|------------|---------------|---------------|--------------|
| instruct              | 72.82 | 87.41 | 19.42 | 71.90                | 71.84      | 81.67         | 60.23         | 77.90        |
| D²PO    (gamma0.98)   | 72.68 | 88.86 | 21.22 | 71.16                | 71.42      | 81.03         | 61.34         | 76.01        |
| DPO                   | 72.21 | 88.53 | 19.42 | 60.07                | 69.88      | 71.48         | 57.65         | 72.69        |
| SimPO                 | 72.35 | 88.17 | 19.00 | 71.53                | 68.34      | 66.51         | 58.87         | 73.72        |
</content>
</thread_object>

</thread>
<thread>
creation date:
1732261831423

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Update the Manuscript

comment:
Thank you once again for all the reviewers' efforts! We have updated our manuscript accordingly, and the revisions are summarized as follows:

- We conducted additional experiments on the OpenLLM benchmark, including MMLU, GSM8K, Math, and IFEval. Our results demonstrate that the improvements in RLHF benchmarks do not compromise general language modeling abilities.
- We performed human evaluations on both Arena-Hard and AlpacaEval2 datasets. The results indicate that our proposed D²PO significantly outperforms DPO in terms of the win rate.
- We provided a theoretical analysis explaining why our proposed temporal-decay mechanism is effective from the token MDP perspective. (Section 4)
- We have added the missing references raised by the reviewers.
- We corrected inaccurate statements to ensure clarity and precision.
- We moved the experimental setups (Appendix A) and some of the analyses (Appendix C) to the appendix.

We hope that these revisions address your concerns, and we look forward to discussing them with you. Since the discussion phase will be ending, we kindly request that you can read the response and make a reconsideration of our work. Please feel free to contact us if there are any points you would like to discuss further. Thank you for your valuable time!
</content>
</thread_object>

</thread>
<thread>
creation date:
1732596750755

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
We are eager to discuss with dear reviewers

comment:
Hi all reviewers,

Thank you once again for your valuable comments and suggestions, which have been very helpful to us. We understand this is a particularly busy time, and we greatly appreciate your efforts.

We kindly ask whether you could take a moment to review our additional experiments and responses to see if they adequately address your concerns. Should there be any further feedback, we are committed to incorporating it in the coming days.

Thank you for your time and consideration.

Best regards,

All Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a token-level reward modeling method (R3HF) for reinforcement learning from human feedback, aiming to address limitations of existing RLHF approaches that use only a single, sequence-level reward. By treating reward prediction as a regression problem, the work was able to assign more granular token-level credit, thereby improving the learned policy.

Reviewers generally agree that the paper addresses a key limitation in current RLHF methods. Some reviewers initially questioned the novelty and technical insight from the work, but the concerns were generally addressed by additional discussions and additional experiments showing the improvements do not come at the cost of general language modeling ability. While gamma discounting of dense reward is a well studied concept in traditional RL, this work demonstrated that it has practical gains in RLHF of LLMs. I recommend accepting the paper.