3D-Properties: Identifying Challenges in DPO and Charting a Path Forward

Authors: Yuzi Yan, Yibo Miao, Jialian Li, YipinZhang, Jian Xie, Zhijie Deng, Dong Yan
Primary area: foundation or frontier models, including LLMs
Keywords: LLM, DPO, RLHF

Abstract:
Aligning large language models (LLMs) with human preferences has gained significant attention, with Proximal Policy Optimization (PPO) as a standard yet computationally expensive method and Direct Preference Optimization (DPO) as a more efficient alternative. While DPO offers simplicity, it remains underutilized in state-of-the-art LLMs, suggesting potential limitations. In this work, we revisit DPO, analyzing its theoretical foundations and empirical performance to bridge this gap. We identify three key properties—termed \textbf{3D}-properties—that emerge from DPO’s learning process: \textbf{D}rastic drop in rejected response likelihood, \textbf{D}egradation into response suppression, and \textbf{D}ispersion effect on unseen responses. We show that these issues arise from DPO’s optimization dynamics, where the interaction between chosen and rejected response gradients leads to instability. Our findings are supported by experiments on both a controlled toy model and real-world LLM tasks, including mathematical problem-solving and instruction following. To address these challenges, we propose simple regularization techniques that improve training stability and performance. Additionally, we examine how preference data distribution impacts DPO’s effectiveness, offering insights into how alignment models handle out-of-domain (OOD) data. Our work connects these observations to broader research and provides a theoretical explanation for DPO’s limitations. We hope these insights will guide future advancements in reward-model-free preference learning, bringing it closer to reward-model-based approaches.

Decision: accept (poster)
Reviewer scores: 6, 5, 8, 6

Reviews and discussion:
<thread>
creation date:
1730623311755

thread_type:
official_review

<thread_object>
creator: reviewer_ky9e

<content>
summary:
This paper investigates the limitations of DPO in aligning large language models with human preferences, identifying three critical properties that hinder its performance: drastic drops in rejected response likelihood, degradation into response suppression, and dispersion effects on unseen responses. The authors provide theoretical explanations for these properties and demonstrate how they arise from DPO's objective. To address these challenges, the paper proposes regularization techniques and validates their effectiveness through experiments on both toy models and real-world language model tasks.

strengths:
- Studying DPO degradation phenomena is important due to its widespread use. This paper originally summarizes and theoretically analyzes several degradation phenomena of DPO discovered in previous work.

- Novel comparative analysis between on-policy and off-policy DPO on toy models.

- The paper is well-written and easy to follow.

weaknesses:
- I think the Explanation for Property 3 is inadequate. Firstly, compared to the previous two explanations, it lacks mathematical formulation and seems to merely restate empirical phenomena. Secondly, since the optimization process is conducted in mini-batches, while the model may ensure that overflow probability won't disperse to recently seen samples, I suspect it could also disperse to samples from the preference dataset that were encountered earlier, rather than necessarily dispersing to unseen samples outside the preference dataset.

- Following the previous point, the toy model setup, as mentioned by the authors in lines 344-353, is closer to treating each input/output as a token rather than a complete prompt/response, which is not a good toy model approximation of the real situation. One possible improvement would be to maintain other settings unchanged while increasing the sample size to enable mini-batch optimization that better resembles real-world conditions, with fewer epoch repetitions.

- While the authors used self-built Poem and Slogan datasets to evaluate the model's instruction following ability and acknowledged their limited scope, these datasets are insufficient to assess the model's general instruction following capabilities. The paper lacks evaluation on widely-used benchmarks in preference optimization work, such as AlpacaEval2, Arena-Hard, and MT-Bench, which are designed to test models' general instruction following ability.

- The proposed regularization techniques lack substantial significance. The first technique, which independently adjusts beta for reject responses, shows effectiveness in the poem task, but the optimal reject beta is merely 0.02 lower than the chosen beta. Without showing gradient comparisons for this technique, it's unclear whether it actually improves performance by addressing the large gap demonstrated in Figure 2. Moreover, the second technique, SFT loss, is already a widely established regularization technique.

- I am not quite convinced by the claims in section 3.4. Although existing works are cited to establish conceptual connections between RM and DPO, the subsequent gradient analysis focuses on r, creating a gap with the previous gradient analysis that focused on $\pi$.

questions:
- The probability distributions in the bottom-right figure don't seem to match with the leftmost figure in Figure 2. In Figure 2, the unseen probability at 500 epochs approaches 1, but in Figure 1 it's all zeros. The chosen probabilities also don't quite align.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730637317861

thread_type:
official_review

<thread_object>
creator: reviewer_dw57

<content>
summary:
This paper presents an interesting theoretical and empirical analysis of Direct Preference Optimization (DPO) and identifies three main challenges in its optimization process, termed as “3D-properties”: Drastic drop in rejected response likelihood, Degradation into response suppression, and Dispersion effect on unseen responses. These limitations, which do not arise in RM-based approaches, impact the stability and effectiveness of DPO. To address these issues, the authors propose regularization techniques, including adaptive gradient weighting and SFT loss. They conduct experiments on toy examples as well as math reasoning and instruction-following tasks to validate the presence of the 3D-properties, the advantages of on-policy over off-policy DPO, the comparative superiority of RM-based methods, and the effectiveness of the proposed regularization technique.

strengths:
- Significance: The paper addresses a crucial and interesting gap by analyzing the limitations of DPO
- Theoretical Analysis and Empirical Validation: The paper provides a theoretical framework alongside empirical results to validate the presence of the 3D-properties in DPO. This combined approach strengthens the findings, offering clear insights into the mechanisms driving DPO’s limitations and supporting the proposed solutions.

weaknesses:
- Presentation: The presentation could be improved to enhance readability. For example, the text size in Figures 2 and 3 is small, and the description of Scenarios 1-4, which is crucial for understanding the on-policy versus off-policy comparison, is currently only detailed in the appendix. Bringing this description to the main text would improve clarity.
- Experimental Design for On-Policy vs. Off-Policy Comparison: The on-policy and off-policy experiments rely on different data sources, which introduces potential confounds in the comparison. Using a more direct on-policy and off-policy setup, such as comparing historical-only data with semi-on-policy DPO (e.g., iterative DPO), would make the findings more robust.
- Parameter Tuning in Flex-DPO: Adjusting Flex-DPO requires tuning two parameters ( \beta^+  and  \beta^- ), and while Figure 4 provides some guidance, this approach may still present challenges for practical implementation due to a lack of clear tuning guidelines.

If the authors address these weaknesses, particularly by improving the clarity of presentation and by using a more controlled comparison between on-policy and off-policy data sources, I would raise my score. Addressing the Flex-DPO would also strengthen the work, though it is not essential for improving the overall contribution.

questions:
- In Section 4.2, it is mentioned that for the MATH dataset, the best and worst responses were selected by GPT-4. Why did the authors choose this method instead of directly verifying the answers? Given that GPT-4’s accuracy on MATH is only slightly above 50%, this approach seems potentially unreliable.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731190973335

thread_type:
official_review

<thread_object>
creator: reviewer_x5ts

<content>
summary:
The paper provides a comprehensive analysis of Direct Preference Optimization (DPO), examining its theoretical foundations and empirical performance to address current limitations. It identifies three perspectives—(1) Drastic drop in the likelihood of rejected responses, (2) Degradation into response suppression, and (3) Dispersion effect on unseen responses. The paper connects these observations to related research and offers a theoretical explanation for the underlying mechanisms. To improve DPO’s stability and performance, the authors propose regularization methods, including adaptive adjustment of gradient weights for chosen and rejected responses, as well as incorporating an SFT loss into the objective.

strengths:
The topic is interesting for RLHF. 

The paper introduces effective regularization methods, including adaptive gradient weighting for chosen and rejected responses.

The experiments are well-conducted and thorough.

weaknesses:
The study could benefit from using a wider range of LLMs.

The experiments can use more datasets except for math. 

The code is not open source, which may limit reproducibility.

questions:
For the toy model setup, which specific model is used in the paper?

Why does the paper focus primarily on math datasets rather than exploring a wider range of tasks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731467327367

thread_type:
official_review

<thread_object>
creator: reviewer_dbcq

<content>
summary:
The paper titled "3D-Properties: Identifying Challenges in DPO and Charting a Path Forward" presents a thorough analysis of the DPO method used for aligning LLMs with human preferences. The authors identify and term three critical properties of DPO's learning process the 3D-properties and propose regularization techniques to address the challenges these properties present. Theoretical analyses, toy model simulations and real-world experiments demonstrate the effectiveness of the proposed method.

strengths:
The paper is well-structured, where toy example can support their claims.
The paper offers a balanced mix of theoretical analysis and empirical evidence, which strengthens the claims made about the 3D-properties and their impact on DPO's performance.

weaknesses:
The three observations have been widely studied by previous works. Besides, one of the proposed regularization methods, incorporating an SFT loss into the objective, has been widely used in existing preference learning approaches [1]. This limits the novelty of the paper.
Considering that there are many existing methods to solve the DPO problem proposed in this paper, there is a lack of comparison with them, such as [2] and others.
Considering the generality of the proposed constraint algorithm, some advanced preference learning algorithms, such as SimPO [3], should also be tested.
More and more general LLMs should be included for evaluation, such as Meta-Llama3.

Reference: 
[1] Pang R Y, Yuan W, Cho K, et al. Iterative reasoning preference optimization[J]. arXiv preprint arXiv:2404.19733, 2024.
[2] Pal A, Karkhanis D, Dooley S, et al. Smaug: Fixing failure modes of preference optimisation with dpo-positive[J]. arXiv preprint arXiv:2402.13228, 2024.
[3] Meng Y, Xia M, Chen D. Simpo: Simple preference optimization with a reference-free reward. NeurIPS, 2024.

questions:
How to ensure that the initialization assumptions of parameter distribution can be applied to, or related to LLMs?

The detailed parameter adjustment strategy is only given in the toy experiment. What is the effect of different β values ​​in the real-world experiments?
</content>
</thread_object>

</thread>
<thread>
creation date:
1733406355894

thread_type:
public_comment

<thread_object>
creator: ~duanyu_feng1

<content>
comment:
Dear ICLR Committee Members,

I would like to bring to your attention the striking similarities between the theoretical section of this paper and previously paper, as well as potential issues regarding improper citation.

The argument presented in Section 3.1 of this paper is essentially identical to that in the theoretical section of the previously paper (arxiv:2404.04626). While this paper cites the previously paper and acknowledges similar conclusions, it fails to indicate that its theoretical part may originate from the cited publication. This raises concerns about improper citation practices and could pose academic risks to the overall integrity of the article.

Therefore, I kindly request that the committee review this matter.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper investigates key limitations of Direct Preference Optimization (DPO) in aligning language models, identifying three critical properties termed “3D-properties”: drastic drops in rejected response likelihood, degradation into response suppression, and dispersion effects on unseen responses. The paper’s main strengths lie in its theoretical analysis of these phenomena and proposed regularization techniques to address them. The work provides both theoretical foundations and empirical validation through toy models and real-world experiments. While the proposed regularization techniques may not be entirely novel, they are thoughtfully adapted to this context. Additionally, although the experimental evaluation could be expanded, the focus on math datasets and custom datasets provides meaningful insights. The theoretical analysis, combined with the practical implications, makes this a valuable contribution to the field. Therefore, I recommend acceptance.