Understanding Likelihood Over-optimisation in Direct Alignment Algorithms

Authors: Zhengyan Shi, Sander Land, Acyr Locatelli, Matthieu Geist, Max Bartolo
Primary area: foundation or frontier models, including LLMs
Keywords: Preference Learning, Large Language Model, Direct Alignment Algorithm

Abstract:
Direct Alignment Algorithms (DAAs), such as Direct Preference Optimisation (DPO) and Identity Preference Optimisation (IPO), have emerged as alternatives to online Reinforcement Learning from Human Feedback (RLHF) algorithms such as Proximal Policy Optimisation (PPO) for aligning language models to human preferences, without the need for explicit reward modelling. These methods generally aim to increase the likelihood of generating better (preferred) completions while discouraging worse (non-preferred) ones, while staying close to the original model's behaviour. In this work, we explore the relationship between completion likelihood and model performance in state-of-the-art DAAs, and identify a critical issue of likelihood over-optimisation. Contrary to expectations, we find that higher likelihood of better completions and larger margins between better and worse completion likelihoods do not necessarily lead to better performance, and may even degrade it. Our analysis reveals that while higher likelihood correlates with better memorisation of factual knowledge patterns, a slightly lower completion likelihood tends to improve output diversity, thus leading to better generalisation to unseen scenarios. Moreover, we identify two key indicators that signal when over-optimised output diversity begins to harm performance: ***Decreasing Entropy over Top-k Tokens*** and ***Diminishing Top-k Probability Mass***. Our experimental results validate that these indicators are reliable signs of declining performance under different regularisations, helping prevent over-optimisation and improve alignment with human preferences.

Decision: reject
Reviewer scores: 6, 3, 8, 3

Reviews and discussion:
<thread>
creation date:
1729157975094

thread_type:
official_review

<thread_object>
creator: reviewer_tr7t

<content>
summary:
The authors explored the effect of increasing the probability of chosen sequences on the overoptimization of direct alignment algorithms (DAA). They concluded that an increased gap between chosen and rejected sequences could lead to overoptimization and linked overoptimization with reduced diversity of samples.

strengths:
- The initial problem of overoptimization is important, and solving it is crucial for the field of alignment.
- Understanding signals of overoptimization is helpful and allows for faster evaluation. Instead of waiting for evaluation with LLMs, we can detect overoptimization through reduced entropy and decreasing mass in top-k tokens.

weaknesses:
- While the authors provided implementation details, it is unclear why they used a 7B model with closed weights when a wide range of open-source models of this size is available.
- My main concern is the limited novelty of the obtained results. From what I see in Figure 2, while the gap between chosen and rejected sequences is increasing for small $\beta$, the likelihood of chosen sequences is decreasing. This aligns with the observations of [1], as probability mass moves towards out-of-distribution (OOD) examples (and to avoid overoptimization, we should prevent leakage of mass to OOD sequences). Therefore, the findings do not seem to present new insights. When demonstrating that increased probabilities for chosen sequences can also lead to overoptimization, it is important to explore the probabilities associated with OOD examples. From this perspective, the main novelty of the paper lies in understanding the mechanism of detecting overoptimization via reduced diversity and aligning these observations with the impact on performance across various tasks, but this aspect is poorly explored.
- The presentation of results is difficult to read. The captions of large figures (2, 3, 4) do not contain useful information for understanding, making some metrics hard to interpret. Additionally, the analysis of these results is far from the figures themselves.

[1] Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms, Rafailov et. al.

questions:
See Weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730221940647

thread_type:
official_review

<thread_object>
creator: reviewer_psr8

<content>
summary:
This paper provides a experimental analysis of the behaviors of margin-based alignment approaches, especially, they focus on the so-call over-optimisation issue. They claim that they are the first to explore the relationship between completion likelihood and performance in alignment algorithms. Their empirical finding is that, likelihood does not have a striong positive correlation with model performance, and it might be affected by diversity. Besides, they identify two indicators of overly generatinve diverse outputs.

strengths:
## Originality
- Though the claim may not be novel, the systematic experiments are original and interesting.
- The reviewer appreciates the statistical metrics.

## Clarity
- This paper is well-written and densely organised.
- The figures are clearly plotted.

## Significance
- This paper may be able to clarify a controversial point, whether DAA should directly increase(decrease) the likelihood of accepted(rejected) completion? which the reviewer believes is a important question.

weaknesses:
## Major
- The two indicators are so trivial, that people already know this before. Thus these two findings might not be counted as contributions.
- The experiments are restricted. Some DAAs with alternate objective (CPO, SimPO, ...) (such as length normalization) are not tested, making the claim less solid.

## Minor
- The curves are a bit confusing, especially Figure 4, making it hard to come to the authors' conclusion.

questions:
- In Figure 3, why the DPO/IPO 7B ultrafeedback model cannot gain much improvement compared with initial checkpoint?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730634721249

thread_type:
official_review

<thread_object>
creator: reviewer_djek

<content>
summary:
This paper presents a detailed analysis of contemporary offline alignment methods, focusing on DPO, IPO, and Hinge. A comprehensive set of metrics, gathered during the alignment process for both a proprietary 7B model and the Cohere Command R 35B model, is employed to investigate the issue of over-optimization. Furthermore, the paper proposes using the entropy of the top-k tokens during generation for DPO and IPO, as well as the aggregate mass of the top-k tokens for Hinge, as indicators of over-optimization.

strengths:
- Evaluation details are comprehensive.
- This work is useful for the community as it could be used to determine offline metrics to identify the best model at various steps.

weaknesses:
- Over-optimization itself is already well-studied in [1]. Some results, like Figure 1, seem to be consequences of the high KL divergence. I think NLL for y_w and KL are highly correlated metrics. It would be useful to see a figure with KL on the x-axis and NLL(y_w) on the y-axis.
- While Figure 1 supports the claim that there is no correlation between NLL(y_w) and win rate in general, Figure 2 shows that, for the given method and hyperparameters, the best step can be determined by observing NLL(y_w). More precisely, I would suggest tracking the difference between NLL from the previous and current checkpoints. If this value becomes larger than a certain threshold, one can stop training and thereby obtain the strongest checkpoint. Therefore, from the reported I would say that tracking NLL(y_w) could be sufficient to detect over-optimization across all methods (instead of proposed methodology in Section 4.4, which depends on method).
- The paper presents an extensive amount of metrics during the training procedure; however, there is no clear criterion to detect over-optimization. First of all, the proposed methods in Section 4.4 are highly dependent on alignment methods, and if one were to use other popular methods (like KTO, ORPO, etc.), it is not clear which metric would serve as a flag for over-optimization. Additionally, it is unclear if these results remain the same for different models (see next weakness point).
- The choice of models is in question. Results obtained from a closed-source 7B model may not generalize to widely used open-weight models with similar sizes like LLAMA, Gemma, or Mistral. Additionally, this has a negative impact on the reproducibility of the experiments.

[1] Scaling Laws for Reward Model Over-optimization in Direct Alignment Algorithms (Rafailov et al.)

questions:
Lower entropy usually indicates that a model is overfitted. Could this fact indicate that the UltraFeedback dataset is not sufficiently diverse, and therefore the model could memorize patterns that are not useful in general? Does the issue of over-optimization hold across different datasets?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730712573879

thread_type:
official_review

<thread_object>
creator: reviewer_vwqc

<content>
summary:
* The paper studies how completion likelihood affects model performance in Direct Alignment Algorithms (DAAs) like DPO, IPO and Hinge loss, using 7B and 35B models.
* Key finding shows that higher likelihood of better completions and larger margins between better/worse completions don't necessarily improve performance. Higher likelihood helps with factual recall but can hurt diversity.
* Authors propose two metrics for detecting likelihood over-optimization: Decreasing Entropy over Top-k Tokens and Diminishing Top-k Probability Mass. These metrics help prevent over-optimization while maintaining good performance.

strengths:
* The work provides comprehensive experiments across multiple dimensions including likelihood, diversity and performance. The ablation studies are systematic and well-documented.
* The paper challenges common assumptions about likelihood optimization in DAAs. The trade-off between memorization and generalization is clearly demonstrated with empirical evidence.
* The proposed metrics for detecting over-optimization are concrete and actionable. The findings provide clear guidance for practitioners working on DAA training.

weaknesses:
* The paper only studies single epoch training when DPO typically needs 2-3 epochs for best performance. This important limitation is not well justified or analyzed.
* The BINARIZEDPREF dataset lacks crucial details about its construction and preference collection. If preferences come from GPT-4 rather than humans, the generalization of findings is questionable.
* The primary evaluation uses GPT-3.5-turbo as baseline which feels dated. Testing against stronger models like LLaMA-3 would strengthen the findings.

questions:
* Could you provide more information about BINARIZEDPREF construction, including how preferences were collected and validated? The source of preferences is particularly important.
* Why was single-epoch training chosen when DPO typically needs more epochs? Do these patterns persist in multi-epoch training?
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper studies the relationship between completion likelihood and model performance in Direct Alignment Algorithms (DAAs), analyzing how increased likelihood of better completions and margins between better/worse completions affect performance. The paper claims that higher likelihood doesn't necessarily improve performance and can hurt diversity, proposing two metrics (Decreasing Entropy over Top-k Tokens and Diminishing Top-k Probability Mass) for detecting likelihood over-optimization. The key strengths include comprehensive experiments across multiple dimensions and clear implications for practitioners working on DAA training. However, major weaknesses include: (1) limited novelty, as the observed patterns appear to be consequences of KL divergence effects already documented in prior work; (2) restricted experimental setup using proprietary models/datasets that limits reproducibility; and (3) lack of systematic validation across a broader range of models, datasets and methods. The paper is recommended for rejection primarily because the consensus from the reviewers that it fails to convincingly demonstrate that its findings represent a new phenomenon distinct from known scaling laws of alignment methods.