Understanding Likelihood Over-optimisation in Direct Alignment Algorithms

## Abstract

Abstract Direct Alignment Algorithms (DAAs), such as Direct Preference Optimisation (DPO) and Identity Preference Optimisation (IPO), have emerged as alternatives to online Reinforcement Learning from Human Feedback (RLHF) algorithms such as Proximal Policy Optimisation (PPO) for aligning language models to human preferences, without the need for explicit reward modelling.
These methods generally aim to increase the likelihood of generating better (preferred) completions while discouraging worse (non-preferred) ones, while staying close to the original model’s behaviour.
In this work, we explore the relationship between completion likelihood and model performance in state-of-the-art DAAs, and identify a critical issue of likelihood over-optimisation.
Contrary to expectations, we find that higher likelihood of better completions and larger margins between better and worse completion likelihoods do not necessarily lead to better performance, and may even degrade it.
Our analysis reveals that while higher likelihood correlates with better memorisation of factual knowledge patterns, a slightly lower completion likelihood tends to improve output diversity, thus leading to better generalisation to unseen scenarios.
Moreover, we identify two key indicators that signal when over-optimised output diversity begins to harm performance: Decreasing Entropy over Top- k 𝑘 k italic_k Tokens and Diminishing Top- k 𝑘 k italic_k Probability Mass .
Our experimental results validate that these indicators are reliable signs of declining performance under different regularisation schemes, helping prevent over-optimisation and improve alignment with human preferences.

## 1 Introduction

Recent advancements in Large Language Models (LLMs) (Touvron et al., ; Achiam et al., ; Roziere et al., ; Dubey et al., ; Land & Bartolo, ) have significantly expanded their capabilities, enabling applications such as code generation, tool use, and interactive communication. As LLMs become increasingly powerful, the challenge of aligning them with human preferences has grown in importance. Direct Alignment Algorithms (DAAs), such as Direct Preference Optimisation (DPO) (Rafailov et al., ) and Identity Preference Optimisation (IPO) (Azar et al., ), have emerged as alternatives to Reinforcement Learning from Human Feedback (RLHF) (Ziegler et al., ; Bai et al., ) for training LMs on human preference data. These methods aim to bypass the traditional RLHF pipeline by directly optimising the policy without explicit reward modelling.

DAAs are designed to increase the likelihood of better completions while reducing the likelihood of worse ones, all while staying close to the original model’s behaviour.
However, a known issue with standard DAAs is that they may decrease the likelihood of better completions as long as the relative probability between better and worse completions increases (Rafailov et al., ; Pal et al., ).
Recent research has sought to address this by focusing on maintaining a high likelihood for better completions (Pal et al., ).
For example, several works (Pang et al., ; Hong et al., ), including Llama-3.1 (Dubey et al., ) and Nvidia Nemotron (Adler et al., ), introduce a scaled negative log-likelihood (NLL) loss on better completions, aiming to stabilise DAA training by preserving the desired formatting and preventing a drop in log probability for better completions.
Despite these efforts, key research questions remain:
Is it truly necessary to maintain a higher likelihood of better completions, and aim for a larger likelihood margin between better and worse completions? And if not, How can we strike a balance for completion likelihood to maximise model performance in terms of alignment with human preferences?

Figure: Figure 1: Mean Log Likelihood (LLH) of Better Completion vs Win Probability (Left) and Average Number of Tokens in Model Outputs (Right). We report 7B and 35B model results on the UltraFeedback dataset. Our results indicate that: (1) A higher likelihood for better completions does not necessarily translate to higher win probability; and (2) There is no obvious correlation between the average number of tokens in model outputs and the likelihood of better completions.
Refer to caption: x1.png

In this work, we first explore the relationship between completion log-likelihood and model performance in state-of-the-art DAAs (§3).
Specifically, we find that neither a higher likelihood of preferred completions nor larger margins between better and worse completion likelihoods necessarily lead to better performance (measured by win probability) and may even degrade it (§4.2), as shown in Figure 1.
Furthermore, our experiments demonstrate that optimising both factors simultaneously also does not guarantee improvement.
Our results reveal that while a higher likelihood of better completion generally has better memorisation of factual knowledge patterns, an excessively high likelihood can result in over-optimisation.
In contrast, slightly lower completion likelihood tends to improve output diversity, thus leading to better generalisation to unseen scenarios (§4.3).

While avoiding an overly high completion likelihood tends to improve model diversity and generalisation, it is crucial to strike a balance between diversity and maintaining a high likelihood for desired outputs preferred by humans.
To this end, our study outlines two key indicators that signal when overly generating diverse outputs begins to negatively impact model performance (§4.4):
(1) Decreasing Entropy over Top-$k$ Tokens(^1^11In this work, entropy measures uncertainty in token distribution, with a uniform distribution giving the highest entropy of 1 (maximum diversity) and a single-token distribution yielding 0 (no uncertainty).):
As the likelihood of better completions decreases during training, an increasing entropy suggests that tokens within better completions still have higher probabilities relative to other tokens in the Top-$k$, though the gap is narrowing.
However, a decreasing entropy over the Top-$k$ tokens is a warning sign that the model is assigning disproportionately low probabilities to tokens within better completions, allowing other tokens to rise in probability, which may lead to outputs that are not aligned with human preferences.
Notably, a reversed entropy trend is a particularly strong indicator of over-optimised diversity; and
(2) Diminishing Top-$k$ Token Probability Mass:
This occurs when the probability mass concentrated on the top $k$ most likely tokens declines, resulting in more random outputs and a higher likelihood of selecting tokens outside the top $k$.
Such a flattening of the probability distribution can lead to phenomena such as code-switching (Doğruöz et al., ; Marchisio et al., ), making the model more prone to confusion.
Our experimental results validate that these two indicators are strong predictors of declining model performance, providing critical markers to help avoid over-optimization while balancing diversity.

## 2 Related Work

#### Preference learning.

Recent years have seen significant progress in aligning large language models (LLMs) with human preferences (Hosking et al., ; Kirk et al., ; Wu et al., ).
RLHF, pioneered by Christiano et al. (); Ziegler et al. () and developed in subsequent works (Stiennon et al., ; Bai et al., ; Ouyang et al., ), typically consists of three stages: supervised fine-tuning (SFT), reward modelling, and RL fine-tuning (Schulman et al., ; Mnih, ; Shi & Lipani, ; Aryabumi et al., ; Ahmadian et al., ).
The reward model is trained to predict human preferences between pairs of model outputs, while the RL phase optimises the model to maximise the reward (Ye et al., ; Lambert et al., ; Zhou et al., ; Liu et al., ). More recently, researchers have proposed Direct Alignment Algorithms (Rafailov et al., ; Zhao et al., ; Azar et al., ) that aim to simplify RLHF by directly optimising the policy without a reward modelling or RL phase.

#### Over-optimisation for preference learning.

Over-optimisation in preference learning occurs when a model’s performance on a proxy measure improves while its true performance declines.
Gao et al. () was the first to extensively characterise this issue in the context of RLHF, where optimisation against a learned reward model leads to increased proxy rewards, while actual task performance plateaus or worsens, a phenomenon termed “reward over-optimisation”. Subsequent studies have observed similar patterns (Eisenstein et al., ; Touvron et al., ; Dubois et al., ).
To mitigate reward over-optimisation, researchers have proposed various approaches, such as using ensembles or data smoothing for reward modelling (Eisenstein et al., ; Zhang et al., ; Coste et al., ; Zhu et al., ; Yang et al., ), and leveraging uncertainty signals (Yang et al., ; Zhai et al., ; Zhou et al., ; Yang et al., ).
Rafailov et al. () extended this analysis to DAAs, showing that even without an explicit reward model, DAAs exhibit similar over-optimisation patterns at higher KL-divergence budgets.
In this work, we explore the DAAs’ over-optimisation in the context of completion likelihood.

#### Generalisation and diversity.

Generalisation and diversity in LM outputs has been a growing concern in the field of NLP, particularly regarding the impact of fine-tuning methods (Hendrycks et al., ).
Several studies have explored how RLHF influences output diversity and generalisation. Khalifa et al. (); Perez et al. () suggests that RLHF tends to produce models with reduced output diversity.
Kirk et al. () highlights a trade-off between generalisation and diversity in current LLM fine-tuning, with RLHF showing better out-of-distribution generalisation but substantially decreased output diversity compared to SFT.
This trade-off between alignment, performance, and diversity relates to the broader concept of “alignment tax” in LM fine-tuning.
Bai et al. (); Ouyang et al. (); Bai et al. (); Kotha et al. () observed that aligning models with human preferences, through RLHF, can sometimes degrade performance on specific tasks, especially for smaller models. Various approaches have been proposed to mitigate the alignment tax (Noukhovitch et al., ; Shi & Lipani, ; Qi et al., ). For example, Ouyang et al. () suggested incorporating pretraining data into RLHF fine-tuning to minimise performance regressions on standard NLP datasets.
However, these studies have not explored how the optimisation of completion likelihood correlates with model performance, including diversity and generalisation.

## 3 Preliminaries

### 3.1 Direct Alignment Algorithms

Direct Alignment Algorithms (DAAs) are a family of methods designed to train LMs to align with human preferences without the need for explicit reward modelling. These algorithms aim to optimise a policy model to maximise the probability of better completions over worse ones.

#### Direct Preference Optimisation.

Direct Preference Optimisation (DPO) (Rafailov et al., ) is a foundational DAA method.
The DPO loss function is defined as follows:

$$ $\displaystyle L_{\text{DPO}}(\pi_{\theta};\pi_{\text{ref}})$ $\displaystyle=-\mathbb{E}_{(x,y_{w},y_{l})\sim D}\left[\log\sigma\left(\beta \Delta(x,y_{w},y_{l})\right)\right],$ (1) $\displaystyle\Delta(x,y_{w},y_{l})$ $\displaystyle=\log\frac{\pi_{\theta}(y_{w}|x)}{\pi_{\text{ref}}(y_{w}|x)}-\log \frac{\pi_{\theta}(y_{l}|x)}{\pi_{\text{ref}}(y_{l}|x)},$ (2) $$

where $\pi_{\theta}$ is the policy model being optimised, $\pi_{\text{ref}}$ is a reference model where $\pi_{\theta}$ is initialised from, $D$ is the dataset of preference pairs, $x$ is the input, $y_{w}$ and $y_{l}$ are the better and worse completions respectively, $\sigma$ is the sigmoid function, and $\beta$ is a temperature hyperparameter.
The term $\Delta(x,y_{w},y_{l})$ quantifies the difference in log probabilities between better and worse completions.

#### Identity Preference Optimisation.

Identity Preference Optimisation (IPO) (Azar et al., ) is a variant of DAA methods. Specifically, IPO uses a quadratic loss function, which is defined as:

$$ $L_{\text{IPO}}(\pi_{\theta};\pi_{\text{ref}})=\mathbb{E}_{(x,y_{w},y_{l})\sim D }\left[\left(\tau\Delta(x,y_{w},y_{l})-\frac{1}{2}\right)^{2}\right],$ (3) $$

where $\tau$ is a temperature hyperparameter. This formulation aims to push the difference in log probabilities $\Delta(x,y_{w},y_{l})$, defined within the DPO framework, towards a target value of $\frac{1}{2\tau}$.

#### Hinge Loss.

The hinge loss method (Zhao et al., ; Liu et al., ) represents another variation within the DAA framework. Specifically, we adopt the loss function from SLiC-HF (Zhao et al., ), which is defined as follows:

$$ $L_{\text{Hinge}}(\pi_{\theta};\pi_{\text{ref}})=\mathbb{E}_{(x,y_{w},y_{l}) \sim D}\left[\max\left(0,\gamma-\log\frac{\pi_{\theta}(y_{w}|x)}{\pi_{\theta}( y_{l}|x)}\right)\right],$ (4) $$

where $\gamma$ is a hyperparameter and we set to $\gamma=1$ for simplicity. In line with Zhao et al. (), we incorporate a regularisation term into the hinge loss, defined as follows:

$$ $L_{\text{reg}}(\pi_{\theta};\pi_{\text{\text{ref}}})=\mathbb{E}_{(x,y_{w},y_{l })\sim D}\left[\log\left(1+\exp\left(1-\log\left(\frac{\pi_{\theta}(y_{w}|x)}{ \pi_{\text{\text{ref}}}(y_{w}|x)}\right)\right)\right)\right],$ (5) $$

which represents a smoothed version of hinge loss (Huber, ; Cristianini & Shawe-Taylor, ). This term encourages the likelihood of better completions to remain higher than that of the reference model.
The total hinge loss is given by $L_{\text{Hinge}}(\pi_{\theta};\pi_{\text{\text{ref}}})=L_{\text{Hinge}}(\pi_{
\theta};\pi_{\text{\text{ref}}})+\alpha L_{\text{reg}}(\pi_{\theta};\pi_{\text
{\text{ref}}})$,
where $\alpha$ is a scaling coefficient.

### 3.2 Better Likelihood Support

Standard DAAs do not guarantee an increase in the absolute probability of better completions. This can lead to scenarios where the model assigns very low probabilities to both better and worse completions, as long as the better completion has a higher relative probability.

#### Negative Log-Likelihood Loss.

To mitigate this issue, Negative Log-Likelihood (NLL) loss is commonly employed as a regularisation term in DAA (Hong et al., ; Pang et al., ; Adler et al., ; Dubey et al., ). It encourages the policy to maintain a high likelihood of better completions. The NLL loss is formulated as:

$$ $L_{\text{NLL}}(\pi_{\theta})=-\mathbb{E}_{(x,y_{w})\sim D}\left[\log\pi_{ \theta}(y_{w}|x)\right],$ (6) $$

where $y_{w}$ represents the better completion for a given input $x$. This loss term is typically combined with the primary objective of the DAA using a scaling coefficient $\lambda$.

Several other regularisation methods have been proposed to address this issue. For example, Pal et al. () introduces an additional term, $-\max\left(0,\log\frac{\pi_{\theta}(y_{w}|x)}{\pi_{\theta}(y_{l}|x)}\right)$, to $\Delta(x,y_{w},y_{l})$ to ensure that the log-likelihood of better examples remains high relative to that of the reference model. In this work, we mainly discuss the impact of Negative Log-Likelihood Loss.

## 4 Understanding the impact of Completion Likelihood

### 4.1 Experimental Setup

#### Model and Datasets.

In our experiments, we utilise two instruction-tuned models: Cohere Command R (7B) and Cohere Command R (35B) (Cohere For AI, ).
We train and evaluate them on two datasets:
(1) A binarised version of UltraFeedback (Tunstall et al., ), which is collected based on Cui et al. (), containing 62,600 training examples and 647 examples for evaluation.
(2) A Binarised preference dataset BinarizedPref, which comprises over 100,000 examples. These include annotated conversational data across multiple languages, synthetic code generation, and specialised tasks such as length control, safety, tool use, and natural language-to-SQL generation.

#### Training and Evaluation Details.

For each method (Hinge, DPO, and IPO), we test six different values for its hyper-parameter (*i.e., *$\alpha$, $\beta$, or $\tau$), respectively.
We use a batch size of 32 for both training and evaluation, with a maximum sequence length of 8192.
The model is trained with a peak learning rate of either $5\times 10^{-6}$ or $1\times 10^{-5}$ and an end learning rate ratio of 0.1.
Following recent studies (Ouyang et al., ; Howard & Whitaker, ; Shi et al., ), we train all models within a single epoch.
The learning rate warms up over 128 steps.
We monitor the model training every 50 steps to apply early stopping.
We use the Adam optimiser (Kingma, ) with $\beta_{1}=0.9$, $\beta_{2}=0.95$, $\epsilon=1\times 10^{-8}$, an additive weight decay of 0.1, and a gradient clipping norm of 1.0.
The model training is conducted on TPU v5-128 for the 7B model and TPU v5-256 for the 35B model, utilising the flash attention (Dao et al., ) to improve training efficiency.
For both DPO and IPO, we use the sum of the token log-likelihoods as the completion log-likelihood during training. For the Hinge method, we compute the average token log-likelihood instead for better performance.
During evaluation, we calculate the log-likelihood for both the better and worse completions from the validation set.
For all methods, we report the average of token log-likelihoods for better and worse completions respectively, without normalising against the reference model’s likelihood.
Additionally, we monitor the difference in log-likelihood between better and worse completions.

#### Generalisation Evaluation.

Following the previous work (Kirk et al., ), we evaluate the model in open-ended text generation tasks to assess generalisation ability.
Specifically, we employ the LLM-as-a-Judge framework (Zheng et al., ; Taori et al., ) based on the reward model to compare our models’ outputs against leading models, including GPT-3.5-Turbo, GPT-4o (Achiam et al., ), Claude-3-Sonnet (Claude, ), Llama-3-8B, and Llama-3-70B-Chat (Dubey et al., ).
The evaluation uses the closed-source reward model, which ranked the top position on RewardBench (Lambert et al., ).
This validates that the evaluation provides a reliable proxy for human preferences.
We use win probability, denoted as $P_{\text{win}}$, as the primary evaluation metric.
It is computed as:

$$ $P_{\text{win}}=\sigma(r_{v}-r_{c}),$ (7) $$

where $\sigma(\cdot)$ is the sigmoid function, $r_{v}$ is the reward assigned to the trained policy model’s output, and $r_{c}$ is the reward assigned to the competitor model’s output by the same reward model.
We prompt models with $433$ diverse prompts, including code generation, chain-of-reasoning questions, closed QA, and length control (see Appendix A for examples and more details). During the decoding, we use a top-$p$ probability threshold of $p=0.75$, a temperature of $0.5$, and a maximum limit of $2048$ tokens.

#### Diversity Evaluation.

To assess output diversity, we also measure Per-Input Diversity, defined as the average diversity of the output sets over inputs, and Cross-Input Diversity, which captures the diversity of outputs across different inputs, similar to previous works (Kirk et al., ; Hong et al., ).
However, instead of generating a set of $K$ outputs from the model, we take a more efficient way to measure Per-Input Diversity. Specifically, we compute the entropy over the top $k$ tokens with the highest probability in the model’s next token distribution (Kuhn et al., ).
Let $p_{k}$ represent the probability distribution over the top $k$ tokens, and $H(p_{k})$ represent the entropy of the distribution. The entropy is calculated using the following formula:

$$ $H(p_{k})=-\sum_{i=1}^{n}p_{i}\log_{b}(p_{i}),$ (8) $$

where $b$ is the logarithm base.
Here we set $b=2$ and $k=10$.
This formula quantifies the uncertainty within the top $k$ token predictions as a proxy for Per-Input Diversity.
This entropy is highest when the output is minimally informative: predicting the same probability for all possible tokens, indicating more diverse outputs.
To evaluate Cross-Input Diversity, we use distinct N-grams (Li et al., ), which counts the unique N-grams across model outputs and averages them over $n=1,2,3,4,5$.
Following Kirk et al. (), we use the expectation-adjusted distinct N-grams (EAD) formula to remove the bias towards shorter outputs.

Figure: Figure 2: Learning curves across training steps for various metrics. Results are reported for the 7B models using the Hinge, DPO, and IPO on the UltraFeedback dataset.
Refer to caption: x2.png

#### Factuality Evaluation.

We also evaluate model factuality performance on open-domain question-answering tasks using NaturalQuestionsOpen (Kwiatkowski et al., ) and TriviaQA (Joshi et al., ) validation sets, with 3610 and 7993 examples respectively. Greedy decoding is used to ensure deterministic outputs, and the word-level F1 score is reported.

### 4.2 Evaluating Likelihood Over-optimisation

In this section, we explore the relationship between model likelihood and performance. Below, we discuss our key findings in detail.

Figure: Figure 3: Win Probability Heatmaps Across Better and Worse Mean Log-Likelihoods. Results are reported for both 7B and 35B models on UltraFeedback and BinarizedPref datasets. Best performance does not always occur at the Pareto frontier of high likelihood for better completions and low likelihood for worse completions.
Refer to caption: x3.png

#### 1) Higher likelihood for better completions and larger gaps between better and worse completions do not necessarily improve model performance.

As shown in Figure 1, we plot the likelihood of better completions against the win probability (compared to GPT-3.5-Turbo) with different methods across two model sizes, with points recorded every 500 steps.
Our analysis reveals that simply increasing the likelihood of better completions does not consistently result in performance improvements.
Previous work in classical RLHF has established scaling laws for reward model scores (Gao et al., ).
Similarly, Figure 1 exhibits a clear scaling law behaviour.
We extend their analysis to the relationship between win probability and the log-likelihood of better completions in DAAs.
When fitting the data to a second-degree polynomial, the Root Mean Square Error decreases by approximately 24.42% for the 7B model and 25.78% for the 35B model, compared to a linear fit.
We show similar results when comparing against different competitor models, including GPT-4o, Claude-3-Sonnet, Llama-3-8B, and Llama-3-70B-Chat, in Figure 7 of Appendix §B.

Figure: Figure 4: Learning curves for DPO with different weights ($\lambda$) of NLL loss. We report the performance with different values of $\beta$ and $\lambda$ on the UltraFeedback dataset. A reversed entropy trend trending for entropy is a strong indicator of diversity over-optimisation.
Refer to caption: x4.png

Figure 2 tracks win probability alongside the average log-likelihood difference between better and worse completions throughout training.
Notably, while larger differences in log-likelihood, such as those represented by the pink line typically with the largest difference, are often observed, they do not correspond to better performance.
Instead, excessively larger likelihood gaps can lead to performance degradation in win probability, especially for DPO and IPO after 1,000 steps.
We observe similar results for the 35B model on BinarizedPref using Hinge, DPO, and IPO in Appendix §B.

Figure 3 presents a heatmap of win probabilities based on the better and worse completion log-likelihoods on UltraFeedback and BinarizedPref datasets, using both 7B and 35B models.
Points are plotted every 50 steps.
Our findings indicate that the best performance (highlighted by the red star) does not occur at the Pareto frontier of maximising the likelihood of better completions while minimising it for worse ones. Instead, optimal performance is often found in the middle range.

#### 2) Length Correlation.

We investigate the relationship between the mean log-likelihood of better completions and the average number of tokens in completions, as shown in Figure 1.
To quantify this relationship, we calculate the Pearson correlation coefficient and perform its associated significance test.
The null hypothesis posits no linear relationship between these two variables.
For the 7B model, we find a weak negative correlation ($r$ = −0.114, $p$-value = 0.266), while the 35B model shows a weak positive correlation ($r$ = 0.198, $p$-value = 0.173).
In both cases, the p-values exceed the conventional significance level of 0.05, indicating insufficient evidence to reject the null hypothesis.

#### 3) Training Negative Log-Likelihood Loss on better completions has limited influence on the model when it cannot affect completion likelihood.

As shown in Figure 4, we experiment with DPO using three different values of $\beta$, adding NLL loss as an auxiliary loss with four $\lambda$ coefficients.
Our results indicate that when there is limited impact on the likelihood (from the left column to the right column), the NLL loss has minimal impact on model performance.
This suggests that NLL loss can be seen as a tool to regulate completion likelihood, but it remains susceptible to likelihood over-optimisation: higher likelihood may lead to a sub-optimal performance.
We observe similar results on BinarizedPref using the 35B model, as shown in Figure 11 of Appendix §B.

Figure: Figure 5: NaturalQuestionsOpen and TriviaQA vs Better Mean LLH on the UltraFeedback dataset. A higher LLH tends to memorise the factuality knowledge better.
Refer to caption: x5.png

### 4.3 Generalisation and Diversity

In this section, we explore the impact of model likelihood on generalisation and diversity.

#### 1) Lower Completion likelihood improves the models’ Cross-Input Diversity .

Figure 2 presents Cross-Input Diversity (measured by distinct N-grams) of the model outputs throughout training.
Specifically, within each DAA, models with lower likelihood tend to produce more diverse outputs.
For example, the pink lines for DAAs indicate that models with lower completion likelihood typically show the highest level of Cross-Input Diversity scores throughout training.
Better output diversity tends to improve their generalisation to unseen scenarios, as reflected in increased win probability at the early stage of the training phase.
Figure 4 further demonstrates that output diversity follows a similar trend under the different regularisation (*i.e., *Negative Log-Likelihood Loss), suggesting a strong correlation between likelihood and model diversity.
However, it is worth noting that the relationship between diversity and win probability is not linear. While some diversity is beneficial for generalisation, excessive diversity can lead to performance degradation, similar to our previous discussion in §4.2.
We will explore this phenomenon further in §4.4.

#### 2) Higher Likelihood tends to have better memorisation of factual patterns.

Figure 5 showcases the relationship between model performance on NaturalQuestionsOpen and TriviaQA and the log-likelihood of better completions.
Our findings reveal a clear trend: higher mean log-likelihood values are associated with improved F1 scores.
A higher F1 reflects better memorisation for some specific patterns, which can come at the expense of diversity.
This can create a trade-off between the ability to recall facts and the capacity to generate diverse, adaptive outputs in more creative or open-ended tasks.
To understand the potential issue of stylistic variations in answers, we provide a further analysis with case studies and LLM-as-a-Judge as evaluation in Appendix §C.
Specifically, instead of relying on exact string matching, which can be overly rigid, we employ an LLM-as-a-Judge using the GPT-4o model.
Our analysis reveals that while the model performance from LLM-as-a-Judge evaluation consistently yields higher performance metrics,
it demonstrates a trend similar to the F1 score.

### 4.4 Signals for Likelihood Over-optimisation

We have demonstrated that a slight decrease in the likelihood of better completion correlates with improved performance, which can be explained by an increase in output diversity. However, the critical question remains: when should we stop lowering the likelihood of better completion? In this section, we outline two key indicators that signal the over-optimisation of likelihood.

#### 1) Decreasing Entropy over Top- k 𝑘 k italic_k tokens ( Per-Input Diversity ).

Figure 2 and 4 presents Per-Input Diversity (measured by the entropy) of the model outputs throughout training.
For DPO and IPO curves, at the beginning of the training, the Per-Input Diversity increases, signifying a broader distribution of selected tokens and a more uniform output distribution for the next token prediction.
Considering that the better completion likelihood is decreasing across the training, the increase of entropy at the beginning phase indicates that those tokens from better completion have a higher probability at the initial policy model over other tokens in the top $k$ (here $k=10$).
The decrease better completion likelihood gives the model a better chance to select other tokens, which increases diversity and enhances generalisation, as reflected in the win probability.
However, at a certain point in training, this trend reverses.
As Per-Input Diversity (entropy) starts decreasing, the model begins to over-prioritise certain tokens.
This suggests that those tokens in the better completion now have an overly low likelihood, lower than other tokens in the top $k$.
Despite this, Cross-Input Diversity keeps increasing, which indicates that the model is still generating diverse outputs, but now it includes tokens that are less relevant or nonsensical, *i.e., *tokens that humans do not prefer.
Notably, the turning points of entropy often coincide with those of win probability for DPO and IPO, as the model’s outputs become less aligned with desirable outcomes.

#### 2) Decreasing in Probability Mass in Top k 𝑘 k italic_k Tokens.

In another scenario, the entropy of the top 10 tokens continues to increase, suggesting a progressively broader and more uniform output distribution (refer to the hinge curves in Figure 2).
This suggests that even as the likelihood of better completions decreases, the model does not tend to over-prioritise any specific tokens during training.
However, this can result in degraded model performance.
As depicted in the bottom row of the figure, the probability mass of all top-$10$ tokens diminishes, leading to more random outputs, with an increased likelihood of selecting tokens outside the top $10$.
This can introduce issues such as code-switching, where the model becomes prone to world-level language confusion when the number of tokens in the sampling nucleus is high and the distribution becomes too flat (Doğruöz et al., ; Marchisio et al., ).
Interestingly, hinge loss models do not exhibit the same patterns observed with DPO and IPO. This could be attributed to the fact that DPO and IPO apply different forms of regularisation compared to hinge loss.

To demonstrate the generalisability of our findings, we provide additional experimental on different datasets with different model sizes in Figure 8, 9, and 10 of Appendix §B.

## 5 Epilogue

#### Limitations.

This study primarily focuses on two models (7B and 35B), which may not fully represent the broader spectrum of LLMs available.
However, most LLMs are very standard transformers (Vaswani et al., ), and we would not expect other LLMs to behave differently.

#### Implications for Practical Applications.

The findings of this study have several implications for enhancing offline preference learning methods in practical applications:
(1) Early stopping signal.
In practice, we can integrate entropy/probability mass monitoring into the training loop.
Training can employ adaptive methods like early stopping once entropy falls below a specific threshold.
(2) Adaptive regularisation for over-optimisation.
Rather than using a fixed coefficient for the NLL loss (Dubey et al., ), we could implement an adaptive regularisation based on the entropy and probability mass, *i.e., *adding dropout or noise to prevent over-prioritisation of tokens or adding an explicit regularisation term that maintains a certain degree of entropy and the probability mass of the top-$k$ tokens.
While maintaining a certain degree of entropy and probability mass of the top-$k$ tokens is important, care should be taken not to overly constrain the model, as some tasks inherently require a broader token distribution (*e.g., *give me a random number between 0 0 and $10$).

## Reproducibility Statement

To ensure the reproducibility of our results, we have taken comprehensive steps to provide detailed information about our experimental setup. In Section 4.1, we offer full details on the models used (7B and 35B parameter models) and the datasets (UltraFeedback and BinarizedPref), including exact versions and sizes.
While the 7B model and reward model are closed-source, and the 433 prompts for the LLM-as-a-Judge framework are proprietary, we provide a summary of the prompt dataset to give insight into its composition.
All hyperparameters for training, including learning rates, batch sizes, and optimizer settings, are specified.
We detail the hardware used (TPU v5-128/256) and provide comprehensive descriptions of all evaluation metrics.
Statistical analyses, including Pearson correlation coefficients and p-values, are reported in Section 4.2.
The UltraFeedback dataset is publicly available, and while BinarizedPref is proprietary, we describe its contents and size.
Importantly, we test our findings on UltraFeedback, which is a public dataset, indicating that our findings are generalisable.
While some aspects could not be fully open-sourced due to the use of proprietary models or data, we have described these in as much detail as possible.
Furthermore, we posit that our findings are likely generalisable to other LLMs, as most LLMs (*e.g., *Llama, Gemini) are based on standard transformer architectures (Vaswani et al., ).
For example, the Llama model family has very standard features such as RoPE embeddings (Su et al., ).
Indeed, the designers note that they tried to avoid innovating on the model architecture (Dubey et al., ).
As such, we would not expect significantly different behaviours. We welcome questions from the community and are committed to providing additional clarification.

## Appendix Overview

The appendix is structured as follows:

#### Appendix § A

provides a detailed description of evaluation datasets, including examples and statistical summaries.

#### Appendix § B

presents supplementary experimental results, including analyses of win probability, likelihood scaling, and the effects of different regularization techniques.

#### Appendix § C

further investigates model performance on NaturalQuestionsOpen and TriviaQA.

## Appendix A Evaluation Datasets

This section provides an in-depth look at the datasets used in our experiments, focusing on the LLM-as-a-Judge framework, NaturalQuestionsOpen, and TriviaQA datasets.

#### LLM-as-a-Judge Framework Dataset.

We utilize a diverse set of prompts for the LLM-as-a-Judge framework. Figure 1 illustrates a representative example from this dataset, showcasing different generations from various competitor models. To provide insight into the composition of our LLM-as-a-Judge dataset, Figure 6 presents the distribution of prompt examples. This visualisation helps to understand the variety and balance of the prompts used in our evaluation framework.

Table: Table 1: Examples for datasets.

Figure: Figure 6: Distribution of LLM-as-the-judge prompt dataset.
Refer to caption: x6.png

#### NaturalQuestionsOpen Dataset.

Table 2 presents examples from the NaturalQuestionsOpen dataset(^2^22[https://huggingface.co/datasets/google-research-datasets/nq_open/viewer/nq_open/validation](https://huggingface.co/datasets/google-research-datasets/nq_open/viewer/nq_open/validation)), showcasing the types of questions and answers used in our evaluation.
The NaturalQuestionsOpen dataset, introduced by Kwiatkowski et al. (), is an open-domain question-answering benchmark.
It consists of English questions paired with possible answer strings, all answerable using English Wikipedia content.
Each data instance contains a question field and an answer field with potential correct responses.
We use the validation set for our evaluation.
Table 2 presents representative examples from this dataset, illustrating the types of questions and answers used in our evaluation.

Table: Table 2: Examples for NaturalQuestionsOpen.

Table: Table 3: Examples for TriviaQA.

#### TriviaQA Dataset.

The TriviaQA dataset(^3^33[https://huggingface.co/datasets/mandarjoshi/trivia_qa/viewer/rc.wikipedia/validation](https://huggingface.co/datasets/mandarjoshi/trivia_qa/viewer/rc.wikipedia/validation)) is a comprehensive reading comprehension benchmark containing over 650,000 question-answer-evidence triples Joshi et al. ().
It includes 95,000 question-answer pairs, each accompanied by an average of six independently gathered evidence documents. This structure provides high-quality distant supervision for question-answering tasks.
However, we do not use any evidence in our experiments.
We use the validation set for our evaluation.
Table 3 presents representative examples from the TriviaQA dataset.

## Appendix B Additional Experimental Results

As supplementary of the main experiment, we provide the following experiments.

#### Win Probability vs. Better Completion Likelihood.

Figure 7 illustrates the relationship between win probability and better mean likelihood across different competitor models, including GPT-4, Claude-3-Sonnet, Llama-3-8B, and Llama-3-70B-Chat.
We record points every 500 steps across varying hyperparameters for each method.
Our results are consistent with our findings in the main text (§4.2), suggesting that simply increasing the likelihood of better completions does not consistently result in performance improvements.

Figure: Figure 7: Win Probability vs Better Mean Likelihood Scaling Law. with different competitor models. including GPT-4o, Claude-3-Sonnet, Llama-3-8B, and Llama-3-70B-Chat
Refer to caption: x7.png

Figure: Figure 8: Learning curves across training steps for various metrics. Results are reported for the 7B models using IPO on the UltraFeedback dataset with varying values of $\tau$ and $\lambda$.
Refer to caption: x8.png

Figure: Figure 9: Learning curves across training steps for various metrics. Results are reported for the 7B models using the Hinge, DPO, and IPO on the BinarizedPref dataset.
Refer to caption: x9.png

Figure: Figure 10: Learning curves across training steps for various metrics. Results are reported for the 35B models using the Hinge, DPO, and IPO on the BinarizedPref dataset.
Refer to caption: x10.png

#### IPO Learning curves with 7B model on the UltraFeedback dataset.

To demonstrate the generalisability of our findings, we experiment with the IPO using three different values of $\tau$, adding NLL loss as an auxiliary loss with four $\lambda$ coefficients on the UltraFeedback dataset using the 7B model.
Figure 8 illustrates several key findings:

- 1.
Likelihood and Performance Correlation: As shown in the first and second rows of the figure, a Higher likelihood for better completions and larger gaps between better and worse completions do not necessarily translate to improved model performance.
- 2.
Likelihood and Cross-Input Diversity: Lower completion likelihood tends to enhance the models’ Cross-Input Diversity, as shown in the second and fourth rows, where lower better completion likelihood generally corresponds to improved Cross-Input Diversity.
- 3.
Entropy and Over-optimisation: Decreasing entropy over top-$k$ tokens (Per-Input Diversity) appears to be an indicator of over-optimisation for diversity. The fifth row demonstrates that curves with lower entropy typically do not perform as well, as reflected in their win probability. Particularly, this result shows that the turning points of the entropy, which transits from the increasing diversity to the decreasing entropy is a strong indicator of the over-optimisation for diversity.
- 4.
Probability Mass Distribution: We do not observe a decrease in probability mass in top $k$ tokens in this case, as shown in the last row of the figure. This observation aligns with our findings: in runs without decreasing entropy, we do not observe a significant decline in win probability.

#### Learning curves with 7B model on the BinarizedPref dataset.

To demonstrate the generalisability of our findings, we perform additional experiments using the 7B model on the BinarizedPref dataset.
The results, consistent with our previous observations, underscore the broad applicability of our insights across various datasets.
Figure 9 illustrates several key findings:

- 1.
Likelihood and Performance Correlation: Higher likelihood for better completions and larger gaps between better and worse completions do not necessarily translate to improved model performance. This is evident in the first and second rows of the figure, where models with the highest better completion likelihood do not achieve the best performance.
- 2.
Likelihood and Cross-Input Diversity: Lower completion likelihood tends to enhance the models’ Cross-Input Diversity. This trend is observable when comparing the second and fourth rows, where lower better completion likelihood generally corresponds to improved Cross-Input Diversity.
- 3.
Entropy and Over-optimisation: Decreasing entropy over top-$k$ tokens (Per-Input Diversity) appears to be a good indicator of over-optimisation for diversity. The fifth row demonstrates that curves with overly low entropy do not perform as well (*i.e., *pink curves), as reflected in their win probabilities. Additionally, as the entropy begins to rise again, an improvement in win probability is also observed.
- 4.
Probability Mass Distribution: We do not observe a decrease in probability mass in top $k$ tokens in this case, as shown in the last row of the figure. This observation aligns with our findings: in runs without decreasing entropy, we do not observe a significant decline in win probability.

#### Learning curves with 35B model on the BinarizedPref dataset.

To demonstrate the generalisability of our findings, we perform additional experiments using the 35B model on the BinarizedPref dataset.
The results align well with our previous observations.
Figure 10 illustrates several key findings:

- 1.
Likelihood and Performance Correlation: Similarly, results from larger model sizes suggest that higher likelihoods for better completions and larger gaps between better and worse completions do not necessarily lead to improved model performance, as shown in the first and second rows of the figure.
- 2.
Likelihood and Cross-Input Diversity:
Lower completion likelihood tends to enhance the models’ Cross-Input Diversity. Specifically, the curve with a lower better completion likelihood generally tends to have a higher Cross-Input Diversity.
- 3.
Entropy and Over-Optimisation: A decrease in entropy over the top-$k$ tokens (Per-Input Diversity) appears to indicate over-optimisation for diversity. For instance, the pink lines for DPO and IPO show a clear drop in entropy after 500 steps, accompanied by a decline in win probability.
- 4.
Probability Mass Distribution: Similarly, we do not observe a decrease in probability mass in top $k$ tokens in this case, as shown in the last row of the figure.

#### Training Negative Log-Likelihood Loss on better completions has limited influence on the model when it cannot affect completion likelihood.

To demonstrate the generalisability of our findings, we perform further experiments with 35B models on the BinarizedPref dataset.
As shown in Figure 11, we experiment with DPO using three different values of $\beta$, adding NLL loss as an auxiliary loss with four distinct coefficients for each $\beta$.
Similarly to our findings in the main text, results indicate that when there is limited impact on the likelihood, the NLL loss has minimal impact on model performance.
Training Negative Log-Likelihood Loss on better completions remains susceptible to over-optimisation.

Figure: Figure 11: Control Likelihood via training on better completion on the BinarizedPref dataset, using the 35B model. When different runs have similar likelihoods, the win probability and diversity of their model outputs tend to follow the same trend throughout training.
Refer to caption: x11.png

Table: Table 4: Examples for TriviaQA.

Table: Table 5: Model output examples for NaturalQuestionsOpen and TriviaQA.

## Appendix C Further investigations for Question Answering Tasks

#### Case studies for NaturalQuestionsOpen and TriviaQA tasks.

Table 5 provides two examples for NaturalQuestionsOpen and TriviaQA tasks, respectively.

Figure: Figure 12: NaturalQuestionsOpen vs Better Mean LLH on the UltraFeedback dataset using the 7B model. The F1 score and LLM-as-a-Judge results are reported.
Refer to caption: x12.png

#### LLM-as-a-Judge for the NaturalQuestionsOpen task.

We implement a more flexible evaluation method to understand the potential issue of stylistic variations in answers.
Instead of relying on exact string matching, which can be overly rigid, we employ an LLM-as-a-Judge using the GPT4o model.
As shown in Table 4, this LLM-based evaluation system is presented with the original question, the reference answer, and the model’s output.
It then assesses whether the model’s output is correct, incorrect, or if there’s not enough information to make a determination, responding with “Yes”, “No”, or “Unsure” respectively.
We compute the model performance based on the percentage of “Yes”.
Figure 12 shows the model performance on the UltraFeedback dataset using the 7B model.
Our analysis reveals that while the LLM-as-a-Judge evaluation method demonstrates a trend similar to the F1 score, it consistently yields higher performance metrics.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 6
presentation: 4
questions: * Could you provide more information about BINARIZEDPREF construction, including how preferences were collected and validated? The source of preferences is particularly important.
* Why was single-epoch training chosen when DPO typically needs more epochs? Do these patterns persist in multi-epoch training?
soundness: 3
strengths: * The work provides comprehensive experiments across multiple dimensions including likelihood, diversity and performance. The ablation studies are systematic and well-documented.
* The paper challenges common assumptions about likelihood optimization in DAAs. The trade-off between memorization and generalization is clearly demonstrated with empirical evidence.
* The proposed metrics for detecting over-optimization are concrete and actionable. The findings provide clear guidance for practitioners working on DAA training.
summary: * The paper studies how completion likelihood affects model performance in Direct Alignment Algorithms (DAAs) like DPO, IPO and Hinge loss, using 7B and 35B models.
* Key finding shows that higher likelihood of better completions and larger margins between better/worse completions don't necessarily improve performance. Higher likelihood helps with factual recall but can hurt diversity.
* Authors propose two metrics for detecting likelihood over-optimization: Decreasing Entropy over Top-k Tokens and Diminishing Top-k Probability Mass. These metrics help prevent over-optimization while maintaining good performance.
weaknesses: * The paper only studies single epoch training when DPO typically needs 2-3 epochs for best performance. This important limitation is not well justified or analyzed.
* The BINARIZEDPREF dataset lacks crucial details about its construction and preference collection. If preferences come from GPT-4 rather than humans, the generalization of findings is questionable.
* The primary evaluation uses GPT-3.5-turbo as baseline which feels dated. Testing against stronger models like LLaMA-3 would strengthen the findings.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: Lower entropy usually indicates that a model is overfitted. Could this fact indicate that the UltraFeedback dataset is not sufficiently diverse, and therefore the model could memorize patterns that are not useful in general? Does the issue of over-optimization hold across different datasets?
soundness: 2
strengths: - Evaluation details are comprehensive.
- This work is useful for the community as it could be used to determine offline metrics to identify the best model at various steps.
summary: This paper presents a detailed analysis of contemporary offline alignment methods, focusing on DPO, IPO, and Hinge. A comprehensive set of metrics, gathered during the alignment process for both a proprietary 7B model and the Cohere Command R 35B model, is employed to investigate the issue of over-optimization. Furthermore, the paper proposes using the entropy of the top-k tokens during generation for DPO and IPO, as well as the aggregate mass of the top-k tokens for Hinge, as indicators of over-optimization.
weaknesses: - Over-optimization itself is already well-studied in [1]. Some results, like Figure 1, seem to be consequences of the high KL divergence. I think NLL for y_w and KL are highly correlated metrics. It would be useful to see a figure with KL on the x-axis and NLL(y_w) on the y-axis.
- While Figure 1 supports the claim that there is no correlation between NLL(y_w) and win rate in general, Figure 2 shows that, for the given method and hyperparameters, the best step can be determined by observing NLL(y_w). More precisely, I would suggest tracking the difference between NLL from the previous and current checkpoints. If this value becomes larger than a certain threshold, one can stop training and thereby obtain the strongest checkpoint. Therefore, from the reported I would say that tracking NLL(y_w) could be sufficient to detect over-optimization across all methods (instead of proposed methodology in Section 4.4, which depends on method).
- The paper presents an extensive amount of metrics during the training procedure; however, there is no clear criterion to detect over-optimization. First of all, the proposed methods in Section 4.4 are highly dependent on alignment methods, and if one were to use other popular methods (like KTO, ORPO, etc.), it is not clear which metric would serve as a flag for over-optimization. Additionally, it is unclear if these results remain the same for different models (see next weakness point).
- The choice of models is in question. Results obtained from a closed-source 7B model may not generalize to widely used open-weight models with similar sizes like LLAMA, Gemma, or Mistral. Additionally, this has a negative impact on the reproducibility of the experiments.

[1] Scaling Laws for Reward Model Over-optimization in Direct Alignment Algorithms (Rafailov et al.)

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: - In Figure 3, why the DPO/IPO 7B ultrafeedback model cannot gain much improvement compared with initial checkpoint?
soundness: 3
strengths: ## Originality
- Though the claim may not be novel, the systematic experiments are original and interesting.
- The reviewer appreciates the statistical metrics.

## Clarity
- This paper is well-written and densely organised.
- The figures are clearly plotted.

## Significance
- This paper may be able to clarify a controversial point, whether DAA should directly increase(decrease) the likelihood of accepted(rejected) completion? which the reviewer believes is a important question.
summary: This paper provides a experimental analysis of the behaviors of margin-based alignment approaches, especially, they focus on the so-call over-optimisation issue. They claim that they are the first to explore the relationship between completion likelihood and performance in alignment algorithms. Their empirical finding is that, likelihood does not have a striong positive correlation with model performance, and it might be affected by diversity. Besides, they identify two indicators of overly generatinve diverse outputs.
weaknesses: ## Major
- The two indicators are so trivial, that people already know this before. Thus these two findings might not be counted as contributions.
- The experiments are restricted. Some DAAs with alternate objective (CPO, SimPO, ...) (such as length normalization) are not tested, making the claim less solid.

## Minor
- The curves are a bit confusing, especially Figure 4, making it hard to come to the authors' conclusion.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 1
questions: See Weaknesses
soundness: 2
strengths: - The initial problem of overoptimization is important, and solving it is crucial for the field of alignment.
- Understanding signals of overoptimization is helpful and allows for faster evaluation. Instead of waiting for evaluation with LLMs, we can detect overoptimization through reduced entropy and decreasing mass in top-k tokens.
summary: The authors explored the effect of increasing the probability of chosen sequences on the overoptimization of direct alignment algorithms (DAA). They concluded that an increased gap between chosen and rejected sequences could lead to overoptimization and linked overoptimization with reduced diversity of samples.
weaknesses: - While the authors provided implementation details, it is unclear why they used a 7B model with closed weights when a wide range of open-source models of this size is available.
- My main concern is the limited novelty of the obtained results. From what I see in Figure 2, while the gap between chosen and rejected sequences is increasing for small $\beta$, the likelihood of chosen sequences is decreasing. This aligns with the observations of [1], as probability mass moves towards out-of-distribution (OOD) examples (and to avoid overoptimization, we should prevent leakage of mass to OOD sequences). Therefore, the findings do not seem to present new insights. When demonstrating that increased probabilities for chosen sequences can also lead to overoptimization, it is important to explore the probabilities associated with OOD examples. From this perspective, the main novelty of the paper lies in understanding the mechanism of detecting overoptimization via reduced diversity and aligning these observations with the impact on performance across various tasks, but this aspect is poorly explored.
- The presentation of results is difficult to read. The captions of large figures (2, 3, 4) do not contain useful information for understanding, making some metrics hard to interpret. Additionally, the analysis of these results is far from the figures themselves.

[1] Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms, Rafailov et. al.

Meta-review: This paper studies the relationship between completion likelihood and model performance in Direct Alignment Algorithms (DAAs), analyzing how increased likelihood of better completions and margins between better/worse completions affect performance. The paper claims that higher likelihood doesn't necessarily improve performance and can hurt diversity, proposing two metrics (Decreasing Entropy over Top-k Tokens and Diminishing Top-k Probability Mass) for detecting likelihood over-optimization. The key strengths include comprehensive experiments across multiple dimensions and clear implications for practitioners working on DAA training. However, major weaknesses include: (1) limited novelty, as the observed patterns appear to be consequences of KL divergence effects already documented in prior work; (2) restricted experimental setup using proprietary models/datasets that limits reproducibility; and (3) lack of systematic validation across a broader range of models, datasets and methods. The paper is recommended for rejection primarily because the consensus from the reviewers that it fails to convincingly demonstrate that its findings represent a new phenomenon distinct from known scaling laws of alignment methods.

Decision: Reject