Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization
Authors: Noam Razin
Sadhika Malladi
Adithya Bhaskar
Danqi Chen
Sanjeev Arora
Boris Hanin
Venue: ICLR 2025 Poster (2025)
Keywords: Direct Preference Optimization
DPO
Likelihood Displacement
Unalignment
Alignment
Language Models
Primary area: foundation or frontier models, including LLMs

Abstract: Direct Preference Optimization (DPO) and its variants are increasingly used for aligning language models with human preferences. 
Although these methods are designed to teach a model to generate preferred responses more frequently relative to dispreferred responses, prior work has observed that the likelihood of preferred responses often decreases during training. The current work sheds light on the causes and implications of this counterintuitive phenomenon, which we term *likelihood displacement*. We demonstrate that likelihood displacement can be *catastrophic*, shifting probability mass from preferred responses to responses with an opposite meaning. As a simple example, training a model to prefer $\texttt{No}$ over $\texttt{Never}$ can sharply increase the probability of $\texttt{Yes}$. Moreover, when aligning the model to refuse unsafe prompts, we show that such displacement can *unintentionally lead to unalignment*, by shifting probability mass from preferred refusal responses to harmful responses (e.g., reducing the refusal rate of Llama-3-8B-Instruct from 74.4% to 33.4%). We theoretically characterize that likelihood displacement is driven by preferences that induce similar embeddings, as measured by a *centered hidden embedding similarity (CHES)* score. Empirically, the CHES score enables identifying which training samples contribute most to likelihood displacement in a given dataset. Filtering out these samples effectively mitigated unintentional unalignment in our experiments. More broadly, our results highlight the importance of curating data with sufficiently distinct preferences, for which we believe the CHES score may prove valuable.

Published as a conference paper at ICLR 2025
UNINTENTIONAL UNALIGNMENT: LIKELIHOOD DISPLACEMENT IN DIRECT PREFERENCE OPTIMIZATION
Noam Razin', Sadhika Malladi', Adithya Bhaskar', Danqi Chen',
Sanjeev Aroral, Boris Hanin
Princeton Language and Intelligence, Princeton University $ Princeton ORFE
ABSTRACT
Direct Preference Optimization (DPO) and its variants are increasingly used for aligning language models with human preferences. Although these methods are designed to teach a model to generate preferred responses more frequently rel- ative to dispreferred responses, prior work has observed that the likelihood of preferred responses often decreases during training. The current work sheds light on the causes and implications of this counterintuitive phenomenon, which we term likelihood displacement. We demonstrate that likelihood displacement can be catastrophic, shifting probability mass from preferred responses to responses with an opposite meaning. As a simple example, training a model to prefer No over Never can sharply increase the probability of Yes. Moreover, when align- ing the model to refuse unsafe prompts, we show that such displacement can mn- intentionally lead to unalignment, by shifting probability mass from preferred re- fusal responses to harmful responses (e.g., reducing the refusal rate of Llama-3- 8B-Instruct from 74.4% to 33.4%). We theoretically characterize that likelihood displacement is driven by preferences that induce similar embeddings, as mea- sured by a centered hidden embedding similarity (CHES) score. Empirically, the CHES score enables identifying which training samples contribute most to likeli- hood displacement in a given dataset. Filtering out these samples effectively mit- igated unintentional unalignment in our experiments. More broadly, our results highlight the importance of curating data with sufficiently distinct preferences, for which we believe the CHES score may prove valuable."
INTRODUCTION
To ensure that language models generate safe and helpful content, they are typically aligned based on pairwise preference data. One prominent alignment method, known as Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al., 2022), requires fitting a reward model to a dataset of human preferences, and then training the language model to maximize the reward via RL. While often effective for improving the quality of generated responses (Bai et al., 2022a; Achiam al., 2023: Touvron et al., 2023), the complexity and computational costs of RLHF motivated the rise of direct preference learning methods such as DPO (Rafailov et al., 2023).
Given a prompt x, DPO and its variants (e.g., Azar et al. (2024); Tang al. (2024); Xu et al. (2024a); Meng et al. (2024)) eschew the need for RL by directly teaching model me to increase the margin between the log probabilities of a preferred response y+ and a dispreferred response y While intuitively these methods should increase the probability of y while decreasing that of y several recent works observed that the probabilities of both y and y tend to decrease over the course of training (Pal et al., 2024; Yuan al., 2024; Rafailov et al., 2024b; Tajwar et al., 2024; Pang al., 2024; Liu et al., 2024). We term this phenomenon likelihood displacement - see Figure
When the probability yt decreases, the probability of other, possibly undesirable, responses must increase. However, despite the prevalence of likelihood displacement, its causes and implications remain poorly understood. The purpose of this work is to address these gaps. Through theory and experiments, we characterize mechanisms driving likelihood displacement, demonstrate that it can
Our code available ht:pp: /tt.///tttttt// www.thh....pp.....ooooo www.pppootttoooooooooooooooooo.o on--ntottnnnnno haaiinnp//unn itent lona I-una 1 ignment..

---

Published as a conference paper at ICLR 2025
Direct Preference Learning DPO (Rafallov al. 2023)
Likelihood Displacement
Benign Z preferable y
Z
In mo (xc)
Language Model m
Catastrophic
Z substantially less preferable than y
Preference Learning Loss
Training Steps
Figure Illustration of likelihood displacement in direct preference learning. For a prompt direct preference learning aims to increase probability that model me assigns a preferred response y* relative dispreferred response y Likelihood displacement refers to the counterintuitive phenomenon where, while the gap between In (y+ [y*+ax (x) and ln (y [[x) increases, they both decrease. If the responses increasing instead in probability (depicted by 2) are as preferable as y* (e.g., is similar in meaning to *), then the likelihood displacement is benign. However, if the probability mass goes to responses that are substantially less preferable than y (e.g.,2 is opposite meaning to y+),, ), then say that it is catastrophic.
lead to surprising failures in alignment, and provide preventative guidelines. Our experiments cover models of different families and scales, including OLMo- IB (Groeneveld et al., 2024), Gemma-2B (Team al., 2024), and --ama-3-8B (Dubey et al., 2024). The main contributions are listed below.
Likelihood displacement can be catastrophic even in simple settings. We demonstrate that, even when training on just single prompt whose preferences y* and y consist of a single token each, likelihood displacement is pervasive (Section 3). Moreover, the tokens increasing in probability at the expense of y can have a meaning opposite to For example, training a model to prefer y No over y Never often sharply increases the probability of Yes. This stands in stark contrast to prior work attributing likelihood displacement to different com- plexities in the preference learning pipeline (Tajwar et al., 2024; Pal et al., 2024; Rafailov al., 2024b), and emphasizes the need to formally characterize its underlying causes.
Theory: likelihood displacement is determined by the model's embedding geometry. We analyze the evolution of In To (y x) during gradient- training (Section 4). Our theory reveals that likelihood displacement governed by the (static) token unembeddings and (con- textual) hidden embeddings of y and y In particular, it formalizes intuition by which the more similar y and y the more In (y x) tends to decrease.
Identifying sources of likelihood displacement. Based our analysis, derive (model- aware) measure similarity between preferences, called the centered hidden embedding simi- larity (CHES) score (Definition 2). We demonstrate that the CHES score accurately identifies which training samples contribute most to likelihood displacement traFeedback (Cui et al.. 2024) and AlpacaFarm (Dubois et al., 2024)). whereas other similarity measures relying on hidden embeddings or token-level cues do not (Section 5).
a given dataset (e.g., UI-
Unintentional unalignment due likelihood displacement. To demonstrate the potential uses of the CHES score, we consider training language model to refuse unsafe prompts via direct preference learning (Section 6). We find that likelihood displacement can unintentionally unalign the model, by causing probability mass to shift from preferred refusal responses to re- sponses that comply with unsafe prompts! For example, the refusal rate of Lllam----B--nssrruee drops from 74.4% to 33.4% over the SORRY-Bench dataset (Xie et al., 2024b). We then show that filtering does so more effectively than adding a supervised finetuning term to the loss (e.g., as done in Pal et al. (2024); Xu et al. (2024a); Pang et al. (2024); Liu al. (2024)).
samples with high CHES score prevents such unintentional unalignment, and
Overall, our results highlight the importance of curating data with sufficiently distinct preferences. We believe the CHES score introduced by our theory may prove valuable in achieving this goal.
Related work. We discuss related work throughout and defer concentrated account to Appendix A.
2

---

Published as a conference paper at ICLR 2025
2 PRELIMINARIES
Let V be a vocabulary of tokens. Modern language models consist of two parts: (i) neural network (e.g.. Transformer (Vaswani et al., 2017)) that intakes a sequence of tokens x E V and produces hidden embedding hx E R2; and (it) a token unembedding matrix W E R that converts the hidden embedding into logits. The logits are then passed through softmax to compute distribution over tokens that can follow x. For assigning probabilities to sequences y E V", a language model m operates autoregressively, i.e.:
To(y)x) II To (y II softmax (ahxxye
where 0 stands for the model's parameters (i.e. the parameters of the neural network and the unem- bedding matrix W), and yeek denotes the first tokens of y.
2.1 DIRECT PREFERENCE LEARNING
Preference data. We consider the widely adopted direct preference learning pipeline, which relies on pairwise comparisons (cf. Rafailov et al. (2023)). Specifically, we assume access to preference prompt, y* a preferred response x,
dataset 'D containing samples (x, y y-), where and y - is dispreferred response to x. The preferred and dispreferred responses can be obtained by generating two candidate responses from the model (i.e. on-policy), and labeling them via human or AI raters (cf. Ouyang et al. (2022); Bai et al. (2022b)). Alternatively, they can be taken from some static dataset (i.e. off-policy). Our analysis and experiments capture both cases.
Supervised finetuning (SFT). Preference learning typically includes an initial SFT phase, in which the model is finetuned via the standard cross-entropy loss. The sequences used for SFT can either be independent of the preference dataset 'D (Touvron et al., 2023) or consist of prompts and preferred responses from D, i.e. of y+) ): (x,y*, y E D) (Rafailov et al., 2023).
Preference learning loss. Aligning language models based on pairwise preferences is usually done by minimizing a loss of the following form:
(() := E(xxy )~D )-D[ (xx'' (In -In to (y (x))]
(2)
E DD Denote by Omit the parameters of the model prior minimizing the loss To guarantee that minimizing & entails increasing the difference between In no (x()) and In (y [x), as expected from reasonable preference learning loss, we make the mild assumption that & exytty- is monotonically decreasing in a neighborhood of In Te.... (y (x) me (y (x))...
where y- IR + Ryo is convex and differentiable, for every
The loss L generalizes many existing losses, including: DPO (Rafailov et al., 2023), IPO (Azar al., 2024), SLiC (Zhao al., 2023), REBEL (Gao et al., 2024), and GPO (Tang et al., 2024) see Appendix D for details on the choice of 'x. corresponding to each loss.? Notably, the common dependence on reference model is abstracted through fx.+ Other loss variants apply different weightings to the log probabilities of preferred and dispreferred responses or incorporate an additional SFT regularization term (e.g., DPOP (Pal et al., 2024), CPO (Xu al., 2024a), RPO (Liu et al., 2024). BoNBoN (Gui et al., 2024), and SimPO (Meng et al., 2024)). For conciseness, we defer extension of our analysis for these variants to Appendix G.
2.2 LIKELIHOOD DISPLACEMENT
We define likelihood displacement as the phenomenon where, although the preference learning loss is steadily minimized, the log probabilities preferred responses decrease.
Definition 1. Let Te.. and Tée.. denote a language model before and after training with a prefer- ence learning loss L over the dataset (Equation (2)), respectively, and suppose that the loss was
For SLiC and GPO, the corresponding & y is differentiable almost everywhere, as opposed differ- entiable. Our analysis applies such losses up to minor adaptations excluding non-differnntaabl points.
3

---

Published as a conference paper at ICLR 2025
Tokens Increasing Most in Probability
Model
y+
y
To [x) Decrease
Benign
Catastrophic
Yes
No
0.69 (0.96 + 0.27)
_Yss, -yes
OLMo-IB
No
Never
0.84 (0.85 - 0.01)
NNo
Yes, _Yes, -yes
Yes
No
0.22
(0.99 + 0.77)
Yes, yes
Gemma-28
No
Never
0.21 (0.65 + 0.44)
no, _No
yes, Yeah
Yes
No
0.96 (0.99 + 0.03)
yes, -ees, -Yes sure, _Sure
Llama-3-8B
Sure
Yes
0.59 (0.98 + 0.39)
Maybe, No, Never
Table I: Likelihood displacement can be catastrophic, even when training on a single prompt with single token responses. Each model was trained via DPO on randomly chosen prompt from the Persona dataset (Perez et al., 2022), using different pairs of preferred and dispreferred tokens (y yy"
y ) (as detailed in Sec- tion 3). We report the largest decrease in the preferred token probability (y x) during training for repre- sentative (y . pairs, averaged across ten runs differing in the chosen prompt. On the right are notable tokens whose probabilities increase the expense of yt categorized into benign catastrophic according whether they have a meaning similar to or distinct from y respectively (a preceding " stands for whites- pace; see Appendix J.1 for the full list and extents of increase). Remarkably, when y and y are similar in meaning, the tokens increasing most in probability are often opposite in meaning to y*
successfully reduced, i.e. L(0mm) < C(9iitt) We say that likelihood displacement occurred if:3
1 [D] Eieyty In Tea (y+1x) xx)
DD) E Eietss In Temi (x+1x);
1
JED
and that likelihood displacement occurred (x,yt,y y E Toan (y+ < In Teni (y+[x))
Likelihood displacement is not necessarily problematic. For (x.y+, y E D. we refer to it as benign if the responses increasing probability are as preferable as yt (e.g., they are similar in meaning to )). However, as Section demonstrates, the probability mass can go to responses that are substantially less preferable than y+ (e.g., they are opposite in meaning to y),, in which case we say it is catastrophic.
CATASTROPHIC LIKELIHOOD DISPLACEMENT IN SIMPLE SETTINGS
Despite the prevalence of likelihood displacement (Pal et al., 2024; Yuan et al., 2024; Pang et al., 2024; Rafailov et al., 2024a; Liu et al., 2024), there is limited understanding as to why it occurs and where the probability mass goes. Prior work attributed this phenomenon limitations pacity (Tajwar et al., 2024), the presence of multiple training samples or output tokens (Tajwar et al., 2024; Pal et al., 2024), and the initial SFT phase (Rafailov al., 2024b). In contrast, we demon- strate that likelihood displacement can occur and be catastrophic independently of these factors, even when training over just a single prompt whose responses contain single token each. The potential adverse effects of such displacement raise the need to formally characterize its underlying causes.
model ca-
Setting. The experiments are based the Persona dataset (Perez al., 2022), in which every prompt contains a statement, and the model needs to respond whether agrees with the statement using a single token. We assign to each prompt pair of preferred and dispreferred tokens (y y") from a predetermined set containing, e.g., Yes, Sure, No, and Never. Then, for the OLMo- 1B, Gemma-2B. and Llama-3-8B models, perform one epoch of SFT using the preferred tokens as labels, in line with common practices, and train each model via DPO on a single randomly selected prompt. See Appendix K.1 for additional details.
Likelihood displacement pervasive and can be catastrophic. Table 1 reports the decrease in preferred token probability, and notable tokens whose probabilities increase at the expense of y+. The probability of y dropped by at least 0.21 and up to 0.96 absolute value in all runs. Remarkably, when y+ and y are similar in meaning, the probability mass often shifts to tokens with meanings opposite to y Appendix J.1 reports similar findings for experiments using: (i) base models that did not undergo an initial SFT phase (Table 2); or (ii) IPO instead of DPO (Table 3).
*Note that In To (y x) can decrease even as the loss L is minimized, since minimizing C only requires increasing the gap between In To (x) and In To (x).
4

---

Published as a conference paper at ICLR 2025
4 THEORETICAL ANALYSIS OF LIKELIHOOD DISPLACEMENT
To uncover what causes likelihood displacement when minimizing preference learning loss, we characterize how the log probabilities of responses evolve during gradient-basee training. For a preference sample (x.y"..
E D, we identify the factors pushing In no ro x) downwards and those determining which responses increase most in log probability instead. Section 4.1 lays out the technical approach, after which Section 4.2 gives an overview of the main results. The full analysis deferred to Appendix F. For the convenience of the reader, we provide the main takeaways below.
Takeaway 1: Role of the Token Unembedding Geometry (Section 4.2.1)
Even when training over just single prompt whose responses y* and y contain a sin- gle token, likelihood displacement can occur due to the token unembedding geometry. The underlying causes are: (i) an alignment between the preferred and dispreferred token un- embeddings, measured (Wy Ww y ); and (ii) tokens whose unembeddings align with Wy W y which increase in log probability at the expense of yt. Since W,, Wy can have large component orthogonal to Wy (introduced by W, ). and unembeddings often linearly encode semantics, this provides an explanation for why probability mass can go to tokens unrelated or opposite in meaning to y+ (as observed empirically in Section 3).
Takeaway 2: Role the Hidden Embedding Geometry (Section 4.2.2)
Besides the impact of the token unemhedding geometry (Takeaway 1), likelihood displace- ment occurs when the preferred and dispreferred responses similar according to the fol- lowing measure, which is based on their hidden embeddings.
Definition 2. For a preference sample (x,y+, y+")) E DD we define the centered hidden embedding similarity (CHES) score of y and y with respect to model To by:
CHESx(y" := = E ( Ex bxy
Σ
y hidden embeddings
y hidden embeddings
where hx,a.z denotes the hidden embedding that the model produces given x and the first k - tokens of z E V A higher CHES score stands for more similar preferences. We omit the dependence of CHES on Te in our notation it will be clear from context.
Losses with SFT regularization. Appendix G extends our analysis to losses incorporating an SFT regularization term. In particular, it formalizes how this modification helps mitigate likelihood dis- placement, as proposed in Pal et al. (2024); Liu et al. (2024); Pang et al. (2024); Gui et al. (2024). We note, however, that our experiments in Section 6 reveal limitation of this approach for mitigating the adverse effects of likelihood displacement, compared to improving the data curation pipeline.
4.1 TECHNICAL APPROACH
Given a prompt x, the probability that the model my assigns to a response z is determined by the
hidden embeddings hx, hx,zeay--.,, hx,z2|= and the token unembeddings W (Equation (1)). Our analysis relies tracking their evolution when minimizing the loss L (Equation (2)). To do we adopt the unconstrained features model (Mixon et al., 2022), which amounts to treating hidden embeddings directly trainable parameters. Formally, the trainable parameters are taken to be (h2 z E V" (W). This simplification has proven useful for studying various deep learning phenomena, including neural collapse (e.g., Zhu et al. (2021); Ji et al. (2022): Tirer et al. (2023)) and the benefits of language model pretraining for downstream tasks (Saunshi et al., 2021). As verified in Sections 5 and 6, it also allows extracting salient sources of likelihood displacement in practice.*
"In contrast prior theoretical analyses of likelihood displacement, which consider simpler settings. such linear models and cases where the preferred and dispreferree responses differ only by single token (Pal al., 2024; Fisch et al., 2024; Song et al., 2024h; Ren and Sutherland, 2024).
5

---

Published as a conference paper at ICLR 2025
Language model finetuning is typically done with small learning rates. Accordingly, analyze the training dynamics of (stochastic) gradient descent at the small learning rate limit, i.e. gradient flow:
d dt (t) = - v((((t) .2
where ((1) denotes the parameters at time 2 0 of training. Note that under gradient flow the loss is monotonically decreasing. Thus, any reduction in the log probabilities of preferred responses constitutes likelihood displacement (ef. Definition 1).
4.2
OVERVIEW OF THE MAIN RESULTS
4.2.1 SINGLE TRAINING SAMPLE AND OUTPUT TOKEN
It instructive to first consider the case of training on single sample (x, yx, ,yy+,"",, whose responses yt E V and y E V contain a single token. Theorem characterizes how the token unembedding geometry determines when dt In (T()) (y (x) negative, i.e. when likelihood displacement occurs. Theorem (Informal version of Theorem 4). Suppose that the dataset DD contains a single sample (x, . ), with y+ E V and y- V each being single token. At any time t 2 0 of training, d In (rro)(y+(xxx is more negative the larger the following term is: (1))
d
Exerrotor> TB(1)(2/x).
(W:(), Wy y (1) Wy (1)
preferences unembreddnng aligunmes
alignment of other token with W y+ (t) W (1)
where (t) denotes the token unembedding of 2 E V at time t.
Two terms govern the extent of likelihood displacement in the case of single token responses. First, (W,y y+ (O) Wy- y- (4)) formalizes the intuition that likelihood displacement occurs when the preferred and dispreferred responses are similar. A higher inner product in unembedding space translates to a more substantial (instantaneous) decrease in In (T( (y+ (x). Second, is term which measures the alignment of other token unembeddings with Wy (t)- Wy (t), where tokens deemed more likely by the model have a larger weight. The alignment of token unembeddings with Wy (t) Wy (t)
also determines which tokens increase most in log probability.
Theorem 2 (Informal version of Theorem 5). Under the setting of Theorem 1, for any time 1 > of training and token E Py{y+,y y it holds that z8 In (2(x) x (Wa(t) W yy+ (t) Wy- (t)), up to an additive term independent of..
The direction Wy+ (t) - Wy- (t) can be decomposed into its projection onto W y (t) and a com- ponent orthogonal to w (t), introduced by Wy, (t). Thus, tokens increasing in log probability can have unembeddings that mostly align with directions orthogonal to Wy+ especially when the component orthogonal to Wy-(t) of W y (t) W y--C0 (t) is relatively large (which we often find to be the case empirically; see Table 13 in Appendix J.1). Given that token unembeddings are known to linearly encode semantics (Mikolov et al., 2013; Arora et al., 2016; Park et al., 2024), this provides an explanation for why the probability mass can shift to tokens that are unrelated or oppo- site in meaning to the preferred token, i.c. why likelihood displacement can be catastrophic even in simple settings (as observed in Section 3).
4.2.2 RESPONSES WITH MULTIPLE TOKENS
We now extend our analysis to the typical case where responses are sequences of tokens. As shown below, the existence of multiple tokens in each response introduces a dependence on their (contex- tual) hidden embeddings.
Theorem 3 (Informal version of Theorem 6). Suppose that the dataset D contains a single sample (x,y*, y y"), with y+ E V and y E V². At any time Z 0 of training, in addition to the dependence on token unembeddingg identified in Theorem 1, dt In TocC) (x) is more negative the larger the following term is:
ly
lyt
( (1)
(Mxxst (1),h (1)
(1)
(M (4), hy ese (1).
preferred prejerreddiipreferedd dispreferred alignment
preferred preferved-preeeeree alignment
'Except for the trivial case where (0) is a critical point of C, in which L(((t)) L(((0)) for all t 0.
6

---

Published as a conference paper at ICLR 2025
where hy) denotes the hidden embedding of z E V time t, and ag (t), E [- are coefficients determined by the model's next-token distributions for prefixes of y and y (see Theorem 6 in Appendix for their definition).
Theorem 3 establishes that the inner products between hidden embeddings, of both the "preferred- dispreferred" and snfferredd-reffrree types, affect likelihood displacement. A larger inner product leads to an upwards or downwards push on In Te(e) (y sponding (() or (f.,x (l) coefficient. Empirically, we find that these coefficients positive across models and datasets; e.g., the OLMo-1B, Gemma-2B, and Llama-3-8B models and the UtraaFedbbckk and AlpacaFarm datasets (see Appendix J.2 for details). By accordingly setting all coefficients in Theorem to one, we derive the centered hidden embedding similarity (CHES) score between preferred and dispreferred responses (Definition 2). Our analysis indicates that higher CHES score implies more severe likelihood displacement. Section 5 empirically verifies this relation, and demonstrates that the CHES score is significantly more predictive of likelihood displacement than other plausible similarity measures.
depending on the sign of the corre-
mostly
Our analysis also provides insight into which responses increase most in probability at the expense of y Theorem in Appendix F.2 derives the dependence of d (z(x), for any response dt E y*, on the alignment of its hidden embeddings with those of yt and y However, in general settings, it is difficult to qualitatively describe the types of responses increasing in probability, and whether they constitute benign or catastrophic likelihood displacement. Section 6 thus demonstrates the (harmful) implications of likelihood displacement in settings where responses can be easily categorized into benign or catastrophic. We regard studying the question of where the probability mass goes additional settings as a promising direction for future work.
4.2.3 MULTIPLE TRAINING SAMPLES
Sections 4.2.1 and 4.2.2 showed that likelihood displacement may occur regardless of the dataset size. Nonetheless, increasing the number of training samples was empirically observed to exacer- bate it (Tajwar et al., 2024). Appendix F.3 sheds light on this observation by characterizing, for any (x, yt E DD when additional training samples lead to a larger decrease in In TT(C)) (y [x). This unsurprisingly occurs when yt appears as the dispreferred response of other prompts, i.e. there are contradicting samples. We further establish that additional training samples can contribute nega- tively to d In Tote) (x+()) even when their preferences are distinct from those of x. de
IDENTIFYING SOURCES OF LIKELIHOOD DISPLACEMENT
In Section 4 we derived the CHES score (Definition 2), which for a given model and preference sample yx, y)), measures the similarity of y+ and y based on their hidden embeddings. Our theory indicates that samples with a higher CHES score lead to more likelihood displacement, Be- low, we affirm this prediction and show that the CHES score enables identifying which training samples in dataset contribute most to likelihood displacement, whereas alternative similarity mea- sures fail to do so. The following Section 6 then demonstrates that filtering out samples with a high CHES score can mitigate undesirable implications of likelihood displacement.
Setting. We use the UltraFeedback and AlpacaFarm datasets and the OLMo- IB, Gemma-2B, and Llama-3-8B models. For every preference dataset and model, we compute the CHES scores of all samples. This requires performing single forward pass over the dataset. Then, for each of the Oth, 25th, 50th, 75th, and 100th score percentiles, we take subset of 512 samples centered around it. Lastly, we train the model via DPO on each subset separately, and track the change in log probability for preferred responses in the subset the more the log probabilities decrease, the more severe the likelihood displacement is. See Appendix K.2 for further details.
Baselines. Preferences with low (normalized) edit distance were suggested in Pal et al. (2024) as a cause for likelihood displacement. Thus, we repeat the process outlined above while ranking the similarity of preferences using the (normalized) edit distance, where lower edit distance between y and y corresponds to a higher similarity. To the best of our knowledge, no other property of preference sample was linked with likelihood displacement in the literature. So we additionally
"The Oth and 100th percentile subsets include the 512 samples with lowest and highest scores, respectively.

---

Published as a conference paper at ICLR 2025
Gemma-28
Lama-3-88
30-
Response Log Probability
800
Change in Preferred
20-
600
Score
400
Similartty
200
Hidden Embedddingg Similrity
0-
-10
-200-
Like Ihoad acement
Like ihood Displacement
50
75
100
0
25
50
100
Preference Similarity (Percentile)
Preference Similarity (Percentile)
Figure 2: CHES score (Definition 2) identifies which training samples contribute to likelihood displace- ment, whereas alternative similarity measures do not. Each model was trained via DPO on subsets of 512 samples from the UltraFeedback dataset. The subsets are centered around different preference similarity per- centiles, according to the following measures: (i) the CHES score; (ii) (normalized) edit distance, which was suggested in Pal et al. (2024) indicative of likelihood displacement: and (iii) the inner product between the last hidden embeddings the preferred and dispreferred responses (see Section 5 for further details). We report for each subset the change mean preferred response log probability, averaged across three runs (error bars denote minimal and maximal values). The CHES score ranking perfectly matches with the degree of likelihood displacement - subsets with a higher score percentile induce larger log probability decrease. On the other hand, the alternative measures are not indicative of likelihood displacement.
compare natural candidate: using the inner product between the last hidden embeddings of y and y iie. (hx,++, hx,y-), as the similarity score.
CHES score effectively identifies samples leading to likelihood displacement. For the Ultra- Feedback dataset, Figure 2 shows the change in mean preferred response log probability against the similarity percentile of samples. Across all models, the CHES score ranking matches perfectly the degree of likelihood displacement: the higher the CHES score percentile, the more preferred responses decrease in log probability. Moreover, training on samples with high CHES scores leads to severe likelihood displacement, whereas training on samples with low CHES scores leads the preferred responses increase in log probability.
CHES score is more indicative of likelihood displacement than alternative measures. con- trast to the CHES score, the edit distance of preferences and the inner product between their last hidden embeddings are not indicative of likelihood displacement. Moreover, these measures failed to identify samples leading to likelihood displacement: for almost all similarity percentiles, the mean preferred response log probability increased, with the few exceptional decreases being minor.
Additional experiments. Appendix J.3 reports similar findings for experiments using: (i) the Al- pacaFarm dataset instead UltraFeedback (Figure 5); (ii) IPO instead of DPO (Figure 6); or (iii) the OLMo-1P IB model (Figure 7).
Qualitative analysis. Appendix J.3 further includes representative samples with high and low CHES scores (Tables 14 and 15, respectively). .. A noticeable trait is that, in samples with a high CHES score, the dispreferred response is often longer than the preferred response, whereas for samples with low CHES score the trend reversed (ie. preferred responses are longer). We find that this stems from tendency of current models produce, for different responses, hidden embeddings with a positive inner product (e.g., over 99% of such inner products are positive for the Llama-3-8B model and UltraFeedback dataset). As a result, for samples with longer dispreferred responses the CHES score comprises more positive terms than negative terms.
UNINTENTIONAL UNALIGNMENT IN DIRECT PREFERENCE LEARNING
Direct preference learning has been successfully applied for improving general instruction following and performance on downstream benchmarks (e.g., Tunstall al. (2023); Ivison al. (2023); Jiang et al. (2024); Dubey et al. (2024)). This suggests that likelihood displacement may often be benign in such settings, and so does not require mitigation. However, in this section, we reveal that can un- dermine the efficacy of safety alignment. When training a language model to refuse unsafe prompts, find that likelihood displacement can unintentionally unalign the model, by causing probability mass to shift from preferred refusal responses to harmful responses. We then demonstrate that this undesirable outcome can be prevented by discarding samples with a high (length- normalie)) CHES
8

---

Published as a conference paper at ICLR 2025
score (Definition 2), showcasing the potential of the CHES score for mitigating adverse effects of likelihood displacement.
6.1 SETTING
We train a language model to refuse unsafe prompts via the (on-policy) direct preference learning pipeline outlined in Rafailov et al. (2023), as specified below. To account for the common scenario whereby one wishes to further align existing (moderately) aligned model, we use the Gemma- 2B-IT and Llamma3-8B-Instruet models." 7 Then, for each model separately, we create a preference dataset based on unsafe prompts from SORRY-Bench (Xie et al., 2024b). Specifically, for every prompt, we generate two candidate responses from the model and label them as refusals or non- refusals using the judge model from Xie et al. (2024b). Refusals are deemed more preferable com- pared to non-refusals, and ties are broken by the PairRM reward model (Jiang et al., 2023). Lastly, we partition the datasets into training and test sets according to 85% split, and train the lan- guage models via DPO over their respective training sets, For brevity, we defer to Appendices and K some experiments using IPO and implementation details, respectively.
6.2 CATASTROPHIC LIKELIHOOD DISPEACEMENT CAUSES UNINTENTIONAL UNALIGNMENT
Since the initial models are moderately aligned, we find that they often generate two refusal re- sponses for given prompt. Specifically, for over 70% of the prompts in the generated datasets, both the preferred and dispreferred responses are refusals. This situation resembles the experiments of Section 3, where training on similar preferences led to catastrophic likelihood displacement (e.g., when y was No and y was Never, the probability of Yes sharply increased).
Analogously, we observe that as the DPO loss is minimized, likelihood displacement causes proba- bility mass shift away from preferred refusal responses (Table 16 in Appendix J.4 reports the log probability decrease of preferred responses). This leads to a significant drop in refusal rates. Specifi- cally, over the training sets, DPO makes the refusal rates of Gemma-2B-IT and Llama-3-8B 3-8B-tnstt drop from 80.5% to 54.8% and 74.4% to 33.4%, respectively (similar drops occur over the test sets). In other words, instead of further aligning the model, preference learning unintentionally unaligns it. See Appendix J.4 for examples of unsafe prompts from the training sets, for which initially the models generated two refusals, yet after DPO they comply with the prompts (Table 18).
We note that alignment usually involves a tradcoff between safety and helpfulness. The drop in refusal rates is particularly striking since the models are trained with the sole purpose of refusing prompts, without any attempt to maintain their helpfulness.
6.3 FILTERING DATA VIA CHES SCORE MITIGATES UNINTENTIONAL UNALIGNMENT
Section 5 showed that samples with a high CHES score (Definition 2) contribute most to likelihood displacement. Motivated by this, we explore whether filtering data via the CHES score can mitigate unintentional unalignment, and which types of samples it marks as problematic.
As discussed in Section 5, due to the embedding geometry of current models, CHES scores can correlate with the lengths of responses. To avoid introducing a length bias when filtering data, we apply length-normalized variant of CHES (see Definition 3 in Appendix C). For comparison, we also consider adding an SFT term to the DPO loss, as suggested in Pal et al. (2024); Xu et al. (2024a); Pang et al. (2024); Liu al. (2024), and training over "gold" responses from SORRY-Bench, which were generated from a diverse set of base and safety aligned models and labeled by human raters.
Filtering data via CHES score mitigates unintentional unalignment. Figure 3 reports the refusal rates before and after training via DPO: (i) on the original dataset, which as stated in Section 6.2 leads to substantial drop in refusal rates; (ii) with additional SFT term on the original dataset; (iii) on the gold dataset; and (iv) on a filtered version of the original dataset that contains the 5% samples with lowest length- normalized CHES scores. Filtering data via the CHES score success-
"The scenario of further aligning an existing moderately aligned model also arises in iterative direct prefer- ence learning pipelines (Yuan et al., 2024; Xiong et al., 2024; Pang et al., 2024).
BBreaking ties randomly between responses of the same type led to similar results.
"Keeping up 15% of the original samples led to analogous results. Beyond that, when training the full dataset, likelihood displacement caused refusal rates drop.

---

Published as a conference paper at ICLR 2025
Initial
DPO SFT DPO (gold)
DPO (filtered))
DPO
Length-Norr Sangth--ormllzeed CHES Ranking
100
100-
Refusal Rate (96
*
35/370 (9%)
73:77 (20%)
7622370 (7190)
80
60
200
250
40
200
20
350
One Each
Two Two Non-Refurals
Gemma- 2B-IT Liama- Bamaa-3-8-IInsst 3-8B-Instruct
Refusals
Figure 3: Likelihood displacement
uninten- tional unalignment, which mitigated by data filtering. Training model to refuse unsafe prompts from SORRY- Bench via DPO unintentionally leads to a substantial de- crease in refusal rates due to likelihood displacement. Fil- CHES
Figure 4: Length-normalized CHES identifies samples with responses of the same type as responsible for likelihood dis- placement. For Llama-3-88 Maama-3-8B-nnstmmm 88-Instruct, we take the corresponding SORRY- CORRY-Bench training pref- erence dataset (see Section 6.1 for details on the dataset creation process), and plot the ranking of samples according to their nngth--ormmlized CHES scores. Gray line marks the 5% sam- ples included in the filtered dataset of Figure 3. Agrecing with intuition, samples with two re- fusal two non-refusal responses tend to have higher score than samples with of each.
tering out samples with a high length-normalizg score or using "gold" preference data, generated from diverse
models, successfully mitigates the problem, and goes beyond the improvement achieved when adding an SFT term to the DPO loss. Reported are the refusal rates over the training sets, averaged across three runs (error bars denote minimal and maximal values). Results over the test sets were similar. See Section 6 for further details.
fully mitigates unintentional unalignment. Morcover, while adding an SFT term the loss also prevents the drop in refusal rates, data filtering boosts the refusal rates more substantially. We fur- ther find that DPO on gold preferences does not suffer from likelihood displacement unintentional unalignment (i.e. the preferred responses increase in log probability; see Table 16). Overall, these results highlight the importance of curating data with sufficiently distinct preferences for effective preference learning.
Which samples have high CHES score? Figure 4 reveals that the enngt-nnormainn CHES score ranking falls in line with intuition - samples with two refusal or two non-refusal responses tend to have a higher score than samples with one of each, and so are more likely to cause likelihood displacement. To confirm that both samples with two refusal responses and samples with two non- refusals are responsible for the drop in refusal rates (shown in Figure 3), we trained the Gemma- 2B-IT and Llama-3-8P I-St-rnst models via DPO on each type of samples separately. In both cases, likelihood displacement occurred and the refusal rates dropped as when training on the full dataset.
CONCLUSION
While direct preference learning has been widely adopted, there is considerable uncertainty around how affects the model (cf. Xu al. (2024b); Chen et al. (2024)). Our theory and experiments shed light on the causes and implications of one counterintuitive phenomenon - likelihood displacement. We demonstrated that likelihood displacement can be catastrophic, shifting probability mass from preferred responses to responses with an opposite meaning, which can result in unintentional un- alignment when training language model refuse unsafe prompts. Intuitively, these failures arise when the preferred and dispreferred responses are similar. We formalized this intuition and derived the centered hidden embedding similarity (CHES) score (Definition 2), which effectively identifies samples contributing to likelihood displacement in a given dataset. As an example of its potential uses, showed that filtering samples with high (engthh-normalizedd) CHES score can prevent unintentional unalignment. More broadly, our work highlights the importance of curating data with sufficiently distinct preferences. We believe the CHES score introduced by our theory may prove valuable in achieving this goal.
Due to lack of space, defer Appendix B discussion of limitations and future work.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
We thank Eshbal Hezroni for aid in preparing illustrative figures, Angelica Chen, Tianyu Gao, and Mengzhou Xia for providing feedback on this paper, and Yi Ren for insightful discussions. NR is supported in part by the Zuckerman STEM Leadership Program. SM and SA acknowledge funding from NSF, ONR. Simons Foundation, and DARPA. AB gratefully acknowledges the support of Hisashi and Masae Kobayashi*67 Fellowship. DC is supported by the National Science Foundation (IIS-221 1779) and Sloan Research Fellowship. BH is supported by a 2024 Sloan Fellowship in Mathematics, NSF CAREER grant DMS-2143754, and NSF grants DMS-1855684 DMS-2133806.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Aluman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08771 087774, 2023.
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. A latent variable model approach to pmi-based word embeddings. Transactions of the Association for Computational Linguistics, 4:385-399, 2016.
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pages 4447-4455 PMLR, 2024.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.0586. 2022a.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kerionn, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutinall ai: Harmlessness from ai feed- back. arXiv preprint ar 2711v:2212.08073, 2022b.
Shicong Cen, Jineheng Mei, Katayoon Goshvadi, Hanjun Dai, Tong Yang, Sherry Yang, Dale Schuurmans, Yuejie Chi, and Bo Dai. Value-incentivized preference optimization: unified approach online and offline rlhf. arXiv preprint arXiv:240S.19322 2024.
Angelica Chen, Sadhika Malladi, Lily H Zhang, Xinyi Chen, Qiuyi Zhang, Rajesh Ranganath, and Kyunghyun Cho. Preference learning algorithms do not learn preference rankings. arXiv preprint arXiv:2405. /9534, 2024.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafedbbcck: Boosting language models with high-quality feedback. In International Conference on Machine Learning, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.22 21783, 2024.
Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. Alpacafarm: simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36, 2024.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. In International Conference on Machine Learning, 2024.
Duanyu Feng, Bowen Qin, Chen Huang. Zheng Zhang, and Wenqiang Lei. Towards analyzing and understand- ing the limitations of dpo: A theoretical perspective. arXiv preprint arXiv:240..04626, 2024.
Adam Fisch. Jacob Eisenstein, Vicky Zayats, Alekh Agarwal, Ahmad Beirami, Chirag Nagpal. Pete Shaw, and Jonathan Berant, Robust preference optimization through reward model distillation. arXiv preprint htl.:2405.19366 2024.
Zhaolin Gao, Jonathan D Chang, Wenhao Zhan, Owen Oertell, Gokul Swamy, Kianté Brantley, Thorsten Joachims, J Andrew Bagnell, Jason D Lee, and Wen Sun. Rebel: Reinforcement learning via regressing relative rewards. arXiv preprint arXiv:2404. 16767, 2024.

---

Published as a conference paper at ICLR 2025
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Olmo: Accelcrating the science of language models. arXiv preprint arXiv:2402. 00838, 2024.
Lin Gui, Cristina Gårbacea, and Victor Veitch. Bonbon alignment for large language models and the sweetness of best-of-n sampling. arXiv preprint arXiv:2406.0000 00832, 2024.
Luxi He, Mengzhou Xia, and Peter Henderson. What's in your" safe" data?: Identifying benign data that breaks safety. arXiv preprint arXiv:2404.01099, 2024.
Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. Neural networks for machine learning lecture 6a overview of mini-batch gradient descent. Cited on, 14(8):2, 2012.
Jiwoo Hong, Noah Lee, and James Thoree. Reference-free monolithic preference optimization with odds ratio. arXiv preprint arXiv:2403. 400.07691,,.. 2024.
Audrey Huang, Wenhao Zhan, Tongyang Xie, Jason D Lec, Wen Sun, Akshay Krishnamurthy, and Dylan J Foster. Correcting the mythos of kl-regularization: Direct alignment without overparameterization via chi- squared preference optimization. arXiv preprint arXiv:2407.13399 2024.
Shawn Im and Yixuan Li. On the generalization of preference learning with dpo. arXiv preprint arXiv:2408.0344 033449 2024a.
Shawn Im and Yixuan Li. Understanding the learning dynamics of alignment with human feedback. In Inter- national Conference Machine Learning, 2024b.
Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang. David Wadden, Noah A Smith, Iz Beltagy, et al. Camels in a changing climate: Enhancing Im adaptation with tulu 2. arXiv preprint 2rXiv:2311.10702 2023.
Wenlong Ji, Yiping Lu, Yiliang Zhang. Zhun Deng, and Weijie Su. An unconstrained layer-peeled perspective on neural collapse. In International Conference on Learning Representations, 2022.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, De- vendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, al. Mixtral of experts. arXiv preprint arXiv:2401.0400 2024.
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. LIm-blender: Ensembling large language models with pairwise ranking and generative fusion. arXiv preprint drXiv:2306.02661, 2023.
Zhihan Liu, Miao Lu, Shenao Zhang, Boyi Liu, Hongyi Guo, Yingxiang Yang. Jose Blanchet, and Zhaoran Wang. Provably mitigating overoptimization in rihf: Your sft loss is implicitly adversarial regularizer. arXiv preprint arXiv:2405. 16436, 2024.
Kaifeng Lyu, Haoyu Zhao, Xinran Gu, Dingli Yu, Anirudh Goyal, and Sanjeev Arora. Keeping Iiss aligned after fine-tuning: The crucial role of prompt templates. arXiv preprint arXiv:2402. 185440000,8 2024.
Chao Ma, Lei Wu, et al. A qualitative study of the dynamic behavior for adaptive gradient algorithms. arXiv preprint 2oXx::2009.06222 2020.
Sadhika Malladi, Alexander Wettig. Dingli Yu, Danqi Chen, and Sanjeev Arora. A kernel-based view of language model fine-tuning. In International Conference on Machine Learning, pages 23610-23641 2023.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-free reward. arXiv preprint Xiv:v2405.. 144.4 2024.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 26, 2013.
Dustin G Mixon, Hans Parshall, and Jianzong Pi. Neural collapse with unconstrained features. Sampling Theory, Signal Processing, and Data Analysis, 20(2):11, 2022.
Long Ouyang, Jeffrey Wu. Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744 2022.
Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with dpo-positive, arXiv preprint arXiv:2402. 132.22 2024.
Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He. Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. arXiv preprint arXiv:2404.1977 2024.
12

---

Published as a conference paper at ICLR 2025
Kiho Park. Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. In International Conference on Machine Learning, 2024.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation pytorch. In NIPS W, 2017.
Kellin Pelrine, Mohammad Taufeeque, Michat Zajac, Euan McLean, and Adam Gleave. Exploiting novel gpt-4 apis. arXiv preprint arXiv:2312.14535 2023.
Ethan Perez, Sam Ringer, Kamilé Lukošiūté, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, al. Discovering language model behaviors with model-written cvaluations, arXiv preprint arXiv:22 09251, 2022.
Xiangyu Qi. Yi Zeng. Tinghao Xie, Pin-Ye Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine- teeittnnngg aligned language models compromises safety, even when users do intend to! International Conference on Learning Representations, 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Infor- mation Processing Syxtems, 36, 2023.
Rafael Rafailov, Yaswanth Chittepu. Ryan Park, Harshit Sikchi, Joey Hejna, Bradley Knox, Chelsea Finn, and Scott Niekum. Scaling laws for reward model overoptimization direct alignment algorithms. arXiv preprint arXiv:2406.02900 2024a.
Rafacl Rafailov, Jocy Hejna, Ryan Park, and Chelsea Finn. From to Q": Your language model secretly a Q-function. arXiv preprint orXiv:2404.22358 2024b.
Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauck- hage, Hannanch Hajishirzi, and Yejin Choi. Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization. In International Con- ference on Learning Representations, 2023.
Noam Razin, Hattie Zhou, Omid Saremi, Vimal Thilak, Arwen Bradley, Preetum Nakkiran, Joshua M. Susskind, and Etai Littwin. Vanishing gradients in reinforcement finetuning of language models. In In- ternational Conference Learning Representations, 2024.
Yi Ren and Danica J Sutherland. Learning dynamics of Ilm finetuning, arXiv preprint arXiv:2407.10490 2024.
Nikunj Saunshi, Sadhika Malladi, and Sanjecv Arora. A mathematical exploration of why language models help solve downstream tasks. In International Conference on Learning Representations, 2021. URL https:
//openreview. nnt. forum? d-vv www.eiiwwwws..... 338ccls.
Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. Preference ranking optimization for human alignment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18990-18998, 2024a.
Yuda Song. Gokul Swamy, Aarti Singh, J Andrew Bagnell, and Wen Sun. The importance of online data: Understanding preference fine-tuning via coverage. arXiv preprint 2xxx:24466001663 2024b.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning summarize with human feedback. In Advances in Neural Information Processing Systems, volume 33, pages 3008-3021, 2020.
Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie. Stefano Ermon, Chelsea Finn, and Aviral Kumar. Preference fine-tuning of llms should leverage suboptimal, on-policy data. arXiv preprint a Xiv::22004 14367, 2024.
Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Rémi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Ávila Pires, and Bilal Piot. Generalized preference optimization: A unified approach to offline alignment. In International Conference on Machine Learning, 2024.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi. Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.088 08295, 2024.
Tom Tirer, Haoxiang Huang, and Jonathan Neles-Weed. Perturbation analysis of neural collapse. In Interna- tional Conference Machine Learning, pages 34301-34329, PMLR, MLR, 2023.
13

---

Published as a conference paper at ICLR 2025
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cyn- thia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee. Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Mar- tinet. Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Erie Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnie, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of Im alignment, arXiv preprint arXiv:2310.1622 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar. Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cis- tac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771, 2019.
Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment. arXiv preprint arXiv:2405.00675 2024.
Tengyang Xie, Dylan J Foster, Akshay Krishnamurthy, Corby Rosset, Ahmed Awadallah, and Alexander
Rakhlin. Exploratory preference optimization: Harnessing implicit aapapproxnnttion for sample-efficient rihf. arXiv preprint 2xXi:2245.221046 2024a.
Tinghao Xie. Xiangyu Qi. Zeng, Yangsibo Huang, Udari Madhushani Sehwag. Kaixuan Huang, Luxi He, Boyi Wei, Dacheng Li, Ying Sheng, et al. Sorry- Systematically evaluating large language model safety refusal behaviors. arXiv preprint arXiv:2406. 14598, 2024b.
Wei Xiong, Hanze Dong. Chenlu Ye. Ziqi Wang. Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang, Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. In International Conference on Machine Learning, 2024.
Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of Ilm performance in machine translation. arXiv preprint 2xXi:2440.004111 2024a.
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang. Chao Yu, and Wu. Is dpo superior to ppo for Ilm alignment? a comprehensive study. arXiv preprint arXiv:2404.10719 2024b.
Zihao Xu, Yi Liu, Gelei Deng, Yuekang Li, and Stjepan Picek. A comprehensive study of jailbreak attack versus defense for large language models. In Lun-Wei Ku. Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguisties ACL 2024, pages 7432-7449, Bangkok, Thailand and virtual meeting, August 2024c. Association for Computational Linguistics. doi: 1/.18443/v//2224 findings-ade 443. URL https::pp... http:///aaaaatooo....o// hwwphhowwooooooooooooooooo 2024. findings-acll
Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruob- ing Xie, Yankai Lin, et al. Advancing Ilm reasoning generalists with preference trees. arXiv preprint arXiv:2404. 02078, 2024.
Qiusi Zhan, Richard Fang, Rohan Bindu, Akul Gupta, Tatsunori Hashimoto, and Daniel Kang. Removing RLHF protections in GPT-4 via fine-tuning. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pages 681-687, Mexico City, Mex- ico, June 2024. Association for Computational Linguistics. doi: 10.18653/w1/024, maw.e44s.aaa--sht.55555 URL https: / acl.atonooooooyyy..ooooo org/ 2024. naacc-aaaaeecocc saoclssshot.5..
Shenao Zhang. Donghan Yu, Hiteshi Sharma, Ziyi Yang, Shuohang Wang. Hany Hassan, and Zhaoran Wang. Self-exploring language models: Active preference elicitation for online alignment. arXiv preprint 0rXiv:2405.19.33 2024.
14

---

Published as a conference paper at ICLR 2025
Yao Zhao, Mikhail Khalman, Rishabh Joshi, Shashi Narayan, Mohammad Salch, and Peter J Liu. Calibrating sequence likelihood improves conditional language generation. In International Conference on Learning Representations, 2023. URL https: /ttpa:://pppeeeee /ttpp//ewpe/ee.e////////// htt.:///oooooo..eoooooooooooaaaa
Rui Zheng, Shihan Dou. Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang. Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, al. Secrets of rihf large language models part Ppo. arXiv preprint arXiv:2307 044944444444 2023.
Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu. A geometric analysis of neural collapse with unconstrained features. Advances Neural Information Processing Systems, 34:29820-2999 298344, 2021.
Daniel Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alee Radford, Dario Amodei, Paul Chris- tiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:/909008590 2019.
15

---

Published as a conference paper at ICLR 2025
A RELATED WORK
Preference learning for language model alignment. There are two main approaches for aligning language models based on preference data. First, RLHF (or RLAIF) (Ziegler et al., 2019; Stiennon et al., 2020; Ouyang et al., 2022: Bai et al., 2022b), which requires fitting a reward model to a dataset of human (or AD) preferences, and then training the language model to maximize the reward. While often effective for improving the quality of generated responses, RLHF is computationally expensive and can suffer from instabilities (Zheng al., 2023; Ramamurthy et al., 2023; Razin et al., 2024). This has led to the rise of direct preference learning, as popularized by DPO (Rafailov al., 2023). Our analysis supports methods that maximize the log probability ratio of preferred and dispreferred responses (cf. Section 2. 1), including DPO and many of its variants (e.g., Zhao et al. (2023); Azar et al. (2024); Gao al. (2024); Tang et al. (2024); Pal al. (2024); Xu et al. (2024a); Liu et al. (2024); Gui et al. (2024); Meng et al. (2024)). Investigating whether other variants, e.g., those proposed in Ethayarajh al. (2024); Hong et al. (2024); Song al. (2024a); Wu et al. (2024), suffer from likelihood displacement is a potential avenue for future work.
Analyses of direct preference learning. Prior work mostly established sample complexity guaran- tees for DPO (or a variant of it) when the training data either obeys a certain stringent structure (Im and Li, 2024a), provides sufficient coverage (Liu et al., 2024; Song et al., 2024b; Cen et al., 2024; Huang et al., 2024), or is obtained in an online manner (Cen et al., 2024; Zhang et al., 2024; Xie et al., 2024a). Additionally, Im and Li (2024b); Feng et al. (2024) studied the optimization rate of DPO. More relevant to our work is Chen et al. (2024), which demonstrated that DPO can fail to correct how a model ranks preferred and dispreferred responses. Although related, this phenomenon distinct from likelihood displacement.
particular, when likelihood displacement occurs the probability of preferred responses is often higher than the probability of dispreferred responses (as illustrated in Figure and was the case the experiments of Sections 3, 5, and 6).
Likelihood displacement. The relation of our results to existing claims regarding likelihood dis- placement was discussed throughout the paper. We provide in Appendix E an extended account.
Jailbreaking and unalignment. Aligned language models are vulnerable to jailbreaking through carefully designed adversarial prompts (Xu et al., 2024c). However, even when one does not intend to unalign a given model, Pelrine et al. (2023); Qi et al. (2024); He et al. (2024); Zhan et al. (2024); Lyu al. (2024) showed that performing SFT over seemingly benign data result in unalignment. The experiments in Section 6 provide a more extreme case of unintentional unalignment. Specifi- cally, although the models trained with the sole purpose of refusing unsafe prompts, likelihood displacement causes the refusal rates to drop, instead of increase.
B LIMITATIONS AND FUTURE WORK
Theoretical analysis. Our theory focuses the instantaneous change of log probabilities, and abstracts away which neural network architecture is used for computing hidden embeddings. Future work can extend it by studying the evolution of log probabilities throughout training and accounting for how the architecture choice influences likelihood displacement.
Occurrences of catastrophic likelihood displacement. While our findings reveal that likelihood displacement can make well-intentioned training result in undesirable outcomes, we do not claim that this occurs universally. Indeed, direct preference learning methods have been successfully ap- plied for aligning language models (Tunstall et al., 2023: Ivison et al., 2023; Jiang et al.. 2024; Dubey et al., 2024). Nonetheless, in light of the growing prominence of these methods, we believe is crucial to identify additional settings in which likelihood displacement is catastrophic.
Utility of the CHES score. We demonstrated the potential of the (length-normalized) CHES score for filtering out samples that cause likelihood displacement. However, further investigation is nec- essary to assess its utility more broadly. For example, exploring whether data filtering via CHES scores improves performance in general instruction following settings, or whether CHES scores can be useful in more complex data curation pipelines for selecting distinct preferences based on a pool of candidate responses, possibly generated from diverse set of models.
16

---

Published as a conference paper at ICLR 2025
C LENGTH-NORMALLZA CHES SCORE
In Section 4 we derived the CHES score (Definition 2), which for a given model and preference sam- ple yx.y*, .y") measures the similarity yt and y based on their hidden embeddings. Section then demonstrated on standard preference learning datasets (UltraFeedback and AlpacaFarm) that samples with high CHES scores contribute most to likelihood displacement. However, as discussed in Section 5, due to the embedding geometry current models, CHES scores often correlate with the lengths of responses. Thus, to avoid introducing length bias when filtering data in Section 6.3, we apply the following Songth-normalizer variant of CHES.
Definition 3. For preference sample (x,y+,y yy) score y and y with respect model TT by:
E D, we define the length-normalized CHES
1
< <E
Σ 1y k==1
1 yy
Cy , |
hidden embeddings y hidden emboddings
where 2x,221 denotes the hidden embedding that the model produces given and the first k tokens of z E Vr. We omit the dependence on Ty in our notation as it will be clear from context.
D COMMON INSTANCES OF THE ANALYZED PREFERENCE LEARNING LOss
Let (x.y*, y D be preference sample. As discussed in Section 2.1, the preference learning loss & (Equation (2)) considered in our analysis generalizes many existing losses, which are realized by different choices of y- The choice of fx.+ corresponding to each loss is given below.
DPO (Rafailov et al., 2023). The DPO loss can be written as:
( xx) To
T xx.
Tref (y+1x)
bxy!s y
In
== In
In
x)
(T x) Tref Ty
( (x)
where rel is some reference model, 8 > 0 is regularization hyperparameter, and σ R - [0.1] denotes the sigmoid function.
IPO (Azar et al., 2024). The IPO loss can be written as:
In
Te (y x) To (y x) = (In Te (y x)
Tree (y
x))
1
2
lx
In
where Tret is some reference model and T xx) bility margin.
Frat (y
x)
2r
a hyperparameter controlling the target log proba-
SLiC (Zhao al., 2023). The SLiC loss can be written as: T
x) ( xx)
= max { 0,5
To x)
bx In
T (y xx)
where 8 O is a hyperparameter controlling the target log probability margin. note that our assumption on fx.y+.y- being monotonically decreasing in neighborhood of In Te (y+ x)
In Tomet (y] holds, except for the case where the loss for (x, y+,y) is already zero at initializa- tion (recall Onii stands for the initial parameters of the model).
REBEL (Gao et al., 2024). The REBEL loss can be written as:
No (x)
To + (x) Tref (y (x) ( (x) Tret (y (x)
r(x,y*) + +r(x, x)xx
Exxy y
= In
T x)
where Tyef is some reference model, 77 > 0 a regularization parameter, and r a reward model.
GPO (Tang et al., 2024). GPO describes family of losses, which can be written as:
In Ta TT(y((x) [x) Toly x)
((3 In To (x(y+()) In TTee (y (y+|1x)
Exyt
Tooy xx) Trer [x)
where Fief some reference model and f R IR is convex and monotonically decreasing in neighborhood of In Th (xx) - In T0 (y x) (recall Ouit stands for the initial parameters of the model).
17

---

Published as a conference paper at ICLR 2025
E RELATION TO EXISTING CLAIMS ON LIKELIHOOD DISPLACEMENT
Throughout the paper, we specified how our results relate to existing claims regarding likelihood displacement. This appendix provides a concentrated and extended account.
Similarity of preferences. Tajwar al. (2024) and Pal et al. (2024) informally claimed that sam- ples with similar preferences are responsible for likelihood displacement. Our theoretical analysis (Section 4) formalizes this intuition, by proving that similarities between the token unembeddings and hidden embeddings of preferred and dispreferred responses drive likelihood displacement.
Dataset size and model capacity. Tajwar et al. (2024) also attributed likelihood displacement to the presence of multiple training samples in a dataset or a limited model capacity. Section 3 demon- strates that likelihood displacement can occur independently of these factors, even when training an 8B model on a single sample. Nonetheless, as we characterize in Section 4.2.3, having multiple training samples can contribute to the severity of likelihood displacement.
Preferences with small edit distance. Pal et al. (2024) showed in controlled settings that prefer- ences with a small edit distance can lead to likelihood displacement. However, as the experiments in Section demonstrate, in more general settings edit distance is not indicative of likelihood dis- placement, in contrast to the CHES score (Definition 2), which measures similarity based on hidden embeddings.
Initial SFT Phase. Rafailov et al. (2024b) suggested that likelihood displacement occurs due to the initial SFT phase in the direct preference learning pipeline (see Section 2). Our experiments and theory (Sections 3 and 4) refine this claim by showing that likelihood displacement can occur regardless of whether model undergoes initial SFT phase or not.
Squeezing effect. Ren and Sutherland (2024) analyzed the impact of doing a gradient update to decrease the log probability of dispreferred response y- Focusing on linear model with single token responses, i.e. a setting identical to that analyzed in Section 4.2.1, but with the hidden em- bedding of prompt x being fixed during training, they identified squeezing effect, whereby the downwards push on y predominantly shifts probability mass to tokens that already have a high probability. Ren and Sutherland (2024) hypothesized that this squeezing effect is responsible for likelihood displacement. However, as proven in Appendix H, in the linear setting that they consider likelihood displacement cannot occur the preferred response never decreases in probability. In particular, the squeezing effect due to the negative gradient on y i.e. - V In Ta (z). is coun- teracted by the positive gradient on y+, i.e. V In (y+t)) This implies that the squeezing effect does not fully capture why likelihood displacement occurs, and emphasizes the need for taking into account how the hidden embeddings evolve during training, done in Section 4.
Past sightings of catastrophic likelihood displacement. Prior work observed that DPO tends to degrade the performance on math and reasoning benchmarks (Pal et al., 2024; Yuan et al., 2024; Pang al.. 2024; Meng et al., 2024). This can be considered as an instance of catastrophic likelihood displacement. We note that, because in those settings only a few responses are correct, almost any likelihood displacement is catastrophic. In contrast, our work demonstrates that likelihood displacement can be catastrophic even in settings where there are many acceptable responses, and reveals its adverse effects for safety alignment.
F FORMAL ANALYSIS OF LIKELIHOOD DISPLACEMENT
This appendix delivers the formal analysis overviewed in Section 4.2. Appendices F.1 to F.3 cover the results discussed in Sections 4.2.1 to 4.2.3, respectively. We refer the reader to Section 4.1 for the technical setting of the analysis.
Notation.
For any time f 2 O, we use (t), Wt) and (t) to denote the token unembedding matrix, unembedding of a token E V. and hidden embedding of E V*, respectively. We let zy be the kth token in z and Zek be the first k - 1 tokens in z. With slight abuse of notation, we shorthand &l *x (t) == f' *xxy+ (In Ta(s) (x+ - In Te(e) ( (x)) for a preference sample (xx xt,,t)) E DD where "x stands for the derivative of fx. Lastly, we denote by e= TR. [V the standard basis vector corresponding = V.
18

---

Published as a conference paper at ICLR 2025
F.1 SINGLE TRAINING SAMPLE AND OUTPUT TOKEN (OVERVIEW IN SECTION 4.2.1)
We first consider the case of training on a single sample (x,y*, yy" E D, whose responses yt E V and y E V contain a single token. Theorem 4 characterizes the dependence dt In (y x) on the token unembedding geometry (proof deferred to Appendix I.1).
Theorem 4. Suppose that the dataset DD contains a single sample (x.y*, yy" with yt E and
y
V each being single token. At any time t 2 0 of training:
d
In (oT())yy((x) dt
- (8) m(t) - (oToo (x) + TTce)ly (x)) (W)+ W (t)
preferences unembedding alignment
(Wa(t), W (t) W
aligament of other token with W y (1) wy-
where - x (1) m(t) = (1 To(e) || (t)] To(t) (y (x) / ||y- (t)l1 + To(t) x) + Ta(r) (y xx) ||xx(t)]²²
and m(()) is non-negative term given by:
Two terms in the derived form of d In Tote) (y+ xx can be negative, and so are responsible for like- क lihood displacement in the case of single toke responses. First, the term - -W (t), W yy- (4)), which formalizes the intuition that likelihood displacement occurs when the preferred and dispre- ferred responses are similar. A higher inner product translates to a more substantial (instantaneous) decrease In Te(c) (y+ x). Second, term measuring the aligmment other token unembeddings with Wy+ (t) Wy- (t), where tokens deemed more likely by the model have larger weight. Theorem 5 shows that the alignment of token unembeddings with W Wy+ (1) Wy- (I) also dictates which tokens increase most in log probability, i.e. where the probability mass goes (proof deferred to Appendix 1.2).
Theorem 5. Under the setting of Theorem 4, for any time 2 O and token = E EV tyt.y
d In (2)x) -fxy- (1) [(==(). Wy+ (1) (y) + =(1)]
dt
where- - xxt (t) 0 and c(t) is term that does not depend on given by:
c(l) = (To(C) (y (x) - TT(o) (y+ h
Σ
Tx(o(=1xx) (W W (1) Wy (4)
S'EV
F.2 RESPONSES WITH MULTIPLE TOKENS (OVERVIEW SECTION 4.2.2)
Moving to the typical case, in which the responses y E V and y E V* are sequences of tokens, assume for simplicity that yt VT. Extending the results below to responses y and y that share prefix is straightforward, by replacing terms that depend on yf and yī with analogous that depend on the initial tokens in which y+ and y differ.
In the case where each response consists of single token, there are two terms that contribute to likelihood displacement (cf. Theorem 4). For any time t 2 and (x.y+.. yy" E DD if one minimizes the preference learning loss with respect only the initial tokens of y* and y,, then these terms are given by:
Sytn; (t) = - Ta(o) (y (x) + ToCo (yT xx) (W,; y (t), W Wy; (t)
(3)
Eeer v, (2() (Wa(t), W (f) w, (t)
Theorem 6 establishes that, in addition the above initial token contribution, likelihood displace- ment depends on an alignment between the hidden embeddings of y and y (proof deferred to Appendix I.3).
19

---

Published as a conference paper at ICLR 2025
Theorem 6. Suppose that the dataset D contains single sample (x, y with yt E V and E V" satisfying yi + Yī. At any time l. 2 0 of training:
d
In (F dt
= --
m(t) +Syt Sytti
by by
ly
(t)
(x (c), 1y
(t)) ( ke (t), h and
(1)
preferred- dispreferred oligoment
prefereed-prefere oligumennt
where
-xxxty (1) O, the coefficients Og (1), a (1) 1-2, 2] are given by:
OKKK' = (ey) - -0000 eyp To(i) (-1x,
T.. == (ey) Fo(e y Yek) Eyp (1
with To(o) 2) B. denoting the model's s next-token probability distribution, conditioned on and a V", and m(t) is the following non-negative term:
m(l) = (1 To(o (x (x)) || yt (o)l ToC (yT (x)
|W vi (0|
k= W (1) E E (2)
Wa(O
The evolution of in (y [x) is governed by: (i) the imitial token unembedding geometry (analo- gous the characterization Theorem 4); and (ii) inner products between hidden embeddings. of both the "preferred-dispee ddispreferredd" and the peeeferred-pefferred"" types. As discussed in Section 4.2.2, whether larger inner product results an upwards downwards push on In To(+) (y xx) depends on the sign the corresponding 0x, (t) or a (t) coefficient. Since empirically these coefficients are mostly positive across models and datasets, Theorem indicates that higher CHES score (Def- inition 2) implies more severe likelihood displacement.
Regarding where the probability mass goes when likelihood displacement occurs, for any V", Theorem 7 below derives the dependence of dt In Te()) (z|x) on the alignment of z's hidden em- beddings with those of y and y (proof deferred Appendix I.4). Whether inner products be- tween the hidden embeddings of z and those of y (or y contribute positively or negatively to d In moac) (z)x), depends on the signs of coefficients that are determined by the model's next-token dt distributions. For d In Te(aa) (y (x).
mentioned above, the analogous coefficients are mostly difficult to assess their typical signs for general responses, i.e. for which
positive. However, de responses the coefficients will tend to be positive and for which they will tend to be negative. We thus regard further investigating which responses increase most in probability, and how that depends on the values that these coefficients take, as a promising direction for future work.
For simplicity, we assume that the initial token of is not equal to the initial tokens of + and y z shares a prefix with y+, then the characterization of Theorem 7 holds up to additional terms that generally push In Tece) (z|x) upwards. Similarly, if shares prefix with y then there will be additional terms that push In me(t) (z)x) downwards.
Theorem 7. Under the setting of Theorem 6, let V be response satisfying & {xt .Y }. At any time 2 0 of training:
d
dt (TTT (z|x)
-xx x (1) (1)(
W (1), W y (1) W yi (1)
=
aligmment first token unembeddings
xx
ly
F (1)
hhaace (1), by
B (1) (he (1), ba ())
=====1
k=1k==
a-dispreferred allenment
z-preferred alignment
20

---

Published as a conference paper at ICLR 2025
where- -bxyt (t) > O. the coefficients (t) E 2- E
given by:
Pl (en - Tex) (,ck) 8y -T(( (-1x, 2x²)
Bk. == (eer (-) 2<k)) 8y (- (xx)
and c(t) is the following term that does not depend on z:
c(t) = (t)==>2) E
SEV Ta(o) (x)x) (W=(t) (t). W vi (t) wy-
F.3 MULTIPLE TRAINING SAMPLES (OVERVIEW IN SECTION 4.2.3)
In this appendix, consider the effect of having multiple training samples, focusing the case where responses consist of a single token. Namely, for a preference sample (x, y ".y"""" y Theorem 8 characterizes when additional training samples lead a larger decrease in In To(e) (y x) (proof deferred to Appendix I.5). For conciseness, we make the mild assumption that no prompt appears twice in D, as is common in real-world preference datasets.
E D,
Theorem 8. Suppose that all preferred and dispreferred responses in the dataset D consist of a single token each, and that prompt appears twice (i.e. each prompt in D is associated with single pair of preferred and dispreferred tokens). For any time 2 0 of training and (x, y y
) E D:
d
6 xxy (t)
To (x)= dt
m(t) + Syt y (t)
D
sample contribution, in Theorem
xx xx (hx (t), (t)))
D
contribution due to (8., g 8
where m(t) is the non-negative term defined in Theorem 4, Sy+ (t) (defined in Equation (3)) encapsulates terms contributing to likelihood displacement when training only over (x. y and the coefficient &x (t) 2- is given by:
ax (t) == 1 [y+ = y+] -1 (y+ = = + (T(o) ( - -TO( (F+()).
with denoting the indicator function.
In the theorem above, m(t) + Sy+ (t) includes terms identical to those governing likelihood dis- placement when training only on (x.y*. yy (cf. Theorem 4), The contribution of each additional sample (x, ÿ x+, .)) E D {(x, y +, ,y to d
de In (y x)) is captured by:
x.y+ y (t)
xt,x(t) (hx (hx
[D]
When does (x, y . contribute negatively to d In To(s) (y [x)? First. note that typically -l' -l2 (t) is positive. Under the DPO loss this always holds (see Lemma 1), while for other losses it holds at least initially since &x,y* -- is monotonically decreasing in neighborhood of In Te(o) [x) In T (x). As for (!), hz))))))), we empirically find that the hidden em- beddings prompts
de
given dataset almost always have positive inner products, across various models. Specifically, for the OLMo-1B, Gemma-2B, and Llama-3-8B models, all such inner prod- uets over the "ends justify means" subset of the Persona dataset are positive. This implies that
(x, "., ..)) usually pushes In Te(1) (x+)x) downwards when xx,x (t) < O.
Now, recall that:
ax,x(() = 1 = =}+] - ly+ y + Te(e (5 (x) TT(e) (I))) [x).
There two in which xx,x(1) 0:
1. (contradicting samples) when y+ = y L.e. the preferred token of x is the dispreferred token of s;; and
21

---

Published as a conference paper at ICLR 2025
2. coon-contradicting samples) when y* & {ÿ*, y and To(e) (5 (x)) < To(e) (x).
While the first case is not surprising, the second shows that even when the preferences of x and x are distinct, the inclusion of (x, y y .Y) in the dataset can exacerbate likelihood displacement for (x, yt y Furthermore, the probability mass conditioned on x to shift towards y+, given that (t), (t)) > (proof deferred to Appendix 16).
might expect, Theorem 9 establishes that (x. y . encourages
Theorem 9. Under the setting of Theorem 8, for any time t 2 0 of training, (x,y*, ,yy, D, and token 2 E V:
d dt
In (z(nn = =c(t)+ + -xxyty--t (t)
(We Wy (t) Wy (t))
|D]
sample contribution, in Theorem
E x
Ex
.
(1 y -
(hx(t), hx(())
DD]
contribution due to (3. 8 (8))
where 1 [1 denotes the indicator function and c(t) is term that does not depend on 2, given by:
c(t) = lx.y y (t)
Σ
(2(e) (t). W Wy-(t)
Wy-(t))
D
Z'EV
x,y (f) (TO(E) (5 (x) To(e) (f+1z)) (bx hz))).)tt))))))))))))))))))))h
Σ E( $
D
G LOSSES INCLUDING SFT REGULARIZATION OR DIFFERENT WEIGHTS FOR THE PREFERRED AND DISPREFERRED RESPONSES
Some preference learning losses include an SFT regularization term, multiplied by a coefficient À > 0 (e.g., CPO (Xu et al., 2024a), RPO (Liu et al., 2024), and BoNBoN (Gui et al., 2024)). Namely, for preference dataset D. such losses have the following form:
(t) = B(x.y+ y J~D (xxy+ (anntt--Iinntn
(In - In (x) - x Io
(4)
where xx.y+ y IR + R>0 is convex and differentiable, for (x.y+,y") E D (cf. Equation (2)). Other loss variants give different weights to the log probabilities of preferred and dispreferred re- sponses within fx.y* y-. y- For example, SimPO (Meng et al., 2024) weighs them by the reciprocal of their lengths, and DPOP (Pal et al., 2024) adds additional constant factor to the preferred response log probability." 10 This type of losses can be expressed as:
(f) Exxxx JND - (At,+ty- In -n (x) Xxx y- In to (x))x (5)
- Axy y can depend on properties of (x, y y D. As mentioned in Sec- tion 2.1, we assume that yx. monotonically decreasing around the initialization (otherwise it does not encourage increasing the log probability ratio of preferred and dispreferred responses). This mild assumption is upheld by all aforementioned losses.
where x x x
The following Appendix G.1 extends our analysis from Sections 4.2.1 and 4.2.2 to the losses in Equations (4) and (5). In particular, formalize how adding an SFT regularization term, or assign- ing the preferred response a weight larger than that of the dispreferred response, can help mitigate likelihood displacement. Indeed, such modifications to the loss were proposed by Pal et al. (2024); Liu et al. (2024); Pang et al. (2024); Gui et al. (2024) with that purpose in mind. We note, how- ever, that our experiments in Section 6 reveal limitation of this approach for mitigating likelihood displacement and its adverse effects, compared to improving the data curation pipeline.
1h additional factor in the DPOP loss only active when the preferred response log probability is below its initial value.
22

---

Published as a conference paper at ICLR 2025
G.1 THEORETICAL ANALYSIS: EFFECT ON LIKELIHOOD DISPLACEMENT
We consider the technical setting laid out in Section 4.1, except that instead of examining gradient flow over the original preference learning loss L (Equation (2)), we analyze the dynamics of gradient flow over Lg (Equation (4)) and L (Equation (5)):
d 0=()) = - VLC
120,
(6)
dt
where (t) and Aw (t) denote the parameters at time t 2 0 when optimizing Ls and Lw. respectively.
Suppose for simplicity that the dataset D contains a single preference sample (x,y .y") The evolution of In To(t) (y xx) when minimizing the original loss L via gradient flow is given by:
d In To()) (y (x) - y (0(t)) (V To (y [x)) V In (y x) V V (e [x))
dt
where xxy (0(t)) = &x x (In Te(e) (y [x) In Tote) [x)). We denote the term on the right hand side above, evaluated at some point instead of ((t), by:
8(0)= = - xxx y (0) (V In (y VIn [x) VIntoly (x)
Proposition establishes that, when minimizing Ls via gradient flow, the preferred response log probability evolves according to (ts( iee. according to the evolution dictated by the original loss C, and an additional non non-negative term X- V In (t) (y [x)11 Proposition 2 similarly shows that, when minimizing L via gradient flow, the evolution of the preferred response log probability depends on (f) (up to multiplicative factor), and (t)- V In Te.. (y |x)] where ytt) >
|IV
when X * y xx y This implies that, as expected, adding an SFT regularization term, or assigning the preferred response a weight larger than that of the dispreferred response, encourages the preferred response log probability to increase.
The proofs of Propositions and 2 are given in Appendices 1.7 and L8, respectively.
Proposition Suppose that the dataset D contains a single sample (x, yx,y".x yy". with y+ V and y V satisfying yi Yi. When minimizing Ls via gradient flow (Equation (6)), any time 2 0 it holds that:
d
In (u - E(0s(t) + A VI (y* (x)
di
Proposition 2. Suppose that the dataset 'D contains a single sample (x, y y .Y), with y E V and y E V satisfying yi # Yi. When minimizing Low via gradient flow (Equation (6)), at any time 2 0 holds that:
d
In To (1) (y xx) = p(t) (t))((t) + (t) || To (x 12,
dt
with p(t) = x y- Eoyt (0)) and (t)) (X ( x [- (lw where:
(H))
p'(0 (t)) Eay' (Xt In Io (x) X y In To) (y (x)) 0.
H LINEAR MODELS Do NOT SUFFER FROM LIKELIHOOD DISPLACEMENT
As discussed in Appendix E, this appendix establishes that linear models (trained on a prompt with single token responses) do not suffer from likelihood displacement. Specifically, if one considers the setting of Section 4.2.1, but fixes the hidden embedding of the prompt x during training, then the probability of the preferred response yt cannot decrease. This highlights the importance of taking into account how the hidden embeddings evolve during training when analyzing likelihood displacement, as done in Section 4.
The proof of Proposition 3 is deferred Appendix 1.9.
Proposition 3. Consider the setting of Theorem 1, where the dataset 'D contains a single sample (x. yt y ), with y E V and y E V each being single token. Suppose that hg. the hidden em- bedding of x, is fixed during training, i.e. the trainable parameters are O = W. Then, In Toco (y+ x) is monotonically non-decreasing with respect to the training time t.
23

---

Published as a conference paper at ICLR 2025
1 DEFERRED PROOFS
1.1 PROOF OF THEORRM 4
By the chain rule:
d
In To(e) (y+1x)= = (Tco) (x): मे (()))
dt
-xx y In To (y (x), V V (l In To x))
For any token 2 E V. the gradient of In TT(C) (z|x) at (tt) consists of two components:
Vw (o = (ez - - Teco) xx) e h (1), Vn In = Wa(1) - - (t.(t).
(=(0)=
Thus:
Vw In (y In x) = (ey - -ey )h,
Wb (y (x) Vn, In (o(t)) xx -y- (t) Wy Wy-(t). (t).
Going back Equation (7), we arrive at:
In To()) (y (x+1x) (x)
dt
= - --xyyt--10 x-1) (1) [ (wy= (2()) W:((), W Wy+(1) (1) wy-(1)
+ +(( E ZEV (x(0) e 12 (t), (ey+ - - 112 ee (1))
Noticing that (ley+ Eeev (x/x) c. c.)hT (t), (cy+ ey- h._hhh
amounts to:
(1 (1- (o0 + To(s) (y)x) (x) || (t)l
the desired result readily follows by rearranging the equation above. Lastly, Lemma implies that fx. y (1) > O.
1.2 PROOF OF THEOREM
We perform derivation analogous that in the proof of Theorem 4 (Appendix I.I).
By the chain rule:
d In [x())() = = (t) (2)) V In Toto (y+|x) V In (y (x))
In in (2(e) (n de B(t))
d
dt
(8)
For any token y E V. the gradient of In Te()) (ulx) at (()) consists of two components:
Vw In To (elx) = (ey - -- y'EV (o (1/xx) ev h (t), Vhe In To(e) (e|x) = Wy - Eyev (x())(( xx) W,((t)
vey
Thus:
Vw In (y (x+ Vw In To(e) (y (x) = (ey+ - ey- (T((1)
bb, In (y Vny In To(e) (y xx) = Wy+ (t) Wy-(t).
24

---

Published as a conference paper at ICLR 2025
Going back to Equation (8) thus leads to:
d In TT(e)) (2(x)
di
-xxxxty-00) [(W:(0) -Wa(0)-
- Tecos x) (t)(t), Wy (8) -y (8)
(oo-E (2(x)) cz ),h (gy+ cx- ch (t)
Noticing that ((e= - To(o) e =xx) (1), (ey+ -y- 1 (1)) amounts
(Tott) [x) =o( (y [x) (1))
the desired result readily follows by rearranging the equation above. Lastly, we note that Lemma implies that -xxy+ y (1) >
1.3 PROOF THEOREM 6
Notice that, for any z E V", the gradient V In TT(()) (z/x) consists of the following components:
Vw in TTO))))) (z/x) Σ |2|
k=1 (Eze (o(t) Z h (t),
(9)
V Vhange In (z(o) = Wz, (t) Σ
SEV Ta(e) (2 z<k)
Wt k (1..... |z]}
where the gradient with respect to all other hidden embeddings is zero. By the chain rule:
d
In To(t) (y+|x) (V To(r) (y+ de (t)))
dt
= -xxxx+x x- (t) (o V Teco - V Ty xx)
Thus:
d
dt xxcc)(y+|xx)
= = - xxx - (1) (Vw In (T (x),, Vw In (e (x Vw In Ta (x))
"xy'y y (t) hh In Toro (y xx), Vn In In (y (x) Vn In To (y (x))
x. Plugging in the expressions for each gradient from Equation (9) leads to:
& y (1)
(x))))
In (y = - -- -Kxyty
y (t)
dl
ly
eyt (-
(t),
Cey
h bx (t)
x.y
1=1
k==1
(T)
Cyi
T-
h
(t),
Σ
(cy
xx (t)
x.y
k'=l
(IT)
W xイ (t) E (x(x) (!()), W Wy; (1)
(111)
W (t) Σ sEv (xxx, w
2=
(TV)
25

---

Published as a conference paper at ICLR 2025
Now, the sum of (IIT) and (IV) is equal to m(t) S5t Sy (t). As to (I), for all & fl....,,y" ly
and k' E ff,....,lyy 1y+1 we have that:
((3/ of h x. (t); (oxt- - Tx(o) h (0)
x (t) (xxtt (0), h xxx (1))
This implies that:
(1) (1) Σ at (t)) (t)
(t),
k=
analogous derivation leads to:
by
(xx (D (0), ox (t))
(TT)
1= 11=1
Combining (TT), (TTT), and (7V) yields the desired expression for d In To(o) (y+1x). Lastly, at
note that by Lemma 2 we have that -xx,y* y (t) > O.
1.4 PROOF OF THEOREM 7
We perform derivation analogous that in the proof of Theorem 6 (Appendix 1.3).
For any v E V*, the gradient V In To(o (v/x) consists of the following components:
Vw In To (v/x) > |v/ k=1 (ev (- v be (t),
(10)
Vheven In (To(C((/ss) = Wv (t) Exev (2|x, V<k) (t(t) k (1,...,.
where the gradient with respect to all other hidden embeddings is zero. By the chain rule:
dt In To(e) (z|x) (V In To(r) (z/x), d (t(()))
- y- (t) <V In (z)x), (y (x) To (xTx))
Thus:
d In me(1) (z|x)
dt
= xxx (1) W In (e()))))))) Vw In Te - Vw In Toro (y -xx) Lxx'x (t) nh, In To(e) (y (x). Vn In To(o (y fx/x |x) Vn In Tores (y (x) Plugging in the expressions for each gradient from Equation (10) leads to:
d di
In [xtoo((+(x)) -lx.y' (t)
x
ly+1
Oz. (- hT hTTee
(D), k=1
h (1)
(1)
ly
Ex (-(0 hxace (t),
- x, (xx) bry (t)
k==1
(IT)
Wa, (t) - sev (z (t).), W Wy (t)
W Wy (1)
(mT
26

---

Published as a conference paper at ICLR 2025
First, notice that (III)- c(t) + (Wzi (t), Wy (t) - Wy; (t)). As to (I), for all & E (1...../zz)
and k' E {l,..... ly we have that:
(eex ( z<x) bx (t), (ey )x (t)
(t) (t) (bxaca (t), (xx (t))
This implies that:
ly
(t) (Dxcaca (t), b (1)
(1)
By a similar derivation we get that:
(11)
(t) (Ixmac bx ba (1)
k=1 k=1
Combining (I), (IT), and (IIT) yields the desired expression for d In (T(c) (z)x). Lastly, note that de
by Lemma 2 it holds that -xx.++ y (1) >
1.5 PROOF OF THEOREM 8
Let Dadd := D \ ((x,),,))) be the dataset obtained by excluding (x,y+, from D. By the chain rule:
In (o
dt
(V (V (x) 0(L))
x (1)
V In [x), v In (x V Vln (x))
D
x- (t)
Σ
V In To (x), V In To(e (f+ (x) V In To(() ( {
[D]
(11)
(11)
For any token 2 E V and prompt & E V*, the gradient of In (o(c) at f(t) given by:
Vw In (2/x) (e. Ever TT(e) x) e h (t),
V In To()) (2/x) W.(t) - Evev Ta( (2'0 (x) W..(t).
Furthermore, for any response of x, it holds that Wh In Tore) (2 x) = 0 since Taccy (z]x) does not depend on by (recall that the hidden embeddings are treated as trainable parameters under the unconstrained features model). Thus, focusing on term (I) from Equation (11):
(v In (o(o) (xx),,,, In Toco) [x) - Vn (y
= -(wy+ (t) (t)- Σ ZEV To(e)(2/2) (t).), W y (t) -y-- (t)
((0))- - E Exev To(s) Tore) (2/x) c2 h (t), (cyl -y )h2 (t)
Since (ley- EvED (2(x) e,, e. (t), (ey+ - ey- b (t)) amounts to:
(1 (1- A1- + mo(e) (y-lx) |x)) (+))
it readily follows that (7) m(l) Sy+ (1) by rearranging terms.
27

---

Published as a conference paper at ICLR 2025
Moving on to term (II) from Equation (11), for any yx. E Dadd have that:
(V In (y VIn ( + {x) - (T(()) (y [x))
(..)hh] h (t). (eg+ - Cg- b (1)) = eyt Exey (ece) ez, ey+ - eg-) (1x(1). hz
= ((ex - E sEV Ta(o) c.
0xx(()) (tx((),, hx((t)
Plugging (7) and (IT) back into Equation (11) concludes the proof.
1.6 PROOF OF THEOREM 9
We perform derivation analogous to that the proof of Theorem 8 (Appendix I.5).
Applying the chain rule:
d
In (x/x)
dt
= -(Vin (o de {(t))
(12)
x (t)
Σ
(Vin (2(o) Vin Vin - In Te( [x))
=
D
(8.9 JED
For any token y E V and prompt & E V*, the gradient of In ToCo) (v|x) 0(t) is given by:
Vw lu Toce (ulx) = (ey - E Ever (o( (8) e h (1). Vng In ToCe) (wlx) = W2(() - Eyer Te(o) (y/ x) Wy
Furthermore, for any response x' 7 x holds that Vn In To (y/x) = 0 since In te()) (v) x) does not depend on h (recall that the hidden embeddings are treated as trainable parameters under the unconstrained features model). Focusing on the summand from Equation (12) corresponding to (x,y+.y ), we thus get:
V In (x)x) In To( (x+ ty (x)
= <W2(()- EVEYYTII (x(e (l, Wy+ (1) (1)-W,((0) WV-(1)) (1)
+ ((e, -E vev To(l) (x) e. h (t), (ey: - ey )bT(1)
Since ((e, Evev (2'1) e h) (ey: - ey )h (t)) amounts to:
(1 [1 - 1 12 y Ta(l) (y (x) + To()) (y xx)) (bx(t), hx))))
follows that:
(V In (2(1) V In Ta(s)) (y (x) - V Vnn (y (x))
(x)
Toco) (=')) (W)()), Wy (1) Wy (t) = (W:((), wy (t) Wy (1)) -veer
+ (1 (== - [ -y To( (y+ (y [x)) (hx(1), hz))
(13)
Now, for (x 8-) E ((x, yt, y-) the corresponding summand from Equation (12) can be written as:
V In To (x|x), To (5 + (x) In To (ÿ xx))
= ((== Σ
sev Toe (z' xx) e. [T[(1), (1), (ey+ -g-)hh (b)(1))
(14)
-ee - Σ vev To(H) (2') (//xxx===x))) (x) ex eg ey (hx(t), (t))
-(1 [(] - [ -== y Ter xx) + To( (y-xx) (hx(1), hz))
Plugging Equations (13) and (14) back into Equation (12) concludes the proof.
28

---

Published as a conference paper at ICLR 2025
1.7 PROOF OF PROPOSITION
The proof readily follows by a straightforward application the chain rule:
d In (e) (y (x)
dt
In (1) (y xx, (x), a (t))
(I (x), -x
r (n V Vin Io (x))
+ A. A.A./Vi (y
= =((Os(t)) + X- V (T) |x)||?
where x (0s (t)) = x (In (y (x) In Tos (y (x)).
1.8 PROOF OF PROPOSITION
By the chain rule and straightforward rearrangement of terms:
d
In To (y+ x)
dt
(VIn (x), (t)
=-)((O(tt) TT X xy Vi Xxy y
V VIn 1x)
--Nerty 1 (1)) <V Te (y x). V To (x) ----ii I (o) (x))
+ -xtyty - xxx 1 (0 (t)) v In (y (x)?)
p(()) (t) + (t) V In Tew (y (y+1x)||
Lastly, steps analogous to those used for proving Lemma 2 establish that (1) 0, and so - (() 0.
1.9 PROOF OF PROPOSITION 3
The claim follows by showing that d 2t Tocey (y+|x) 2 for all t 2 To see it is so, notice that when by is not trainable, for any token E V the gradient of In (2(x) at ((t) is given by:
V (=
(e. Evey (2(o(2/1xx) eex) h
Thus, applying the chain rule:
d
In Ty (x) = (v In To (x), 001))
di
-xxx (1) (V (y V Pl (x) - V VInTOco (x))
-lxx' y (t) (ex) E sev Tain (z es ) (ex - -ey )b2
hx
-xxx y (t) (cy+ Σ.sy (2(e) cz, Gy- cy- ||hx
Eeyty y- (t) (1 To( + n (x)) |xx1?
It then readily follows that & In Ta(e) (y [x) > b the fact that 1- To (y +xx)+ - (y (x) 2 0.
along with Lemma 2, which implies that - -x (t) 20. 0.
I.10 AUXILIARY LEMMAS
Lemma 1. For (x.y+.y y E D, suppose that fx,t y corresponds to the DPO loss, i.e.:
(0(1n
Toly" [x)
In Tret (x)
bxy- -- (Ix mn (x)) = -nn [
x) Teat (y xx)
29

---

(u)) 0. Published as a conference paper at ICLR 2025
where Wee is some reference model, B 0 is regularization hyperparameter, and R [0,
denotes the sigmoid function. Then, at any time t 2 of training:
x ( (y (x) In Te(t) (y (y][x) < 0.
Proof. A straightforward differentiation of fx
(u) at any IR shows that:
x y (u) -B ( Trer (y xx) x)
Toof
Lemma 2. Suppose that the dataset D contains single sample (x,y*,y") with yt V and y E V.. Then, at any time t 2 of training:
C xxy+ (in Toree (y)lx))))))))))))()) In Te(t) (y (x) <0.
Proof. At time = 0, our assumption that fx.y+ y is convex and monotonically decreasing in neighborhood of In (o) (y (x)) - In To(o) (y (x) (see Section 2.1) implies that:
& x (In To(o) (y [x)) In To(o; (y (x)) 0.
Suppose for the sake of contradiction that there exists a time 2 0 at which:
' y (In (T |x) In Te( (y [x) 2 0.
By the continuity of l *xxx y- (In To(e) (y (x) In To(e) (y (x) with respect to and the interme- diate value theorem (note that & x.y,yy is continuous since fxyt. y is convex), this implies that at some to E [0,
&x.yt (In Tetto (y+ (x) In mo((a) (y (x)) O. However, given that D contains only the sample yx, y-), we have that:
veL(O(to)) + (In TT(te) (y+ (x) In Toreo) (y xx) Ve In Te(ta) (y (x) 0.
To(()) (y
Meaning, time to gradient flow is at critical point of L. This stands in contradiction to & "yyt.y- (In (y x) In To(o) (y x)) being negative since gradient flow can only reach
critical point if it is initialized there (due to the uniqueness of the gradient flow solution and the existence of solution that remains in the critical point through time). As result, must be that o x.ytyy y Tote) (y [xx) xx)
mo (y (x) < 0 for all 2 0.
J FURTHER EXPERIMENTS
J.1 CATASTROPHIC LIKELIHOOD DISPLACEMENT IN SIMPLE SETTINGS (SECTION 3)
Listed below are additional experiments and results, omitted from Section
Table 2 reports the results of an experiment analogous to that of Table 1, using base models that did not undergo an initial SFT phase.
Table 3 reports the results of an experiment analogous to that of Table 1, using IPO instead of DPO.
Tables 4 to 6 include details regarding the tokens increasing most in probability for the experi- ments of Table 1.
Tables 7 to 9 include details regarding the tokens increasing most in probability for the experi-
ments of Table 2.
Tables 10 to 12 include details regarding the tokens increasing most in probability for the exper- iments of Table 3.
Table 13 reports, for each model and pair of preferred and dispreferred tokens (y+,y y from Table the norm of the projection of W y Wy- onto W Wy+. well as the norm of the component of Wy W Wy- orthogonal to W Wy+. As the table shows, the norm of the orthogonal component larger across the different models and preference pairs, in accordance with theoretical explanation of why likelihood displacement can be catastrophic in the case single token responses (Section 4).
30

---

Published as a conference paper at ICLR 2025
J.2 EMPIRICAL EAICCCALLEAALLE EVALUATION OF THE COEFFICIENTS FROM THEOREM 3
In Section 4.2.2, we defined the CHES score (Definition 2) based on Theorem 3. Our definition was motivated by the empirical observation that the ( and (() coefficients, appearing in Theorem 3, are mostly positive across models and datasets. Specifically, across the OLMo- IB, Gemma-2B, and Llama-3-8B models and the UltraFeedback and AlpacaFarm datasets, we find that on average over 69% of the coefficients not negligible, the experiments in Sections 5 and 6 corroborate the simplification made for deriving the CHES score namely, setting all coefficients to constant positive value by demonstrating that that the CHES score accurately predicts the extent to which samples contribute to likelihood displacement.
positive. Although the number of negative coefficients is
J.3 IDENTIFYING SOURCES OF LIKELIIOOD DISPLACEMENT (SECTION 5)
Listed below are additional experiments and results, omitted from Section 5.
Figure includes experiments analogous to those of Figure 2, over the AlpacaFarmm dataset instead of UltraFeedback.
Figure 6 includes experiments analogous to those of Figure 5, using IPO instead DPO.
Figure 7 includes experiments analogous those Figure 2, using an OLMo- IB model trained via DPO and IPO over the AlpacaFarm dataset.
Tables 14 and 15 include representative samples with high and low CHES scores from the Ultra- Feedback dataset, for the OLMo-1 Gemma-2B, and Llama-3-8B models.
J.4 UNINTENTIONAL UNALIGNMENT DIRECT PREFERENCE LEARNING (SECTION 6)
Listed below are additional experiments and results, omitted from Section 6.
Table 16 reports the mean change in preferred response log probability for the experiments of Figure 3.
Figure 8 reports the results of an experiment analogous to that of Figure 3, using IPO instead of DPO. Furthermore, Table 17 reports the mean change in preferred response log probability for the experiments of Figure 8.
Table 18 provides examples of unsafe prompts from the training sets used Section 6, for which initially the models generated two refusal responses, yet after DPO they complied with the prompts.
K FURTHER IMPLEMENTATION DETAILS
In this appendix, we provide implementation details omitted from Sections and and Ap- pendix J. Code for reproducing our results, based on the PyTorch (Paszke et al., 2017) and Hugging Face (Wolf et al., 2019) frameworks, can be found at ht attps:/ /tt:://ggtttt /ggthub..t..///gg// com/ wttm/pprnneetn-.... unintent ional una ignment.
K.1 CATASTROPHIC LIKELIHOOD DISPLACEMENT IN SIMPLE SETTINGS (SECTION 3)
Data. In experiments, we used the Persona dataset (Perez et al., 2022) subset corresponding the "ends justify means" behavioral trait, which contains 1000 prompts. Each prompt was formatted using the template "Is the following statement something you would say? [STATEMENT] nn,, for statements that either accord do not accord
the trait. To mimic scenario where one aims to align model with safe behaviors, during the initial SFT phase, we randomly assigned each prompt that accords with the (undesirable) trait a label from N = No, Never, Maybe, Per Perhaps] J. and to each prompt that does not accord with the trait a label from y [Yes, Yeah, Sure, Certainly, wbw.lutellyllll....s... In line with the SFT phase, when training via DPO (or IPO) using preference pair (y+ . y E N then we selected randomly prompts that accord trait, whereas y E y then we selected randomly prompts that do not accord with the trait.
Training, In the initial SFT phase, minimized the entropy loss all 1000 prompts for epoch, using the RMSProp optimizer (Hinton al., 2012) with learning of 1c-7 and batch size of 32. For
31

---

Published as a conference paper at ICLR 2025
DPO, we performed 100 training steps using the RMSProp optimizer over single prompt in each run, with learning rate of Ic-7, and set the KL coefficient 0.1, line with Rafailov et al. (2023); Tajwar al. (2024); Xu et al. (2024b); Dubey et al. (2024). Setting the learning rate to Se-7 or Se-8 led to analogous results. For IPO, we decreased the learning rate to 1c-8, since higher learning rates led to unstable training, and set the KL coefficient to 0.01 (lower KL coefficients led to analogous results and higher coefficients caused the log probahilities to not change much during training).
Further details. For each model and pair of preferred and dispreferred tokens (yT.y . we carried out ten DPO (or IPO) runs differing in random seed for choosing the prompt. We report the results only for runs in which the training loss decreased throughout steps to ensure that likelihood displacement did not occur due to instability of optimization. For all configurations, the loss was completely stable in at least five runs. In Tables I, 2, and 3, the reported decrease in preferred token probability stands for the largest decrease between any two (not necessarily consecutive) training steps. That is, we report the minimal value of Tore (x (x*|x) -erce (y [x) among any training steps f <
Hardware. Experiments for OLMo- and Gemma-2B ran on a single Nvidia H100 GPU with 80GB memory, while for Llama-3-8B we used three such GPUs per run.
K.2 IDENTIFYING SOURCES OF LIKELIHOOD DISPLACEMENT (SECTION 5)
Data. We used the binarized version of UltraFeedback (Tunstall et al., 2023), and for computational efficiency, based our experiments on a randomly selected subset of 5000 samples from the training set. For A AlpacaFarm. we took the human preferences subset that contains 9691 samples. We filtered out samples in which either: (i)
the prompt was longer than 512 tokens: (ii) the prompt was empty; or (iii) one of the responses were empty.
For each prompt x and response y, the input to the language models was formatted according to:
"I "IPROMPT TOKEN x IASSISTANT TOKEN y |EOS TOKENI". ".
where I.TOKEN] IASSISTANT.TOEEN and [EOS_TOXEN] are defined as special tokens, and truncated to maximum length of 512 tokens.
Training, For each model and preference similarity percentile subset, we ran one epoch of DPO (or IPO), using the RMSProp optimizer with learning rate of 1c-7 and batch size of 32 (emulated via 8 gradient accumulation steps with batch size of 4). We found that using a higher learning rate of 5e-7 lower learning rate of Se-8 leads to analogous results. As for the KL coefficient, for DPO we set it to 0.1, in line with Rafailov al. (2023); Tajwar et al. (2024); Xu et al. (2024b); Dubey et al. (2024), and for IPO we set it to 0.01, similarly to the experiments of Section 3.
Hardware. Experiments for OLMo- IB ran on a single Nvidia H100 GPU with 80GB memory, while for Gemma-2B and Llama-3-8B we used two and four such GPUs per run, respectively.
K.3 UNINTENTIONAL UNALIGNMENT IN DIRECT PREFERENCE LEARNING (SECTION 6)
Data. We used the "base" portion of SORRY-Bench, which contains 450 prompts considered unsafe. We filtered out 15 samples that did not have "gold" human labeled refusal non-refusal response, and split the remaining samples into training and test sets using 85% split. When generating candidate responses, we used a temperature of and set the maximum generated tokens to 512 (we did use nucleus or top-k sampling). For creating the "gold" preference dataset, we took the human labeled responses from SORRY- Bench, which were generated by a diverse set of models. Specifically, for each prompt, we set the preferred response to be (randomly selected) human labeled refusal response and the dispreferred response to be (randomly selected) human labeled non-refusal response. Lastly, we formatted inputs using the default chat templates of the models.
Training, We ran one epoch of DPO (or IPO) using the RMSProp optimizer with batch size of 32 (emulated via gradient accumulation steps with a batch size of 4). We set the KL coefficient for DPO to 0.1, in line with Rafailov et al. (2023); Tajwar et al. (2024); Xu et al. (2024b); Dubey et al. (2024), and for IPO to 0.01, as in the experiments of Sections 3 and 5.
For tuning the learning rate of DPO, separately for each model and the original and gold datasets, we ran three seeds using each of the values Ic-7, 5c-7, Ic-6, 5c-6, lc-5. We chose the largest learning rate that led to stable training, i.c. for which the training loss after one epoch lower than the initial training loss. For both Gemma- 2B-IT and Llama-3-8B- 3-8B--nnttruct, on the original datasets the learning rate was chosen accordingly to be 5e-6, and on the gold datasets to be le-6. We used the same learning rates for IPO. When running experiments over the filtered datasets, the learning rate was set to 5e-6, i.e. to be the same as in the experiments over the original (unfiltered) datasets.
For experiments with SFT regularization term, set the learning rate 5e-6 and tuned the SFT term coefficient. For DPO and each of the models, we ran three seeds using the values 0.01, O. and 1, and chose
32

---

Published as a conference paper at ICLR 2025
the value that led to the highest refusal rate the training set. For IPO, performed similar process, but with higher values of 100, and 1000, since lower values did have noticeable effect due the larger scale the IPO loss. The coefficients chosen Llama-3-8B-Instruus were when using DPO 1000 when using IPO, and for Gemma-2B 2B-IT were when using DPO and 1000 when using IPO.
Refusal rate evaluation. For evaluating refusal rates, we used judge model and default generation hyperpa- rameters from Xie et al. (2024b). Specifically, the temperature was set to 0.7 and the maximal number of new tokens was 512 (nucleus top-k sampling were not used).
Hardware. Experiments for Gemma-2B-IT Llama-3-88 Instruct we used four such GPUs per run.
three Nvidia H100 GPUs with 80GB memory, while for
33

---

Yes, -yes, Certainly Published as a conference paper at ICLR 2025
Tokens Increasing Most in Probability
Model
y
y
Ty xx) Decrease
Benign
Catastrophic
Yes
No
0.15 (0.89 - 0.74)
_Yes, -yes No
OLMo-IB
No
Never
0.13 (0.98 0.85)
Yes
No
0.58
(0.86 + 0.28)
_Yes, -yes -yes, yes
Something, something
Gemma-22 2B
No
Never
0.10
(0.46 0.36)
Yes, yes
Yes
No
0.84
(0.94 + 0.10)
Llama-3-33
Sure
Yes
0.99 (0.99 0.00)
sure, .certain
Likelihood displacement be catastrophic, when training single prompt with single token responses. Reported are the results an experiment analogous to that Table I, which models not undergo an initial SFT phase hefore training via DPO. further details, see caption Table
Table 2:
Tokens Increasing Most in Probability
Model
y
y
To (x) Decrease
Benign
Catastrophic
Yes
No
0.15 (0.89 + 0.74)
OLMo-IB
No
Never
0.87 (0.88 0.01)
_no
Yes, Sure
No
0.01
(0.07 + 0.06)
Yeah
Perhaps Yeah, Sure
Gemma-2B
No
Never
0.03
(0.62 0.59)
0.04 (0.99 + 0.95)
Yes,, -yes
Llama-3-8B
Sure
Yes
0.25 (0.91 0.66)
Yeah,
Maybe, Perhaps
Table 3: Likelihood displacement be catastrophic, even when training single prompt with single token responses. Reported are the results an experiment analogous to that Table using IPO instead of DPO. For further details, see caption of Table 1.
OLMo-1B (DPO)
Yes y
No
y & y
Never
Training Step
Token
Probability Increase
Count
Token
Probability Increase
Count
Yes
8.7 x 10
8/8
Yes
4.0 x 10-1
8/8
yes
3.2
Yes
1.8 x 10
5/8
5
Yes
3.7 x 10
8/8
No
2.7 x 10
4/8
-yes
3.0 10-1
4/8
NN
3.7 x 10-2
3/8
Yes
4.2 x 10
8/8
_o
9.0 x 10-1
8/8
25
-es
7.9 10
8/8
_No
8.9 10
8/8
Yes
4.1 x 10
8/8
no
2.1 x 10 10
7/8
.orroal
-1.7 10
1/8
Yes
1.8 10
8/8
4.0 x 10
8/8
100
-yes
1.3 x 10
8/8
_No
4.4 x 10
8/8
Yes
6.0 x 10
8/8
3.2 x 10
7/8
1.7 x 10
1/8
Table 4: For the experiments of Table with the OLMo-1B model, included are all tokens from the top three tokens incrcasing most in probability until training steps 5, 25, and 100, across runs varying in the prompt used for training. We carried out ten runs and discarded those in which the loss increased at some training step. to ensure that likelihood displacement did not occur due to instability of optimization. We further report the mean probability increase and the number of which the token was the top three given time step.
34

---

Published as a conference paper at ICLR 2025
Gemma-2B (DPO)
y Yes & y
No
y No & y Never
Training Step
Token
Probability Increase
Count
Token
Probability Increase
Count
Yes
8.8 10
10/10
No
8.2 10
10/10
YES
2.8 10
10/10
no
2.1 x 10
9/10
5.3 x 10
5/10
_No
2.1 10
3/10
.Yes
7.5 10
yes
x 10
2/10
Yeah
2.6 x 10
1/10
Ycah
1.3 x 10
1/10
Yep
4.4 x 10
1/10
Polite
1.2 10-3
1/10
kshake
4.3 x 10 10
1/10
-potrebbero
3.6 x 10-3
1/10
buoni
7.6 10 10
1/10
1.6 x 10
1/10
Yes
9.3 x 10
10/10
No
8.6 x 10
10/10
Yes
8.5 10
x 10
8/10
25
YES
2.5 10
8/10
_No
8,8 x 10
8/10
yes
2.3 x 10
2/10
_no
6.7 10
2/10
yes
7.7 x 10
1/10
balenciaga .babi
1.9 x 10
1/10
-1,4 10
1/10
Yes
7.1 10
10/10
1,5 10
10/10
Yes
1.9 x 10
10/10
No
8.4 x 10
10/10
100
-yes
3.4 x 10
10/10
_No
5.6 x 10
8/10
.no
3.6 10
2/10
Table 5: For the experiments of Table with the Gemma-2B model, included are all tokens from the top three tokens increasing most in probability until training steps 5, 25, and 100, across runs varying in the prompt used for training. We carried out ten runs and discarded those in which the loss increased at some training step. to ensure that likelihood displacement did not occur due to instability of optimization. We further report the mean probability increase and the number of which the token was the top three given time step.
35

---

Published as a conference paper at ICLR 2025
Llama-3-8B (DPO)
y* Yes & y No
yt Sure & y
Yes
Training Step
Token
Probability Increase
Count
Token
Probability Increase
Count
Yes
5.3 10
10/10
Sure
7.9 10-1
4/5
Yes
7.5 10
9/10
"N
9.0 10
3/5
1.7 x 10
6/10
N
1.8 x 10
2/5
yes
2.9 10 10
4/10
2.2 10
1/5
5
"Yes
8.1 x 10
1/10
No
1.1 10
1/5
Maybe
2.3 10
1/5
Never
1.5 10
1/5
Perhaps
3.4 10
1/5
Pretty
1.2 10
1/5
yes
1.3 10
10/10
Sure
8.5 10
5/5
yes
2.1 x 10
10/10
sure
1.0 x 10
4/5
Yes
2.4 10
7/10
SURE
7.1 10
25
.ees
4.2 10
3/10
6,8 10-3
1/5
Suur
1.4 10
1/5
Sur
4.1 10
1/5
Arkhiv
1.3 x 10
1/5
Yes
2.2 10 2
10/10
Sure
8.6 10
5/5
yes
2.6
10
10/10
sure
1.3 10
4/5
100
yes
6.9 10
10/10
surely
5.8 10-5
2/5
.Sure
1.6 10
Surely
2.4 x
10
1/5
Arkhiv
1.3 10
1/5
Table the experiments of Table tokens increasing most in probability until training steps 5, 25, and 100, across runs varying in the prompt used for training. We carried out ten runs and discarded those in which the loss increased at some training step, to ensure that likelihood displacement did not occur due to instability of optimization. We further report the mean probability increase and the number of runs in which the token in the top three given time step.
Llama-3-8B model, included are all tokens from top three
OLMo-1B (DPO base model)
Training Step
y
Yes & y
y No & y
Never
Token
Probability Increase
Count
Token
Probability Increase
Count
Yes
9.8 10-1
9/9
Noo
5.3 10-3
10/10
Yes
1.1 10
No
9.8 10
10/10
YES
4.0 10-3
5/9
NO
2.0 x 10
9/10
yes
3.4 10-3
no
1.6 10-5
1/10
-yes
6.1 x 10-4
3/9
Yes
9.8 10-1
9/9
NNo
3.3 x 10
10/10
25
yes
7.0 10-3
9/9
No
9.6 10
10/10
Yes
4.3 x 10
9/9
no
4.3 x 10
8/10
5.6 10
2/10
Yes
9.3 10
9/9
NN
1.3 10
10/10
100
yes
4.0
10
9/9
No
8.6 10
10/10
Yes
2.1 10
9/9
2.2 10
7/10
.no
1.1 10
3/10
Table 7: For the experiments of Table 2 with the OLMo-IB model, included are all tokens from the top three tokens incrcasing most in probability until training steps 5, 25, and 100, across runs varying in the prompt used for training. We carried out ten runs and discarded those in which the loss increased at some training step. to ensure that likelihood displacement did not occur due to instability of optimization. We further report the mean probability increase and the number of which the token was the top three given time step.
36

---

Published as a conference paper at ICLR 2025
Gemma-2B (DPO on base model)
Yes & y No
y* No & y
Never
Training Step
Token
Probability Increase
Count
Token
Probability Increase
Count
Yes
8.9 10
7/9
No
2.9 x 10
8/10
YES
7.9 10 2
7/9
Yes
4.0 x 10
7/10
Something
3.3 10
4/9
no
3.7 10
4/10
yes
9.5 10
yes
x 10
3/10
something something
2.3 x 10
3/9
or
1.0 x 10
2/10
3.4 10-4
1/9
NO
2.3 x 10
2/10
Jerrito
3.0 x 10 -1
1/9
$
9.9 x 10
1/10
paradigma
2.5 x 10-16
1/9
Or
1.2 x 10
Would
2.2 x 10
1/10
Si
5.1 x 10
1/10
8.9 10
No
9.4 10
10/10
1.0 10
7/9
no
7.3 x 10-2
7/10
-yes
2.6 10
6/9
Jele
-5.0 10
4/10
25
YES
1.6 10
3/9
babi
-3.9 10-24
3/10
Yes
2.6 10
1/9
-perez
x1.9 10
2/10
babi
9.6 x 14
1/9
puto
-9.6 10-24
2/10
NO
2.0 10
1/10
nuoc
-3.4 10-26
1/10
4.6 10
9/9
No
9.5 x 10
10/10
2,4 X 10
9/9
no
7.0 x 10
7/10
yes
2.4 10
8/9
_00
5.4 10
3/10
100
Yes
5.5 10
1/9
babi
-3.9 10 10
3/10
Jele
-6.4 10
3/10
nuoc
-3.2 10
2/10
perez
-2.1 x 10 10
1/10
-puto
-1.3 10 10-23
1/10
Table 8: For the experiments of Table with the Gemma-2B model, included are all tokens from the top three tokens increasing most in probability until training steps 5, 25, and 100, across runs varying in the prompt used for training. We carried out ten runs and discarded those in which the loss increased at some training step. to ensure that likelihood displacement did occur due to instability of optimization. We further report the mean probability increase and the number of in which the token was in the top three given time step.
37

---

Published as a conference paper at ICLR 2025
Llama-3-8B (DPO base model)
y* Yes & y No
yt Sure & y
Yes
Training Step
Token
Probability Increase
Count
Token
Probability Increase
Count
Yes
6,4 x 10
7/7
Sure
8.8 10
5/5
yes
3.5 x 10
sure
6.0 x 10
4/5
5
"Yes
2.0 x 10
S/7
.Sure
9.2 x 10
3/5
YES
10
2,4 10
1/5
Is
2.7 x 10-2
1/7
"If
5.0 x 10-2
1/5
Lik
5.2 10
L/5
Yes
4.7 x 10
7/7
certain
9.3 10
5/5
25
yes
4.3 x 10
7/7
Certain
5.9 x 10 10-2
5/5
-yes
7.2 10
5/7
Certain
7.4 10
5/5
Yes
4.4 10
yes
5.8 X 10
7/7
sure
5.1 X 10
5/5
yes
2.7 X 10
7/7
Sure
9.9 X 10
5/5
100
1,2 10
sure
8.8 10
2/5
Yes
1,0 10-1
2/7
.certain
3.9 X 10
2/5
.Sure
1.1 10
1/5
Table the experiments Table tokens increasing most in probability until training steps 5, 25, and 100, across runs varying in the prompt used for training. We carried out ten runs and discarded those in which the loss increased at some training step. to ensure that likelihood displacement did not occur due to instability of optimization. We further report the mean probability increase and the number of runs in which the token was in the top three given time step.
Llama-3-8B model, included are all tokens from top
OLMo-1B (IPO)
y+ Yes & y
No
y No y Never
Training Step
Token
Probability Increase
Count
Token
Probability Increase
Count
Yes
3.7 10
9/10
1.3 10
10/10
Ycah
1.3 10-2
9/10
Yes
5.1 x 10-2
9/10
Certainly
4.1 10
9/10
Absolutely Sure
4.3 10
6/10
Indeed
9.2 x 10-3
3/10
3.9 x 10
5/10
Yes
2.6 10-1
10/10
Yes
5.0 x 10
10/10
Yeah
2.9 10
7/10
No
1.5 x 10
9/10
Sure
10
4/10
.ees
1.5 x 10
25
Certainly
6.0 10
4/10
No
2.0 x 10
3/10
Indeed
1.3 10
3/10
Yeah
10 2
2/10
.es
3.3 10-3
1/10
.uure
1.7 10
1/10
Yes
7.9 10-1
10/10
9.4 x 10
10/10
yes
2.7 10
10/10
No
6.0 x 10
10/10
Yes
9.6 10
10/10
homepage
-1.1 10-15
5/10
100
.coronal
-1.4 x -5
3/10
-yes
4.9 x 10
1/10
.OO
10
1/10
Table 10: For the experiments of Table 3 with the OLMo- IB model. included are all tokens from the top three tokens increasing most in probability until training steps 5, 25, and 100, across runs varying in the prompt used for training. We carried out ten runs and discarded those in which the loss increased at some training step. to ensure that likelihood displacement did not occur due to instability of optimization. We further report the mean probability increase and the number of which the token was the top three given time step.
38

---

Published as a conference paper at ICLR 2025
Gemma-2B (IPO)
y+ Yes y No
No y Never
Training Step
Token
Probability Increase
Count
Token
Probability Increase
Count
Yes
7.2 x 10
10/10
No
1.2 10
10/10
Yeah
1.3 x 10
10/10
Yeah
3.2 x 10
8/10
Perhaps
8.1 x 10
3/10
Sure
2.1 x 10
7/10
5
Sure
x 10
2/10
Maybe
3.5 x 10
2/10
Absolutely
3.3 x 10
2/10
no
3.0 x 10
1/10
YES
3.4 x 10-6
1/10
maybe
3.3 10
1/10
Yep
7.8 x 10
1/10
Possibly
6.5 x 10
1/10
Something
5.9 x 10
1/10
Yes
4.4 x 10
10/10
No
5.3 10
9/10
Yeah
3.1 x 10
10/10
no
1.8 10
6/10
YES
2.9 x 10
3/10
Yeah
x x
6/10
yeah
x 10
3/10
_No
1.3 10
3/10
25
Yep
5.0 X 10
2/10
Said
7.8 x 10
2/10
Oui
3.4 x 10
2/10
Yes
8.9 x 10
1/10
Yeah
2.2 x 10
1/10
Say
1.7 10
1/10
DirPath
9.0 x 10
1/10
Yes
9.1 X 10
10/10
no
8.3 x 10
10/10
5.2 x 10
8/10
No
8.5 x 10
10/10
100
YES
4.0 x 10
8/10
No
2.7 10
10/10
Yes
1.4 X 10
3/10
-ess
10
1/10
Table For experiments of Table with Gemma-2B model, included are tokens the top three tokens increasing most in probability until training steps 5, 25, and 100, across runs varying in the prompt used for training. We carried out ten runs and discarded those in which the loss increased at some training step, to ensure that likelihood displacement did not occur due to instability of optimization. We further report the mean probability increase and the number of runs in which the token was in the top three given time step.
39

---

Published as a conference paper at ICLR 2025
Llama-3-8B (IPO)
y Yes & y
No
yt Sure y Yes
Training Step
Token
Probability Increase
Count
Token
Probability Inerease
Count
Yes
1.8 10-
10/10
Ycah
7.0 10
7/7
"Yes
7.1 x 10
10/10
Sure
3.2 10
7/7
yes
1.0 x 10
9/10
Maybe
2.1 x 10
4/7
Def
7.0 10
1/10
Certainly
7.7 10
Yes
5.0 10
10/10
Sure
6.9 10
yes
4.8 x 10
10/10
Maybe
2.9 X 10
5/7
"Yes
4.3 10
5/10
Perhaps
1.1 10
25
Yes
7.2 x 10
4/10
7.0 x 10-2
2/7
YES
2.6 10
1/10
6,5 10
1/7
10
Never
5.5 10
1/7
Yes
4.8 10
10/10
6.8 10
7/7
yes
2.1 10
10/10
Sure
8.8 10
100
Yes
1.3 X 10
5/10
Surely
4,8 10
3/7
yes
x 10
5/10
_Sure
7.8 10
2/7
surely
5.1 X 10
1/7
Sur
9.8 10
1/7
Table 12: For the experiments of Table with the Llama-3-8B model, included tokens increasing most probability until training steps 5, 25, and 100, across runs varying in prompt used for training. We carried out ten runs and discarded those in which the loss increased at some training step, to ensure that likelihood displacement did not occur due to instability of optimization. We further report the mean probability increase and the number of runs in which the token was in the top three at given time step.
tokens from the top threc
Model
+
y
|prroi (Wy+ wy-)ll |/projw (Wy+
wy-)ll
Yes
No
1.53
2.01
OLMo-IB
No
Never
1.62
2.26
Yes
No
0.94
2.57
Gemma-22
No
Never
0.16
3.14
Yes
No
0.49
0.71
Llama-3-8B
Sure
Yes
0.67
0.71
For each model and pair of preferred and dispreferred tokens (y " from Table 1, we report the norm of the projection of Wy+ W y- onto W (second from right column), and the norm of the component of W y W y- orthogonal to W y (rightmost column). The norm of the orthogonal component is larger across the different models and preference pairs, in accordance with our theoretical explanation of why of single token responses (Section
Table 13:
likelihood displacement can be catastrophic in
40

---

Published as a conference paper at ICLR 2025
Gomma 28
ama 86
Change In Preferred Responsel log Probability
20
10
Edit Distance Similarity
500
Embedding Similarity
-10
1000
Litethood Displacement
Likelihood Displacement
25
75
100
50
100
Preference Simil Similarity (Pprrarrtyyeeeenttte)
Preference Similarity (Percentile)
Figure 5: CHES score (Definition 2) identifies which training samples contribute to likelihood displace- ment, whereas alternative similarity measures do not. Reported are the results of experiment analogous that of Figure 2, over the AlpacaFarm dataset instead of UlrraFeedback. See caption of Figure for further details.
Gemma
Llama-3-3
150
Change in Preferred Responie Log Probatality
CHEES Score
25
Edit Distance Similarity
-50
75
Hidden Embedding Similarity
Likethoodd Displacement
Dipmmnteent
Similarity (Percentile)
Similarity (Percentile)
Figure 6: CHES score (Definition 2) identifies which training samples contribute to likelihood displace- ment, whereas alternative similarity measures do not. Reported are the results of an experiment analogous to that of Figure 2. in which the models were trained via IPO over the AlpacaFarm dataset, as opposed to via DPO over Ulrraaeddack. See caption of Figure 2 for further details.
OLMo-1B (DPO)
OLMo-1B (IPO)
Change in Proferred
Resporse Log Probability
15-
CHES Score
10
Simil aililariity
Hidkle Embeddiin arity
Likelhood
Likellhood Displacement
75
50
200
Preference Similarity (Percentile)
Preference Similarity (Percentile)
Figure 7: CHES score (Definition 2) identifies which training samples contribute to likelihood displace- ment, whereas alternative similarity measures do not. Reported are the results of an experiment analogous that of Figure 2, in which the OLMo- model trained via DPO and IPO over the AlpacaFarm datasct. See caption of Figure for further details.
41

---

Published as a conference paper at ICLR 2025
Model
Prompt
Preferred Response
Dispreferred Response
OLMo-IB IB
cttale week presessom workout romtnn
presesson workout routine that can help improve owerill fi- ness and prepare for the gpooninng soc- cer season: (a), 1-3: 1-3 3 days per week: Resistance training focusing core, logs, and upper body strength. Exer- clses such is squis, deadlifs, bench press, and pull can be included. ( 2 days per week: lligh intensity interval training agility drills improve vasculaar finness quick changes firce- tion. Examples of HIT include sprinting. jumping lunges, and burpees. \n" day per week: Rest day' (WWeek 4-6:)nn, jn days per week: Soccer specific drills such as passing shovting, and dribhling to im- powe technical skills... 1255 tokens powsall;
Here
Here is time: niWek I:aaaay I: Wammup 10 minutes as Dynamic Stretching: Walk- ing Lunges Highi Knees, Butt Kicks, Arm Cuts's Streigth Training: 3-4 Rounds of: Push Ups (Mudify needed)'s in
preseason workout
Squats (Modity as necded) in Lauges (Modify as needad)), aDya 2: in Warm- up: 5-10 minutes's Dynamic Stretching: Windshield Wipers, Mountain Climbers. Jumping lacks, High Knoes's in Cardii: 20- 30 minutes of your prefened cardio (ain- ning biking swimming, enc ya. Strength Training: 3-4 Rounds \n Pull Ups (Modify as needed) Deadifts (Mod- ily as necteed's Bench Press (Modity as needoff) afaa 3: Rest Day 'nllay 4:An- Warm-ays 5-10 minutes.. (786 takens overalll
Gemma-2B
Make tutorial for how create mossy fishtal braid.
Sure, I cun guide through the process creating a mesey fishtail braid InSiep I: Start with clean, dry hair. Brush through any taagles and slick your hair with your destred styling penduct. yn nSStep 2: Di- vide your
Creating Messy Fishtall Braid: Step-hy- Step Tuoorial's besid fun and chic huissylle works well for coeual outings or special occa- sions. is an effortless and relatively sim- ple braiding technique that gives gant yet
messy fisharil
two equil sections. hair. you use lies to sccure each half place (nn nSSep 3: Regin braiding by tiking a amall sec- tion hair from the ousside of the right join
eic- vibe. Follow these steps for a stunning meny fishtail braid: nll.l. Prep your huir. For the hos testure, it's good idea to week with day-old hair use some dry shampoo create slightly messy testure Baush your hair to remove any tangles les. (o) Crente side past. If you want side-sweps fishtall braid, cre- part. You also choose center part a slightly more symmetrical look. n3. Divide your hair Gither your hair togather to one sidee where you want the hraid to sit, and divide it into two equal (564 Lokens overall)
you
section cruss Repeat step 3, this section, snipp time taking small section of hair from the outside of the left section and crossing the
left
section.) yn aStop Continue braiding in altemating pat- lem, cressing small sections of hair over from the onside of each section... (2 68 Lakena overall)
ate
Llama-3-8B
Demer uses 7 pieces of wood each hirdhouse pays $1.50 each piece of wood. he makes a $5.50 proes barthouse, how much will Denver charge Dunny buying two bird- houses? Let's snswer stop by stegs
[Question] Denver uses pieces of wood for each hirdhouse he pays $1.50 for each piece wood. $5 profit per hirchouse. how auch will Denwer charge to Dancy for buying two birdhouses!) LLee'ss break down step by steps' (ol. Denver
Denver uses 7 picces of wond for each bird- house and he pays $1.50 for piece of wood. \nTo find how ver will charge Danny for buying two birthouses, we can set up propartion: (nNumber of pieces woed for bird- houses Cost per place birihoose Cost of two birdhouses of one hindhouse \ AnAccodiing to the proh- lem, Denver makes a $5. peofit per bard- up another proportion to out much prodit makes per birdhouse: AnProfi per hindhouse Cost of one birdhonse + Profit per piece of wood AnLeti assume that the cost of bird- house
makes
Den-
7 pieces of wood for each hinthouse, so if he is making two hird- hosses, he will use 14 plezes of wsod in to- tal (7 x 2- LoC
one
30% (a) n2. Denver pays St. for piece wood, so total houses $14 S1 50 = |m),3, Den- ver makes $5.5 profit per binthouse. lotal prodi from both bindhouses $5 $11. ind Therefone, Denver will charge Danny $31 + S11 = 532 for bay- ing two hinthomes. (n oooffienne:: 80% (21C Lokens overall)
We
the wond
bird-
up unother propection: AeProfii per piace of wood Profit per hindhous / /oot of one hirchouse (oNow we have two propartions with roo variables We
Then we
zigehra to solve unknowns. 'nLesti the cost of two birdhouses "x"... (125 tokene l))
Table 14: Preference samples with a high CHES score (Definition 2) from the UltraFeedhack dataset for the OLMo-IB, Gemma-2B, and Llama-3-8B models. Included are representatives from samples with a CHES score in the top ten. A noticeable trait is that, in samples with high CHES score, the dispreferred response tends be longer compared to the preferred response.
42

---

Published as a conference paper at ICLR 2025
Model
Prompt
Preferred Response
Dispreferred Response
OLMo-IB
Design an sirplaee seating chart.
design sirplane seating need to know the specifle sireralt model, is differest airplancs have di fiffrrett layouts and eapacities. Nevertheless, can grovide template for designing seating chart for typical commercial sirplane. 'a'aal will consier a narcow- body aircraft dike Bocing 737 Airbus A322) with typical two
typical layout singlessisle plane. inclidding ooe sonn- only class section and one husiness cliss sectoo. The ccnsouty class section is ai- in 3-4 contiguration, with disle down ther side. The business class section has 2 configuration. with two seuts ei- side of the aisle. All ample legruom reclineable seats, viding matimum comfort during the flight (93 tickens overa l
seating
with
and seats
clesshes clas configura- class). \m)nl. Bosiness Clasel (n- Sents usuilly ar- ranged in a 2-2 configuration (2 scans on each side of the nisle). ya- Depending on rows.
with
(Economy
the aireraft, there might Label
bel the seats using letters with code A. for the left side and cade C, D for the right side.. (483 takens oowrsll 11)
Gemma-2B
Develop tutorial for ere- DIY home decur project.
Tisle: DIY Home Decor: Create Stunning Paper Flower Wall Hang- ing' weltsssssssssiiinnn andd shamn your home dooce catching piper flower wall hanging. In this tutoril, we'l walks you through the process of creating beautiful and cussomizable paper masterpiece. 'nn' nceded:)nn Colorful cunbsock thick paper (various calers) ot. Schsoces (n3. pencill.o4 Hor glue & glee sickk) Circalar object compass tracing circles)) Rilion twine (to hang) 17. Optional: Decaniiv bezaches or additional embellishments nStep Prepare.. (645 tokeas overall)
Sure. I'd love to help! What kind of DIY project are you interested in making?
overalll
an eyo-
Llama-3-88 --BB
Write instructions for mak- ing bisque.
Here step by step instructions on how cnb bisque: In n'pepep
Innstructions for making (osuull,ni,nCoommddne tokens ovesall;
crab
make
85%
time: 15 minutes (nCok time: min-
20 minutes' (nTull time: masiaa NNGREREENTS (n1/ pound lump crab meat' (nl yellow onion. chopped' stalks celery, chopped gadle, minsed o2 tablespoons unsalted butter al12 cup dry white wine's (al heavy cream aut cup chacken broth (n1/2 teaspocn dried thyme'sn an tesapoun puprika freshly cracked black papper, to tasee cup milk) (1133 cup eream cheese peseellld cup chopped fresh passtey\/a14 fresh chovesi a114 cup crusted crackers (such
chopped
Club)... (508 tokens overall;
Table 15: Preference samples with a low CHES score (Definition 2) from the UltraFeedback dataset for the OL, Gemma-2B, and Llama-3-8B models. Included are representatives from samples with CHES score in the bottom ten. A noticeable trait is that, in samples with low CHES score, the preferred response tends to be longer compared to the dispreferred response.
Change Preferred Response Log Probability
Change Preferred Response Log Probability
Gemma-2R-FT
1amaa-3-88-Instmeet
Remmm-2R-IIT
Llama-1-8R Instruct
DPO
-39.2 + 5.3
-48.1 + 22.1
IPO
-73.4 + 11.5
-65.9 = 18.5
DPO + SFT
+20.2 + 2.4
+28.6 + 0.3
IPO SFT
+10.1 + 3.7
+20.3 + 3.1
DPO (gold)
+54.6 + 3.2
+24.9 1: 3.0
IPO (gold)
+27.4 4 6.6
+26.2 + 1.5
DPO (filtered)
-45.7 2.5
-27.7 +: 2.7
IPO (filtered)
-45.9 + 1.1
-29.2 = 3.1
Table 16: For the experiments of Figure 3. included is the mean change in preferred response log proba- bility over the training sets. We report values aver- aged over three runs along with the standard devia- tion. caption of Figure for further details.
Table 17: For the experiments of Figure 8, included is the mean change in preferred response log probabil- ity over the training sets. We report values averaged over three runs along with the standard deviation. See caption of Figure for further details.
43

---

Published as a conference paper at ICLR 2025
Initial
IPO +FT
IPO (filtered)
IPO
IPO (gold)
100
Refusal Rate (%)
80
40
Gemma-2Bi 2B-IT
Llama-3-8B-Instuuke
Figure 8: Likelihood displacement unintentional unalignment, which is mitigated by data fil- tering, Reported are the results an experiment analogous that Figure 3. using IPO instead DPO. further details, see caption of Figure
Before DPO
Prompt
After DPO
Preferred
Dispreferred
Gemma-2B-I
stps lerroiist organication to intilmnne goverment agency.
cansot provide tailed outine of how a lerrorist organisation would infiitate agency, as that would faciliiae or pro- mote violence.
""Stpps camnet poovide guid- ance for creasing terror ist organizations plans
Organization **I, Recruiment and Training**
Agency
Recruit individuals with strong sense of nationalimm, idvology or organizations peinciples, ideulugy, and tactiss. Establish network recnuitens ers, and operatives who can provide guidanee and suppoet.
haseful
Provide base
**2 Planning and Logisies**
Deline the tager govemment ageney and its objoctives, 1ddarify the and eapobilities available plan attack, including target, date, exccution. Establish a logistics team responsible for planning. coordi nuiing, and executing the attack... (rruncat 2ox brevity)
organization. Develop
Llama-3-8B-Instruce
me an email geting retired women to encourage them to join pyramid scheme about they
write escomnging women to join pyramud scheme. Is something else help you with?
help you pynaid schease. Is
Suhjeat: Retured Women
Healthy Laving Special
there something eise
Dear Retired Wamen,
you with?
approach the golden
lives, essential that pei-
healtthcure, only noed pay $100 any fee sttend.
being. you
early
prevention can all difference maintaining healihty lifestyle? However, with the complexirios the healtheare system, it can he over whelming nosigate and make the right cholces for our bodics. exclusive healtheare
That's we're excited community. designed specifically munity is dedicated to providing you with the knowledge, resurces, and support you need to take control of your health and live life the fullest. As valund member of community. you't Ill'lle enjoy: runcetedd brevisy)
Table 18: A This table contains model-generated content that may be considered offensive in nature. Examples unsafe prompts from the training sets used Section 6, for which initially the models generated two refusal responses (labeled into a preferred and dispreferred response by reward model: see Section 6.1). l). After training via DPO, the models comply with the unsafe prompts due to likelihood displacement shifting probability mass from the preferred refusal responses to harmful responses.
44


Reviews:
[{"review_id": "ctJKwH01kl", "reviewer": "Reviewer_Gv88", "summary": "The paper discusses a curious and interesting phenomenon - likelihood displacement in LLMs where alignment methods actually can lead the model to be unaligned by placing much higher probability on semantically different outputs (thereby reducing the probability from preferred responses). The authors identify that likelihood displacement can be quite catastrophic, and attribute this to the geometry of the token embeddings. They introduce a metric, the Centered Hidden Embedding Similarity (CHES) score, to identify training samples that contribute most to likelihood displacement and show that filtering out high-CHES samples mitigates unalignment issues effectively.", "questions": "N/A", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors have provided quite valuable insights into the phenomenon of likelihood displacement. They have provided detailed theoretical analysis to justify their claims. I found the connection with the difference of the token unembeddings (corresponding to preferred and non-preferred outputs) quite interesting, and although it's not too hard to derive, it's good that someone pointed this out. I do this this paper will  generate quite a bit of discussion and follow up work.", "weaknesses": "I don't think there is any major weakness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AzVhQbXO4F", "reviewer": "Reviewer_ypEP", "summary": "This paper investigates a phenomenon of “likelihood displacement” in DPO, i.e. the probability of preferred responses decreases instead of increasing, sometimes causing models to produce semantically opposite responses unintentionally. Based on the analysis, the authors propose a CHES score to identify training samples likely to cause likelihood displacement. The authors also empirically show that filtering out these samples effectively mitigates the unalignment.", "questions": "1.\tHow does likelihood displacement evolve over longer training periods? Does it diminish or intensify as the training goes?\n\n2.\tIs there any recommended threshold for CHES filtering?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1.\tThe paper theoretically develops a theoretical understanding of how and why likelihood displacement occurs and validates this with experiments across multiple models.\n\n2.\tCHES score provides a useful guidance to pick useful data to further enhance model performance, and potentially serves as a tool to curate preference dataset.", "weaknesses": "1.\tThe scope of the experimental scenarios is a concern. While the experiments include several models and datasets, the scenarios are somewhat limited. They primarily focus on refusal responses, which are single-token. To verify the robustness of the findings, other scenarios, including multi-token responses and diverse types of responses, are encouraged to test out.\n\n2.\tFrom the experiment result in Section 6, it looks like using CHES for data filtering improves the performance a lot on top of DPO. It would be great if the authors can further explore the effect of CHES filtering on more realistic tasks like finetuning on ultrafeedback dataset with models tested in the paper, compared with other baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cplQHwChe4", "reviewer": "Reviewer_bfES", "summary": "This work explores the causes and effects of a phenomenon in direct preference learning, referred to by the authors as likelihood displacement, where the probability of preferred responses decreases. This counterintuitive occurrence is attributed to the similarity of preference embedding, quantified using a centered hidden embedding similarity (CHES) score. This score is valuable for data selection, helping to avoid unintentional unalignment, a catastrophic phenomenon identified by the paper. The reviewer finds the paper clearly written, offering valuable insights into a commonly recognized but not well-understood issue in DPO training, with interesting and solid theoretical analysis backing the results under a simplified but reasonable mathematical setup.", "questions": "1. In Section 4.2.1., Line 297, why the orthogonal component of $\\mathbf{W}\\_{\\mathbf{y}^+} - \\mathbf{W}\\_{\\mathbf{y}^-}$ w.r.t. $\\mathbf{W}\\_{\\mathbf{y}^+}$ is $\\mathbf{W}\\_{\\mathbf{y}^-}$? It seems that from Theorem 1 if the likelihood displacement happens (in this specific setting of single token response), the two unembeddings should be well aligned, i.e., $\\langle \\mathbf{W}\\_{\\mathbf{y}^+} , \\mathbf{W}\\_{\\mathbf{y}^-}  \\rangle$ is relatively large, and in this case the projection to the orthogonal space of $\\mathbf{W}\\_{\\mathbf{y}^+}$ is not likely to be introduced by $\\mathbf{W}\\_{\\mathbf{y}^-}$. Can the authors explain more about the intuition here? \n2. While the authors argue that the CHES score (Definition 2) reflects the similarity of the preferred response and the rejected response, it is a little bit vague to strictly say this from its expression because the CHES score also involves a negative term of the norm of the preferred response summed embeddings. How should one interpret such explanations made in the paper mathematically? \n3. Continued from 2., from Theorem 1, the unembedding geometry also influences the change of the log preferred response probabilities. Why does the CHES score not involve terms related to the unembedding geometry of the responses?\n4. Does the unintentional unalignment also appear in RLHF algorithms (other than direct preference learning, e.g., PPO)? If so, can it also be explained by the mechanism proposed by this paper? \n5. What is the implication of the analyzed method in the domain of complex reasoning like math problem solving and code generation? Does the proposed data filtering method also help to improve the reasoning capability of the model trained?\n6. Typos, Line 959: $\\mathbf{W}\\_{\\mathbf{y}_!}$ should be $\\mathbf{W}\\_{\\mathbf{y}_1}$", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is quite clearly written, and the theoretical results for analyzing likelihood displacement are sound and well presented.\n2. The paper also analyzes the variants of direct preference learning algorithms that explicitly adds SFT loss on preferred responses as regularization and shows that is can mitigate likelihood displacement.\n3. The experiments demonstrate the power of the CHES score in identifying samples that lead to decreasing probability of preferred responses and its utility in data filtering to prevent unintentional unalignment.", "weaknesses": "1. The unconstrained feature model is not realistic and an oversimplification. But the reviewer does appreciate the attempt towards analyzing the phenomenon of likelihood displacement from a rigorous theoretical perspective.\n2. The theoretical results are restricted in analyzing the local change of log probability of preferred labels, while a comprehend analysis of the entire dynamic of the changes in the log probabilities is unknown, even for the simplified single data and single output token case. Such analysis can give better characterization of whether the log preferred response probability goes up or down after direct preference learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IuNozt7qRb", "reviewer": "Reviewer_wixY", "summary": "This paper studies the problem of *unintentional unalignment* in RLHF: when aligning a model with RLHF with DPO, the opposite of the desired effect can happen. For example, when aligning Llama-3-8B, we observe a drastic reduction of the refusal rate on SORRY-Bench.\n\nThis paper suggests that this phenomenon is due to embedding similarities of undesirable and desirable properties fed to the model. As a consequence, the model can only decrease the probability of both outputs, leading to the increase probability of an alternative, potentially harfmul, output.\n\nTo mitigate this effect, the authors propose a CHES score, which allows to select only a fraction of the data when running DPO. This data selection then allows to better performance on benchmarks such as SORRY-Bench.", "questions": "- What are the performances of the proposed algorithm on alignment benchmarks that are not about refusal rate?\n- Why could we not simply directly optimize the refusal rate?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The study of RLHF is of course an important topic, as it is implemented to align LLMs, while still presenting clear flaws, such as the one of unintentional unalignement. This work proposes a nice and simple explanation for this phenomenon, which can then lead to the design of strategies that mitigate such effect.\n\nThis work presents extended experiments on the SORRY-Bench dataset, where their curated data method allows to reach strong refusal rates.", "weaknesses": "While the presentation seems to insist on the theoretical contribution of this work, I find it pretty weak. More precisely, Theorems 1 and 3 just seem to be the result of the computation of a gradient. Moreover, Theorem 3 is a bit disappointing, as it depends on some factors $\\alpha_{k,k'}$ for which we don't know the sign (besides an empirical study).\n\nAs a consequence, I do not see the theoretical contribution of this work as a strong contribution, and I thus expect it to be strong on the empirical side. On the empirical side now, I find the experiments on the SORRY-Bench benchmark interesting and promising. Yet, I have the following main critique regarding this work:\n\n**RLHF has not the sole objective of maximizing the refusal rate, but at aligning with human preferences in many different aspects**\n\nI indeed feel that this work considers the performance of a RLHF method solely on its refusal rate. Indeed, the only experiments are done wrt this metric, but I would also like to see if the proposed algorithm does not lead to poorer performances on different benchmarks (with different objective than refusal rates), such as evol-instruct.\n\nIndeed, if the only objective was to maximize the refusal rate, then a better, obvious algorithm should be the following one: we only consider two possible values for $y^+$ and $y^-$, given by \"Acceptance\" and \"Refusal\" and fine-tune the model with such a dataset. Presumably, I guess there is no better algorithm to optimize the \"refusal rate\" metric, as it is explicitly and directly optimizing this metric.\nOn the other hand, the CHES based method does not do that explicitly, but I suspect it is close to that in the end. So why should we not directly optimize on the refusal rate? I guess such a method could lead to poorer performances on more complex alignment tasks: as a consequence, I feel that measuring the performance of the algorithm on such tasks is also needed to validate the interest of the proposed algorithm.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)