A Common Pitfall of Margin-based Language Model Alignment: Gradient Entanglement
Authors: Hui Yuan
Yifan Zeng
Yue Wu
Huazheng Wang
Mengdi Wang
Liu Leqi
Venue: ICLR 2025 Poster (2025)
Keywords: Alignment
Preference Optimization
Large Language Model
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Reinforcement Learning from Human Feedback (RLHF) has become the predominant approach for aligning language models (LMs) to be more helpful and less harmful. 
At its core, RLHF uses a margin-based loss for preference optimization, which specifies the ideal LM behavior only in terms of the difference between preferred and dispreferred responses. In this paper, we identify a common pitfall of margin-based methods---the under-specification of ideal LM behavior on preferred and dispreferred responses individually, which results in two unintended consequences as the margin increases:
(1) The probability of dispreferred (e.g., unsafe) responses may increase, resulting in potential safety alignment failures.
(2) The probability of preferred responses may decrease, even when those responses are ideal.
We demystify the reasons behind these problematic behaviors: margin-based losses couple the change in the preferred probability with the gradient of the dispreferred one, and vice versa, often preventing the preferred probability from increasing while the dispreferred one decreases, and thus causing a synchronized increase or decrease in both probabilities. We term this effect, inherent in margin-based objectives, gradient entanglement. 
Formally, we derive conditions for general margin-based alignment objectives under which gradient entanglement becomes concerning: the inner product between the gradient of preferred log-probability and the gradient of dispreferred log-probability is large relative to the individual gradient norms. Furthermore, we theoretically investigate why such inner products can be large when aligning language models and empirically validate our findings. Empirical implications of our framework further extend to explaining important differences in the training dynamics of various preference optimization algorithms and suggesting future directions for improvement.

Published as a conference paper at ICLR 2025
A COMMON PITFALL OF MARGIN-BASED LANGUAGE MODEL ALIGNMENT: GRADIENT ENTANGLEMENT
Hui Yuan
Yifan Zeng 2 Yue Wu Huazheng Wang", Mengdi Wang", Liu Leqi
1.3.5 1.3.5 Princeton University
2.4 Oregon State University
Thhe University of Texas at Austin
ABSTRACT
Reinforcement Learning from Human Feedback (RLHF) has become the predom- inant approach for aligning language models (LMs) to be more helpful and less harmful. At its core, RLHF uses margin-based loss for preference optimization, which specifies the ideal LM behavior only in terms of the difference between preferred and dispreferred responses. this paper, we identify common pitfall of margin-based methods- the under-specification of ideal LM behavior on pre- ferred and dispreferred responses individually, which results in two unintended consequences as the margin increases: (1) The probability of dispreferred (e.g.. unsafe) responses may increase, resulting in potential safety alignment failures. (2) The probability of preferred responses may decrease, sponses are ideal. We demystify the reasons behind these problematic behaviors: margin-based losses couple the change in the preferred probability with the gradi- ent of the dispreferred one, and vice versa, often preventing the preferred probabil- ity from increasing while the dispreferred one decreases, and thus causing syn- chronized increase or decrease in both probabilities. We term this effect, inherent margin-based objectives, gradient entanglement. Formally, we derive condi- tions for general margin- based alignment objectives under which gradient entan- glement becomes concerning: the inner product between the gradient of preferred log-probubility and the gradieni of dispreferred log-probubility is large relative to the individual grudient norms. Furthermore, we theoretically investigate why such inner products can be large when aligning language models and empirically validate our findings. Empirical implications of our framework further extend to explaining important differences in the training dynamics of various preference optimization algorithms and suggesting future directions for improvement.
when those re-
INTRODUCTION
Reinforcement Learning from Human Feedback (RLHF) has become a primary approach for align- ing Language Models (LMs) to improve their helpfulness and mitigate harmfulness (Bai al., 2022; Ouyang et al., 2022; Stiennon et al., 2020). This pipeline typically consists of two stages: super- vised fine-tuning (SFT), where demonstration data is used to directly teach the model desirable behaviors, and the reinforcement learning (RI. stage, which uses preference tween different responses to the same prompt-to highlight the contrast between chosen and rejected responses, with the goal of helping the model learn distinctions between good and bad behaviors.
comparisons be-
In its vanilla form, the RL stage first employs a contrastive loss-based on the margin between the scores of the chosen and rejected responses- to train reward model, followed by policy opti- mization methods to fine-tune the LM based on the reward model. Leveraging the structure of the problem, recent line of work has combined these two steps by directly optimizing the language model using a masged preference optimization loss of the following general form (Azar et al.,
Leading Contributors. Corresponding buiyuan@princetoon.ee edu. leqil iuuutteaatsssssaaa edu. Work done in part at Princeton Language & Intelligence. Code for the paper can he found https://githhh.com http:/hhhh..ooouumiiii.ioo ah/Understand MarginPO.

---

Published as a conference paper at ICLR 2025
2024: Ethayarajh et al., 2024; Hong et al., 2024; Meng et al., 2024; Pal et al., 2024; Park al., 2024; Rafailov et al., 2024; Wu et al., 2024; Xu et al., 2024; Yuan et al., 2024; Zhao et al., 2023):²
{(x, y Mr; 0) m(h (log To (Bw (z) (log (b(()))))
(1)
where for language model ny. log To (Yu |z) specifies the log-probability of the chosen response yo and log (y (2) specifies that of the rejected response yı" given the same prompt Most of existing preference optimization losses can be interpreted as varying the scalar functions m, Itw hy (Section 3.2 and Table 2). At the core, they all rely on the margin between the chosen agg-robabblity log The (Be (r) and the rejected log-probatle probability log ia ()).
The training dynamics of these margin-basce preference optimization are quite intriguing the log- probabilities of the chosen and rejected responses often show a synchronized increase and decrease (Figure 1). is worth noting that, by the end of the training, even though the margin increases (resulting in minimization of the margin-based based loss), the log probability of both the chosen and rejected responses may increase (Figure la), or both may decrease (Figure lb).
-125
WWW
-105
-130
logP
-110
rejected
-140
-145
-120
1100
2,000
3.000
2,000
3,000
4,000
5,000
Training Step
Training Step
(a) Mistral 7B
(b) Llama-3 8B
Figure 1: Training dynamics of the chosen and rejected log probabilities on the TL;DR dataset (Sti- ennon et al., 2020) during DPO, with log probabilities averaged on the evaluation set. As the margin between the two increases, the chosen and rejected log-probabilities exhibit synchronized increases and decreases per step. In Figure la, both chosen and rejected log- prgoobbabilities have an overall trend of increasing, especially towards the end of training, whereas in Figure 1b, both have trend of decreasing. Similar trends are observed across multiple models and datasets, see appendix F).
This synchronized log-probability change exposes fundamental issue with using margin- based loss for preference optimization in language model alignment: it only specifies the ideal behavior of margin between chosen and rejected probbbbbbllissss but not the ideal behavior of individual terms. This under-speciffcation may have two problematic consequences*:
First, when the primary goal is reduce the probability of generating rejected responses (e.g., in safety-related alignment tasks where certain undesirable responses should not be generated), merely increasing the margin (i.e., ensuring that the chosen response is preferred over the rejected one) does not guarantee that the log-probability of the rejected response is actually decreasing (Figure la).
Second, even when the log- log-probability of the rejected response does decrease, the current margin- based losses often lead to a simultaneous reduction the log-probability of chosen response (Figure 1b). This becomes particularly concerning in some of the current fine-tuning practices where we want to retain or even increase the probability of generating the preferred responses. For example, for distilling strong language models into smaller ones (Chiang et al., 2023; Dubey et al., 2024; Taori et al., 2023; Tunstall et al., 2024), a common practice is to synthesize chosen samples with those strong models; in some aligmment applications (e.g., math proviem-solving and coding), chosen samples can be the human demonstrations collected during the SFT phase (Chen et al., 2024). In both scenarios, the chosen responses are ideal and we want to ensure the model retains a high probability of generating these ideal responses.
"The reward modeling loss in vanilla RLHF is also example of this general form. SSusscript w chosen response yo stands for "winner", in y stands for "loser." 'In some cases, the ideal LM behavior chosen and rejected samples can he unclear, e.g., in the original RL.F procedure (Stiennon et al., 2020), the chosen and rejected pairs drawn from models still in training.
2

---

Published as a conference paper at ICLR 2025
Our study is motivated by the previous two scenarios where, ideally, the LM's probabilities cho- sen samples should increase and that on rejected samples should decrease during alignment. How- ever, most margin- based methods fail to induce the ideal behavior (Figure 1, Figure 2), highlighting the need for understanding this common pitfall.
Throughout the paper, refer log (2) the chosen log-probability and its gradient, Va log ae (Bllaar)) as the chosen gradient; similar definitions apply for the rejected case. In this work, we demystify the reasons why log (ym (r) and log ( (a)) exhibit synchronized increase or decrease during alignment. We uncover that the underlying cause is the gradient entanglement effect inherent in margin-barn based objectives: probability with the gradient of the rejected one, and vice versa, preventing the chosen and rejected probabilities from changing independently.
saarin-basede losses couple the change in the chosen log-
Formally, we characterize gradient entanglement happens because the change in the chosen and rejected probability depends on the inner product (Vo log (B (z)), : Vo log (b((a=))) between the chosen and rejected gradients. This entanglement will result in synchronized changes in the chosen and rejected log-probability when the inner product is "large" relative to their individual norms, which we name by "gradient condition" (Section 3.1). Moreover, the precise definitions of "large" for different margin-base algorithms are captured by a general version of the gradient condition (Section 3.2). The gradient conditions derived enable to characterize existing margin -ased preference optimization methods, explain their differing training dynamics, and identify the most suitable scenarios for deploying these algorithms. Our theoretical findings also validated through empirical observations (Section 3.3).
We further investigate why the gradient inner product can be relative large to individual norms when aligning model using language data. In synthetic settings, we theoretically show that (1) as the cho- sen and rejected responses share more similar tokens, their gradient inner product will increase, and (2) while the sentence-level gradient inner product may be large and positive, individual token-level inner products can be small and negative (Section 1.1.1. 4.1. We validate these theoretical insights empirically (Section 4.2). In Section 5, we discuss the empirical implications of our framework for language model alignment.
To summarize, our contributions are as follows:
We identify a fundamental issue with margin-based alignment: it under-specifies the ideal behav- ior of the LM on chosen and rejected responses individually (Section 1);
We uncover that gradient entanglement the inherent cause the pitfalls in margin- ohjec- tives, and provide general gradient inner product condition that captures when the synchronized movement of chosen and rejected log probabilities occurs (Section 3);
We investigate the gradient inner product and explore when the condition may fail and the syn- chronized movement occurs theoretically and experimentally (Section 4).
2
BACKGROUND AND RELATED WORK
2.1
PROBLEM SETUP
We consider auto-regressive language models (y'(x; [z. (² that specify the distribution of the next token y at index f on a finite vocabulary set V/ given the prefix tokens including the prompt and the partially generated responses yt In the context of LM alignment, there is a reference policy wiefs usually obtained by large-scale pre-training and supervised fine-tuning, and serves the sampling policy and start point of further alignment algorithms.
2.2 PREFERENCE OPTIMIZATION
There have been plenty of works on the design of preference optimization losses, motivated by various assumptions or considerations. Here we briefly review them and discuss their connection to the probability margin:
Rafailov et al. (2024) derive the DPO loss from the KL-constrained reward maximization problem:
max Bww....g....aalllllrr.. XEL=xxKL(TTT((HH
3

---

Published as a conference paper at ICLR 2025
They further derive the DPO loss for any triplet (x,, Mwo Mi) rejected response, respectively:
where the Yues y0 the chosen and
TO (T Trer(Buol
To To (ar) Tree (31 (z)
EDPO (xr Mhes yr; 02 Tret) log o log
log
Following the margin-based objective DPO, advancements have been proposed by IPO (Azar et al., 2024), SlicHF (Zhao et al., 2023), R-DPO (Park et al., 2024), SimPO (Meng et al., 2024), RRHF (Yuan et al., 2024), KTO (Ethayarajh et al., 2024), SPPO (Wu et al., 2024), CPO (Xu et al., 2024) and DPOP (Pal et al., 2024). Pal et al. (2024) is the most relevant work to ours, which touches upon a similar failure mode, but focuses only on the decreasing probability of chosen in DPO. In contrast, we dig deeper obtain a broader view on the synchronized change (inerease or decrease) for a range of margin-based methods, by rigorously analysis and extracting a general success/failure condition. detailed review of these works in Appendix B.
GRADIENT ENTANGLEMENT
Margin-based preference optimization often results in synchronized increase/decrease in chosen and rejected log-probalie pr-prbbabillitis (Section 1). Our key finding is that the synchronized change is caused by an effect term gradient entanglement. Starting with case study on DPO in Section 3.1, we formally define the gradient entanglement effect, from the definition we will see the entanglement passed through the inner product between chosen and rejected gradients. We derive conditions on such inner product under which the gradient entanglement causes concerning synchronized change. In Section 3.2, we identify gradient entanglement for general margin- baaee preference optimization methods and apply our framework to explain the training dynamics of those methods. We validate our findings empirically in Section 3.3.
3.1
CASE STUDY: GRADIENT ENTANGLEMENT IN DPO
Let us start with deriving the gradient of the DPO objective (2). To simplify the formula of DPO gradient, define the implicit reward fn( v) log (o(p)r) the notations:
Wad (3)a) (which is scalar) and introduce
log (0) == log (B)l log Ti (8) == log ( (r), c(0) == (Ne (z. v)) - fo (I, (w)) > 0.
Then considering single sample (xt, Bws 31), the DPO gradient can be rewritten as$
VolDPO -Bc(( (V log T. (0) V log (f)).
Suppose 17 >> 0 is the step size for minimizing the DPO objective and let C' = 3),c((). After one step gradient descent with (3), simple analysis of the log-probability change in chosen and rejected responses uncovers the intriguing gradient entanglement effect as follows:
Gradient Entanglement (DPO)
The chosen log- log-probability change A log vw depends on the rejected gradient V log Ti and similarly, the rejected log- bog-probability change A log क. depends on the chosen gradient
V log Tw:
A log T., C. (IIV log u. 112 (V log Tip V log m))
(4)
A log ग. R3 C ((V log Tw, V log मt) - V V n. 11
(5)
(4) and (5) are derived by upproximating log ww and A log ग. with first-order Taylor expansion (Appendix C.I). Beyond the DPO objective, the gradient entanglement effect is an inherent char- acteristic margin-hased objectives as the chosen and rejected log- prgoobbbaylityy are coupled in the definition of "margin." In Section 3.2, we will formally derive gradient entanglement for general margin-based objectives for preference optimization. In the sequel, we will derive conditions on (V log Tw, V log m)) under which the gradient entanglement will have concerning effects.
'When the context is clear, we omit and just log Tw, log क. and V.
4

---

Published as a conference paper at ICLR 2025
3.1.1 WHEN WILL THE GRADIENT ENTANGLEMENT CONCERNING?
measure the change in the margin between log Tu and log mi i.e. the quantitiy A Allog Tw log mi), then the Cauchy- -aahhy- inequality ensures:
(log ue log m)) = C (I log mell 2(V log Tw; loog m.) + | log क (12) 0,
which fulfills the contrastive goal of the DPO loss: enlarging the difference between the chosen log- probability log ww and rejected (og-probability log क.. However, due to the gradient entanglement effect, to individually ensure the increment of log Tw and the decrement of log Tı, the inner product between chosen and rejected gradient should satisfy the following condition, which will refer to as "gradient condition".
Condition 1 (Gradient condition for DPO). In DPO, to increase log Tw and decrease log कं indi- vidually, (4) and (5) imply the following conditions:
(V logTw: Tw: Vlog mi) & V log wwll?
A log Te 2 0, log Te increases;
(V To V log m)) - V log mi|l
A log S 0, log decreases.
Based on the two conditions above, in Table 1 we summarize three cases that depict all pos- sible changes on the chosen and rejected bog-prbaabilities and are categorized by the value of (V log Tw, V log mi).
Case
A log Tw. A log m.
log Twu log m
Condition
A log Rw 20 0 A log
log to f log 4
(V log Tae, log S min(]/V log at V/ log |/ log mu I1 & (V log Twn log Ri) S log myll
2 log Rio 2 log m
log ww J. log mi 4
3
A log mw 2 A log 2
log mw 1 log mi
|| log mell s (V log Rw.. V log m.) S V log m ||
Table 1: Three possible cases of the changes on chosen and rejected log- prgbbbbbbiilities in DPO. and indicate increase and decrease. Case 1 (Ideal): log Tw increases and log decreases; Case 2: log Tw and log कy both decreases but log क, decreases more; Case 3: log Twe and log ग both increases but log Tar increases more.
3.2 GENERAL GRADIENT ENTANGLEMENT EFFECT
We now move the general margin-based loss (l). Here, additionally consider regularizers used in these losses:
(()) (1 (log mw) - (log m))) + A Allog mw)g..))..
where A(log Ta (yw (r) is scalar regularizer depending on the chosen log probrrbbbbblit.... We instan- tiate popular preference optimization methods from this general form in Table 2, where we denote Cet == log Teer a3 [z), det == log (3r (r), Get == Get Gef Terms that only depend on (B)()) shall be viewed constant, independent of A.
Based on this unified formulation of preference optimization objectives (6), we derive general gra- dient entanglement for all margin-basee based losses (derivations Appendix C.1):
Gradient Entanglement (General)
The chosen probaabbayyltyy change depends the rejected gradient, and vice versa. The mutual dependency is characterized by:
log Tu RS 7 (due | log TTll - de(V log Tw. V log (i)) A log R (dw (V log Tw. V log m)) dil log क. 112)
In the general form of gradient entanglement, dup and d, are sealars defined as de := (hw (log ==) hy (log m)) Mr, (log nw) + A (logmw); Tw); d: m (Io.. (log m.) far (log m: ( (logm))
(7)
(8)
We derive generalized version of Condition for general margin-based losses.

---

Published as a conference paper at ICLR 2025
Condition 2 (Gradient condition for general margin-based objectives). For margin- based preference optimization objectives(6). the conditions for log Tw to increase and for log कą to decrease are:
dw
V log Ta V log m.)
V log
Twe
A log Tw 2 0, log Tu increases;
(9)
di
dp
(V log Tw, V log mi) S
| log T(
log कं S 0, log xn decreases.
(10)
dw
Accordingly, we can instantiate Condition 2 for different algorithms by using their specialized m, m,hw hú, A in Table 2.
mz(a)
(w)
(a)
A(a)
DPO (Rafailov al.)
log Get)
Ba
R DPO (Park et al.)
lug (a (Gd + aa(v/=l la )))
Ba TRI
SimPO (Mong et al)
log (a 7) 7)
War a
IPO (Azar al.)
(a
+ 1)²
RRHP (Yunn
min(0,
Aa
SlicHt (Zhao et al.)
min(0, 6)
CPO (Xu et al.)
log (a)
Ba
Ba Ba (iar dy aud a)
Aa
DPOP
log Get)
max log oa (Ba (log cut aut) 1-
KTO (Bihayarigh
SPPO (Wu et al.)
(a
(a
Table Instantiation of margin-based preference optimization losses. The constants in these losses
satisfy B, 7, 6, Am, À >> O.
3.2. How DO OTHER MARGIN-BASED METHODS WORK DIFFERENTLY FROM DPO?
Utilizing the gradient condition we derived, we provide in the following a brief discussion on some existing preference optimization algorithms and explain why these algorithms may work differently from DPO under certain settings.
DPO: du 4. 1, reproducing the Condition in DPO setting.
d. du
SPPO: du 3-1 Rw. 16, where g-1 a large constant. Compared with DPO, SPPO loss d, 2-1++og nı
ensures that it is easier for log ww to increase based on (9) and harder for log क to decrease due to (10).
KTO: du & Am where Aw \ are two hyperparameters in KTO, fine-tuned according different tasks and datasets. Thus general conclusion the chosen/rejected probability change can be made from our conditions.
di X,
Explicit regularization on chosen log- bog-probability (CPO, DPOP RRHF and Slic-HF):: Ac- cording to the formulas of d, and di (7) and (8), the negative log- likelihood (NLL) regularizer on chosen responses enlarges d,,, while having no influence on dj as A' 2 O and only appears in du (7). As result, larger d; makes condition (9) more lenient and thus the chosen log-probability is more likely to increase.
Lenggiinormallzztion (SimPO, RRHF and IPO): In SimPO, d
and condition (9) and Iaal
ludl
(10) can be rewritten as:
log Te log ni
V log Tu. Wio
V log Tw Buol
V log m
log m
<
(11)
These conditions imply the following: to ensure increasing chosen log-probability while de- creasing rejected rogprobbbbility, (11) should hold. This is lenient than the corresponding condition posed for DPO that (V log Tw, V log m)) < min(//V log T.ll log 12), when the length of chosen and rejected responses is biased so that either the chosen or rejected gradient norm is significantly greater than the other. Therefore, compared to DPO, SimPO leans towards increasing the chosen probability and decreasing that of the rejected when the preference data
6See Scction C.2 the derivation.
Forr DPOP, the regularizer is included in its (a) term in Table 2, due its design to turn on/off the regularizer based the value of chosen log-probability.

---

Published as a conference paper at ICLR 2025
is heavily length-bul biased. The same reasoning also applies to RRHF and IPO for their length normalization design.
3.3 EMPIRICAL OBSERVATIONS
We conduct experiments on the TL;DR dataset (Stiennon et al., 2020) to showcase the widely- existing phenomenon that the chosen and rejected log-probabilities have synchronized changes dur- ing preference optimization. In addition, Figure depicts how different margin- based preference optimization algorithms influence the log-pro log-probability of chosen and rejected responses.
CPO
DPO
DPOP
My
-100
1400
-55
-140
R-DPO
RRHF
-00
logp
-180
-30
chosen
-10
rejected
-100
SimPO
SlickF
-100
wmmmmmg
-20
-150
-200
2,000
4,000
Training Step
Figure 2: Training dynamics of the chosen and rejected log-probabilities on the TL:DR dataset for different algorithms trained on Mistral 7B. The corresponding plot for Llama3 8B is in Figure 6 (Ap- pendix F). For SimPO and IPO, the log- oog-pobbabilitis are normalized by the response length, while the other plots, the log-probabilities are of entire responses. All algorithms exhibit synchronized increases and decreases in the chosen and rejected log-probabilities. We also provide the cosine similarity plots between Vo log ue and Vo log xy in Appendix (Figure 7).
For DPO and R-DPO, both the chosen and rejected log oog-oobbbbilities tend decrease simultane- ously. This behavior proves the existence of gradient entanglement, showing that methods purely dependent on the margin might result both terms decreasing, with the rejected te-probability decreasing more significantly.
SPPO demonstrates a distinct trend where the log-probability of the chosen responses increases, while the log-probability of the rejected responses decreases. This matches the theoretical intuition obtained from the specialized gradient conditions for SPPO in Section 3.2.
For CPO, DPOP, RRHF, and Slic-HF, algorithms with explicit regularization on the chosen log- probability, we observe a consistent increase in the log- log-probability of the chosen responses. This behavior reflects the effect of explicit regularizations in increasing the chosen log-probability, which also aligns with the conditions discussed in Section 3.2.
SimPO and IPO in Figure report the average log- peebbabbbbiltyy of responses. Again, increase in the margin is guaranteed, but not necessarily an increase in the average chosen fog-probability due to the gradient entanglement effect.
Overall, experimental results on various margin-based losses closely align with our analysis on the gradient entanglement and the gradient conditions outlined in Section 3.2, demonstrating how loss structures, explicit regularization, length normalizaooniiiinnnn and other design choices influence the dynamics of preference optimization.
INVESTIGATION ON GRADIENT INNER PRODUCT
The previous section reveals that the gradient entanglement effect is driven by the key quantity: (Vo log Rw, Vo log m) (Condition 1, 2: gradient condition). Margin-based objectives are often trig-
'In TRI. library, the implementation IPO averages the log-prohahilities by the number tokens. In their original paper, Azar et al. (2024) proposed the TPO loss without average log-prohahility. The authors later claimed using average lrobbrobyyility with IPO yields improved performance.
7

---

Published as a conference paper at ICLR 2025
gered to not behave in the ideal way, suggesting that the gradient condition is violated due large gradient inner product. Therefore, this section, we investigate into such inner product to understand why it can be large when aligning language models.
Our investigation focuses on the representative margin-based objective DPO and we use toy syn- thetic settings to analyze this problem and build up general intuition. All proofs in Ap- pendix D. Key insights obtained from our analysis are: (1) the gradient inner product increases as the chosen and rejected responses share more similar tokens; and (2) while the sentence-level gra- dient inner product can be large, individual token-level inner products may be small. 10 We then empirically verify our intuition Section 4.2.
4.1 THEORETICAL RESULTS
4.1.1 POSITIVE RESULT ON WHEN THE CONDITION HOLDS
We first provide a positive result when Condition 1 holds and DPO has the ideal behavior that pushes up the log--roballe pro-obbabilityy chosen and pushes down the log- pog-probability of rejected. We begin with set-ups for the LM and preference data.
Model Setup (LM with learnable last linear layer). Let V = |V| be the vocabulary size. We assume for prompt I and response y, any index i in the response, the LM outputs:
Te | x, ". s(h; 0)(y"},
where L lvl. A RaxV is the learnable parameter, h, E Rd is the hidden state for predicting the i-th token in y and s RV Av denotes the softmax function.. The hidden states are assumed as frozen during DPO.
Data Setup Both chosen and rejected responses contain only one token under the prompt æ. That is, Vwn E VL, and yw [1] f yi [1[11
The following theorem shows this task, (V log Tw: log mi) < so that gradient descent steps of DPO make sure log uw increases and log ni decreases.
Theorem 1. Under Model Setup and data Setup 1, assume after the SFT stage, given prompt I, the model prediction on the first token in response is uniformly concentrated M S V tokens in the vocabulary V, then we have
V log Twi Vlog mi)
M 11113
II log 1 [
mll
M ||A11
with h being the hidden state for predicting the token that follows prompt x. Thus. both parts of Condition I hold, resulting in log Tw increases and log n. decreases.
Theorem can be extended to the data setup where the chosen and rejected responses have multiple tokens but only differ at the last one, i.e., yue [1 - yn L 1]. ym * yx with L being the number of tokens in Yo yt-
Corollary 2. Under Model Setup 1, the chosen and rejected responses only differ at their last token, assume after SFT the model prediction on the L-th token response is uniformly concentrated on M < V tokens in the vocabulary, we have (V log T ( (r, " V log T (w) [x. (2, =² < 0, thus at token L, the chosen log-probability log iT (yt (r,vz) will increase and rejected counterpart will decrease.
From the proof of Corollary 2 in Appendix D, though the log-probabilities on the last token be- have ideally, is not guaranteed that the whole chosen response ya will increase its likelihood and log (I) x)) will decrease, due to the correlation between V log and V log
4.1.2 NEGATIVE RESULT ON WHEN THE CONDITION VIOLATED
From the previous results, we can see that the gradient inner product condition is not violated and DPO has the ideal behavior when the chosen and rejected responses differ only at the last token. To gain theoretical insights on what causes the violation of the condition, we level up our previous data setup the following.
1oo be specific, token-wise gradient, Here, A denote the prohability simplex. For vector y, we y(i] to denote its i-th entry and yfds i2] denote its entry from ái ig.
Va log ne (ry''s,,0 (x
8

---

Published as a conference paper at ICLR 2025
Data Setup 2. Chosen and rejected responses have an edit distance 1 and the difference appears in the middle of a response, i.e., the chosen and rejected responses yu E VL. and yi E V satisfy yu 1] = y1 m 11. Yv [m]] # yt [m], Yue + 1: L] yr [m + 1: L] for 1 m L.
To analyze the optimization steps of DPO under this data setup, adopt simpler setting for parameterizing the LM, where the LM has learnable logits.
Model Setup 2 (LM with learnable logits). We first consider the setting where the LM output follows the structure: For index E [Z].
ng(-1ar y Sw.d T0(110, = 81,1-
where S 81,5 E By are the probability distributions of the chosen and rejected response at token i, respectively. We assume the parameterization: Sw,i = s(ww,)) and S1,1 s(01,a) with being the softmax function, where Ow and 0₁ are learnable in the model and to which we take the derivatives.
1]. we have that = Sm,, 81,1 for i E [m]] Since 8... and 81, are predicted by a shared model, they are not independent and one may impose assumptions to characterize the relationship between them. We denote for i [m + 1 LJ. J be the vocabulary index of token appearing at [t]
Because yho [1 -1] -1] w1
[i] and [r] As in Pal et al. (2024), we assume that sw.ii 2 81, [if] and [ S sū, for + jī. Under this assumption, Theorem 3 shows that in this case the log- probability of the chosen and rejected will likely both decrease after one DPO gradient descent step.
Theorem 3. Under Model Setup 2 and data Setup 2, after one DPO step, the per-token log- probability change in chosen response Yu be characterized with first-order Taylor expansion: for E m - 11, the per-token chosen log-probubility before the differing loken stays unchanged:
A log ( Ir, Veu O.
(12)
For = m, the chosen log probbabbbyi the differing position will increase: suppose j and k* are the indices of yw m)] and yi [m] in the vocabulary V.
A log π TV=
Ir, x,,= x 1 (8w.m [5"] Sw. [k*)) 2 0.
(13)
For E [m 1: L}, the chosen log prgobbbbayility these positions will decrease:
A log (v' F. ye x, zo (1-88.:(7))ss...l sw. sw.llit])
Σ
8w, ( S O,
(14)
0. Given the change in sentence-wise log- probability of chosen is the summation of the per-token changes specified in (12), (13) and (14), as the same suffix following the differing tokens gets longer, log nu decreases more.
since - S O and st 5m, s.
Remark. While Theorem 3 adopts the same assumptions made Pal et al. (2024), we precisely characterize the per-token log probability changes based on the first-order approximation, and ex- plicilly break down the sentence-wise probability change for chosen into 3 parts: before/at/uster the differing position. Therefore, the analysis in Theorem 3 captures the varying probability change directions at different positions, uncovering the underlying dynamic behind the overall decreased chosen probability observed in experiments (Figure 3).
4.2 EMPIRICAL OBSERVATIONS
We verify our intuition regarding when the gradient inner product condition may be held or violated using sentiment classification task trained on GPT-2, where the prompt x statement, e.g., "Happy mothers day mom xoxo." The chosen response yhe specifies the correct sentiment, while the rejected response l gives the wrong one. We consider three styles of responses:
Single token: V positive. Wr negative.
Short suffix: yo has a positive sentiment. yı has a negative sentiment.
Long suffix: Yo It has positive sentiment based on my judgement. yı: It has negative senti- ment based on my judgement.
Empirical observation validates three implications obtained from our theorems:

---

Published as a conference paper at ICLR 2025
First, As showing in Figure 3, the chosen log probability increases only in the single token case, aligning with the theoretical prediction by Theorem 1. The short suffix chosen log probability de- creases less than that of the long suffix responses in long suffix contain more tokens following the differing spot, aligning with the theoretical prediction by Theorem 3.
Taking one step deeper behind the behavior of log- pro--abbbilitiss, the gradient cosine similarity in the single token case quickly declines and stays negative during training, while that in the short suffix and long suffix is positive and increases as the suffix length grows (Figure 4a). This aligns with our gradient condition (Condition 1), where the drop in chosen log probability depends on the magnitude of the gradient inner product.
Finally, we inspect the token-wise gradient inner product in the long suffix case. From the heat map of token-wise gradient similarities (Figure 4b), we observe that on the diagonal, the inner product between the gradients on the tokens "positive" and "negative" is negative, whereas for other identical tokens in the two responses, the gradient cosine similarities are significantly higher and close to 1 for some token pairs.
Single Token
Short Suffix
Long Suffix
-20
log?
cheson
rejected
-40
0
200
400
200
400
200
400
Training Step
Figure Training dynamics of the chosen and rejected log probabilities for sentiment tasks.
Our theoretical and empirical investigation into the token-level gradient inner product suggests broader implications for general alignment tasks. Significant tokens (c.g.. "positive" n"negaiive"" contrasting the chosen and rejected responses the most, exhibit negative gradient correlation and prevent gradient entanglement. Mcanwhile, those non-contrastive insignificant tokens (c.g.. iden- tical tokens) cause gradient entanglement due to the high similarity in their gradients. This insight highlights the importance of token-level gradient dynamics and their contribution to the entangle- ment effect.
Single Token
Short Suffix
Long Suffix
Gradient Cosine Similartt
0.2
0.0
-0.2
100
200
300
400
500
Training Step
(a) Cosine similarity between V log wh and V log Ri-
(b) Token-wise gradient cosine similarity.
Figure 4: Gradient cosine similarity behaviors on the sentence-level and token-level for sentiment tasks, Fig- ure 4a gives the cosine similarity between V log we and Vo log for DPO single token, short suffix and long suffix datasets, defined as: TV log niw Ill o azg Figure 4b shows the token-wise gradient similarity for an instance in the long suffix task.
loo Vn log mi)
5 IMPLICATIONS
In this paper, we touch upon common pitfall of margin-based preference optimization methods in language alignment. At a high level, our work highlights the need to reconsider the current margin-based preference optimization paradigm. While this approach may enable language models to effectively learn contrasts between good and bad responses, it may not be well-suited for settings where the focus is on the behavior of either the rejected or chosen samples-such as in safety-critical alignment tasks or when distilling from a strong model. See more discussion in Appendix A.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
Mengdi Wang acknowledge support by NSF grants DMS-1953686, IIS-2107304, CMMI-1653495, and ONR grant 1006977. Huazheng Wang acknowledges the support by NSF grant IIS-2403401.
REFERENCES
Mohammad Gheshlaghi Azar, Zhaoban Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandricllo. A general theoretical paradigm to understand learn- ing from human preferences. In International Conference on Artificial Intelligence and Statistics, pp. 4447 4455. PMLR, 2024.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.58822 2022.
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01335. 2024.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuaa Imsys. org (accessed 14 April 2023), 2(3):6, 2023.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, et al. Ultrafeedback: Boosting language models with sealed ai feedback. In Forty-first International Conference Machine Learning, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.2 1783, 2024.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. In International Conference Machine Learning, 2024.
Jiwoo Hong, Noah Lee, and James Thome. Orpo: Monolithic preference optimization without reference model. arXiv preprint arXiv:2403.07661, 2(4):5, 2024.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https: //aaxiv..rgg/aaaaaeeee.. 2210. 06825.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-free reward. arXiv preprint arXiv:2405. 14734, 2024.
Sidharth Mudgal, Jong Lee, Harish Ganapathy, Yaguang Li, Tao Wang, Yanping Huang, Zhifeng Chen, Heng-Tze Cheng, Michael Collins, Trevor Strohman, Jilin Chen, Alex Beutel, and Ahmad Beirami. Controlled decoding from language models. arXiv [cs.LG], October 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with dpo-positive. arXiv preprint arXiv:2402. 13228, 2024.
11

---

Published as a conference paper at ICLR 2025
Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization. Findings of the Association for Computational Linguistics. 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
Amrith Setlur, Saurabh Garg, Xinyang Geng, Naman Garg, Virginia Smith, and Aviral Kumar. RL on incorrect synthetic data scales the efficiency of LLM math reasoning by eight-fold. arXiv [cs.LG]. June 2024.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-30211 2020.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpaca: A strong, replicable instruction- following model. Stanford Center for Research on Foundation Models. https://erfm. stanford. edu/2023/33/1// html, 3(6)77, 2023.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models a practical size. arXiv preprint arXiv:2408.001 18, 2024.
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of Im alignment. Conference on Language Modeling. 2024.
Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment. arXiv preprint 4rXiv:2405.00675. 2024.
Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of Ilm performance in machine translation. In International Conference Machine Learning, 2024.
Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback. Advances in Neural Information Pro- cessing Systems, 36. 2024.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305. 2023.
12

---

Published as a conference paper at ICLR 2025
A BROADER IMPLICATIONS
In this paper, we touch upon a common pitfall of margin-based preference optimization methods in language alignment: it underspecifies the ideal behavior of the LM on the chosen and rejected responses individually. Our gradient inner product condition suggests that when the chosen and re- jected gradients are similar, their log probabilities will exhibit synchronized increases and decreases. Using this gradient condition, we can categorize existing RLHF variants into two types: (1) those that modify the criterion for the size of the inner product, as seen in the works listed Table 2, which rely on the same gradient inner product but apply different size criteria; and (2) those that change the inner product of interest directly. As discussed in Section 4, while the sentence-level gradient inner product may be large, the token-level inner product can be small. A line of research, such as advantage-base based meudads et al., 2023; Setlur et al., 2024), focuses on leveraging token-level information to improve RLHF and falls under the second category.
Finally, at a high level, our work highlights the need to reconsider the current margin- based prefer- ence optimization paradigm in language model alignment. While this approach may enable language models to effectively learn contrasts between good and bad responses, may not be well-suited for settings where the focus is the behavior of either the rejected or chosen samples-sude in safety-critical alignment tasks or when distilling from a strong model.
B REVIEW OF PREFERENCE OPTIMIZATION OBJECTIVES
Motivated by non-transitive human preference and language model calibration respectively, Azar al. (2024) and Zhao et al. (2023) propose IPO and SlicHF loss with similar forms that solely depend on the margin log ( x>) - log ()())
Due to the length bias observed in practice, Park al. (2024) propose to add length penalty term in the BT preference model, but the gradient still relies on the margin log To (x) log ( (r). Meng et al. (2024) and Yuan et al. (2024) consider the setting of average rewards and derive loss dependent on the length- norggl-ed margin
[ log Te (B
Teil log To (v]lz))
Unlike prior work, Ethayarajh et al. (2024) and Wu et al. (2024) do not consider the difference between the likelihood, but deal with the chosen and rejected response separately. These works typically assign positive reward signal to the chosen response and negative reward signal to the rejected one, according to the logistic loss (Ethayarajh et al., 2024) or the square loss (Wu et al., 2024).
(Pal et al., 2024) observes a decrease in the log-probability of chosen response during DPO when the edit distances between each pair of completions are small in preference datasets. To fix the decrease, a natural way is to add explicit regularization to the loss objective, to force the increase of the chosen response's 's log-probability. In particular, (Pal et al., 2024) propose the DPOP loss that behaves the
[x) is above O, while adds an explicit same as DPO when the chosen response's ' log-ratio log Reel w)
regularization when the ratio is below 0. Similarly, Xu et al. (2024) and Zhao et al. (2023) also add explicit regularization to maximize the chosen response's log-probability.
Among these works, the most relevant to ours is Pal et al. (2024), which touches upon a similar failure mode of DPO. The main difference is that they focus on mitigating only the decrease mode of the chosen response's 's probability by new loss designs. In contrast, we dig deeper to obtain a broader view on the synchronized change (increase or decrease) in chosen and rejected probabilities. We rigorously analyze the training dynamics and extract general success/failure conditions based on gradient correlation, which applies to a range of margin- based losses for preference optimization.
C DERIVATIONS FOR GRADIENT ENTANGLEMENT AND CONDITIONS IN SECTION 3
C.I DERIVATION FOR GRADIENT ENTANGLEMENT
DPO. After one step of gradient descent with step size 1) > 0 for decreasing the loss EDPO, the change in the log-probabiligg of the chosen response denoted by A log Tu, as well as the change in the probbabbbyl of the rejected response denoted by A log wi can be approximated by the
13

---

Published as a conference paper at ICLR 2025
first-order Taylor expansion:
log The RJ ( (Vo log Tw, - VoDDDO = r)/c(()) (1 log Tw (V log Tw, V log mi)
A log ~ Vo log mi. ~VoDro) - 1)āc(()) ((V log Tur; V Vlog |/ log x
General Losses. First, the gradient of (6) can written as
Vol du Vo log Tee Vo log mi,
where du and di are scalars such that
duy == m' (log Tw) - hn (log m. M (log mo) mw) ) + A ((log
di = m' mm )w) (yog m. (/ m)).
After step of gradient descend with step size n 0 for decreasing the loss L, the changes in rog-probabilities can be approximated by the first-order Taylor expansion:
A log The (Vo log Twy -1) Vn)ael) ol) 1) (dmel Vo log Tw II2 di (Vo log Tie Va log (i))
A log TI (Vo V log mI, - V = (dho V log Fas V log mi) dill Vo log m. m1ll²)
C.2 DERIVATION FOR SPPO
Denote V log T (w) and b Vo log ((). For DPO, we see that the direction of winner and loser is decided by (a, b) and (b, b).
Similarly, for any pairwise loss E(log (w) log x the above statement still holds. Now take
a look at non-pairwise loss (SPPO = (log n (w) 3 1)2 2 + (log + B 1² We have
d8 di
Volspro -(log((w) - 3-1)ao log (w)) - (1) + /> (l).
Then
d0 log xi)) = V log (i), dt
dt
= (log=(w) x(w) - - Vo log n (1), Vo Volog((w)) - (log + +V log Vo logom(()) (1)).
We have
d log T(w) - 3 (a,) (log 7(1) + b)
di
which means if want log (w) to increase, need
aa b)
8 - log (())
a.
a, 3-1 + log x0))(11
Note that the inequality above implicitly assume that 8-1 log (l)) 0. This is true in practice as we set 3- to be extremely large. Similarly, if we want log π (1) to decrease, we need
(a, b) <
g + log π (1)
-1
(b. 8-1 log π (ww)
We have > 1. It seems SPPO can make sure that log T (w) goes up more easily but also make log T() goes up more easily, compared DPO.
D PROOFS FOR THE GRADIENT INNER PRODUCT IN SECTION 4
D.I LM WITH LEARNABLE LAST LINEAR LAYER: SINGLE TOKEN CASE
We prove Theorem below.
log Nw. logm) =(Vo log x), Ve log π(y |x)),
where 9 E R Let h E Idd be the hidden state for the token next to the prompt, s(-) is the softmax function, then
14

---

Published as a conference paper at ICLR 2025
Vo log r (yw x)) = V (log s(h T 0)lul)
(15)
Vo log (yt x)) Ve (log s(h 0)
(16)
Compute the gradient with chain rule,
Ve log Tw = [-> ... (1 s((w (iw))) -())4 . Va log ग (-s()),,
(17)
siww), (1 s(i)),,.... ((V))))
(18)
Buy is are the index of token y and yt in vocabulary, respectively. For any index s(iw) denote LLM's output logit for the i-th token in vocabulary.
Suppose the initialization of s(1 rest V-M entries have they are equal to O. We note that the exact indices j of which s(j) = 1/M does not matter would be the same index for both the chosen and rejected gradients.
s(tw)
- #o for M entries and
1 M
1
V log Tw = =1-
1
h
h
h]. M
(19)
M
M
-wtt
à i--th
V log क. =
h M
M h, ...
1 -
h .....
M h],
(20)
M
in
4-thh
M 2
M
1
V log Tw, logmi m()
|1
||]l2
M 11112
(21)
M
M
V log Tw, V log m) is negative. While in comparison. the norms of V llog vu and V log m follow:
M 1
(1
M
Vog w 12
V log m||
11112
|1112
|
M2
M
M
Therefore, based Condition 1:
1
V log Tw: V log m²)
h11
M
M
-1
V log Tu V log ml|
|11113
M
log Tw increases and log m decreases.
D.2 LM WITH LEARNABLE LAST LINEAR LAYER: MULTI-TOKEN PREFIX CASE
Recall the data setup: the chosen and rejected responses have multiple tokens but only differ at the last one, i.e., you [1: 1] y [1 L 1]. yo [L] 7 yn [L] with L being the length of you and y,. We prove Corollary below.
In this case, up the L-th token where chosen and rejected differ, the hidden states are the same for the two responses. This is true because yn.[l L 1] yı L- 1] and the share the same prompt T, so have that hiw. ha for i 1,. L. thus we denote both hiw and h42 as ht.
For any index i, denote log To (vi. 2)) by log कं. and denote log ( xx)
by log T/. then we have
log Te
log ti
log ग
log t/;
(22)
L
(V log Twy V logm))
(lg गं. log m/)-
(23)
Let h4 E Rdd be the hidden state for predicting the i-th token, s(-) is the softmax function, then
V log Tं. Va (log sh, ())w)) log कf Vo (log s(h, 0) (wd)))
15

---

Published as a conference paper at ICLR 2025
among which have Vg log Th Ve log oं for i E [L - 1] because yu v/ at those indices. For i L, computing the gradient with chain rule, we have
f. = (- " ,1 V log " w
(i())li,, " " (V)..b - s( - hV))))
Vo log .. = (1)hi,, ... - ((tm)),
iw it are the index of token v and yf in vocabulary, respectively.
Suppose at the initialization of 0, s(1) = ... = s(tro) _.. = s((r) s(v) = 37 for M entries and the rest V M entries have {
Similar to the proof of Theorem 1, we have
L log क ie
V log ut) = M |mull?
(24)
V log n = V log o. 112 M -
11r.111 -
(25)
M
Therefore, by introducing notations a; := Ve log Tr and b, == Vo log कt
Vo log कं for E [L 11. bw =- Vo Tog
L
L
Ve log Tw: Vo log m)
Σ
Vo log Thu Ve log log
1-1
Σ
(ai, az) +
Uz bi)
ais bm) + (Bws bi);
==1 1=1
L
V log um 1|2
Σ
Σ
Vo log The Vo ml))
Σ
(ai, az)) + 2
as bw) +
Oi, bw) + 11b
==1 ==1
1=1
V log m||2
flog tł, log nt)
L-1L-1
Σ
(ai, a²) +
ais bi)
ais bi) + 1|b111
From the equations above, it's ensured that
V log Tw 12 + Vo log |/
(Vo log Tan, Vo log m)
(26)
due to (b... bi) than min( 9 log n., 1 Ve log m) depends on the exact absolute value of the term 2-1 0g Vo log The Vo log m)) recall greater than || 2 then (V, log Rw, Vo logog m)) > min( log T 1|2 IIV logm||
0. However, whether
(Vo log Twr: Vo log m) will be greater less
Vo log T Volog nf. If this absolute value is
ml)l2) the condition violated, otherwise the condition is satisfied. When L is large, other words, the prefix is long, then the condition is more likely to be violated, leading to the side effect of gradient entanglement.
D.3 LM WITH LEARNABLE LOGITS SETTING
We prove Theorem 3 below. We will set up some new notations first. First, we work with the case where Tnm = Ti L is sentence length, V is the vocab size, yo m 1] yr Vau [m] A y1 [m].. and yur(m + T. = y1 [m +
1]. T.]. Note that for all E [7.]. the token
yli] [V] is an index, 0n and 0, are learnable logits in LM. Each row of the following matrix is
To(-la |2, ") A[v] where is the row index. (Here, there is a slight abuse of notation: A is the probability simplex.) s: R Av is the softmax function.
16

---

Published as a conference paper at ICLR 2025
s(0i[1,:))
(0wl11.)
s( mm.mmmm
(m) :I)
(00lmm)))
00, 10,11 1x 3 mo yu) = s(Ow)
T0 31) s(Oi)
s s0 (m + 1,:]
s(0:|m 1,:1)
s s00[[m + 1,
s(0 (L,:
Each row :((i,:)) Av. The first are the same for 6., and 0, because the tokens up row m are the same between and yt- The index at row corresponding to the selected token will be denoted as f. generic vocab index Note that, J vi.w Jū,1 for o m, and j7,u + Jī, for i i= m.
Next, the corresponding gradient matrices log sBww.. log s(O)) can be specified by:
0
LLV V s(O.it
V Venit log s0 jil
Ve log s(O.)
V Vo log (Or(t,
0
where
-s(i,,]] s(i,
if
Vallal log s0(ii,))) RV, and for E [V], Vak.a log s(0ji,
if
where sir, ( :D)) log J,1, is jū-th entry of log s(fii,)), and log s(0 ((((.,1
the j-th entry of the gradient log ji)..
The sentence-wise gradient is
log (1, V log (1,))))))
V log (0 [m, im,,,,nma wm V log s(O [m, (nn jmml) log (0 [m + 1m)
LLVV VoL
V log s0,, [m + 1,
log (0
log s0.(L.,t))
0
V log ((0 Sm, w))
V log s0 (m, jth
v log s (0, [m + 1, 1,
V log s m + 1,
V log s0 [L. (7.)
V log (L,
17

---

Published as a conference paper at ICLR 2025
Now, let's first derive the token-wir wise condition for the selected token (learning rate 1): Chosen response: if m, we have
A log
V log s0 (m, (o.l): VL log (m. V. (Lmm,)))>
P=1
= (V log s(0 (m, Im,,,),
V log s(B [m, Jmm wl) - V log s(O (m..))..))))))
Σ
Sw m,, j'2
+ (1 Sw (m, Jm,
J' M'ttiim
sw j9p²
+ 8w. sm, 8w [m, + 8w [m, ,1(1 - 8w [m
=1 + (8w [m, Im,il Sw [m, Jm, wl) 2 0,
(27)
where the last inequality true because s [0, 1]. Here, basically, this margin loss will just encour- age increase the chosen logP (and reduce the rejected one) for the selected token.
Chosen response: if of m. we have
A llog (full =
v log (0ul((O VCF, (V log 5i..
= =V log sf V log @f (i1)
log s( (il))))))
=(1 sw-s [ú. (s:
(1 P. oil (..i.)))
E s j'I s (i,)')
(28)
Here, basically, the loss can only pick one direction change both chosen and rejected entry.
Connection to the derivation in Pal et al. (2024). The assumption in Pal et al. (2024) mainly ensures the sign of (28). Basically, smaug's assumption ensures that for i [m + L}. sw (i. si(i,j]] and sw [i I [i for + jf.
stli, 1]
8. 1ú, 1]
0
v log ((w(i,)))) - V log (0.(i,))))
8mu (i, = s((s,
00
st Vl - 1i', V
For (28), we have
(1 (1-8ul (=l15,11))ss(( sw(i.ft))
Σ
swli, sw ('l s 0.
J'
This ensures the chosen token will have reduced logP.
Condition chosen tokens increasing and rejected token decreasing m, and chosen and rejected tokens decreasing after m + 1:
(27) 2 always holds, Vi E [m + 1, L), sw s.ii 2 stbifi Vj + sw stb.j] (28) < 0
E EXPERIMENT DETAILS
E.1 HARDWARE AND SOFTWARE SETUP
Our experiments implemented using TRL version 0.11.0. The training performed hardware setup consisting of two NVIDIA H100 GPUs, providing substantial computational power for the training process.
18

---

Published as a conference paper at ICLR 2025
E.2 TL;DR TASK SETUP
For the TL:DR summarization task, utilized the ChaprpA/lopmiiiissmmmmaaaaaaaoon dataset. We employed two LLMs for this task:
misstalai/Mistrrl/BBBnnsooooooo (referred to Mistral 7B)
meta-llama/Mett-llam--33
(referred to as Llama-3 8B)
We did not perform any supervised fine-tuning step prior to the RLHF training for these models. To optimize the training process, we applied Low-Rank Adaptation (LoRA) with rank of 64 to both models. The learning rate was set at 5 x 10-6 for all RLHF training.
E.3 RLHF ALGORITHM CONFIGURATIONS
We implemented several RLHF algorithms, each with its own specific confi configurations:
Direct Preference Optimization (DPO): 0.1
Chosen NLL term (used in CPO, RRHF, and SLiC-HF): À
SLiC-HF: =
SimPO: 7 = 0.5
R-DPO: a = 0.2
DPOP: À 50
E.4 SENTIMENT ANALYSIS TASK SETUP
For the sentiment analysis task, we used a specially curated sentiment dataset. Unlike the TL:DR task, we performed supervised fine-tuning on the GPT-2 model before proceeding with the RLHF training. The learning rate for this RLHF training was also set to x 10-6
F ADDITIONAL EMPIRICAL RESULTS
Llama3.1-88
Llama3.2-88
Mistral-TB
Gemmaz-98
W
TLOR
mM
Ultral Ultral eedback
Figure 5: Training dynamics of the chosen and rejected log probabilities during DPO, observed across models: I.lama.11-8 1-8B (Dubey et al., 2024), 1lama3.2-3B, Mistral-7B (Jiang et al., 2023) and Gemma2-9B (Team al., 2024) TL:DR (Stiennon et al.. 2020) and UltraFeedback (Cui et al., 2024) datasets. Log probabilities are averaged on the evaluation splits.
19

---

Published as a conference paper at ICLR 2025
-80
-130
-100
-100
-120
IPO
R-DPO
RRHF
-4
-60
logP
chosen
-6
wwm
-80
rejected
-100
SPPO
SimPO
SlicHF
-100
-60
-20
-80
-150
-40
100
0
2,000
4,000
0
2,000
4,000
2,000
4,000
Training Step
Figure 6: Training dynamics of the chosen and rejected log probabilities on the TL:DR dataset for different preference optimization algorithms trained on Llama-3 8B. All algorithms exhibit synchro- nized increases and decreases in the chosen and rejected log probabilities. Note: For SimPO and IPO, the log probabilities are normalized, while the other plots, they are the original log probabil- ities.
CPO
DPO
DPOP
0.90
0.85
Gradlent Cosine Similarity
IPO
R-DPO
RRHF
Mistral 7B
0.85
Llama-3 8B
0.80
SimPO
SlicHF
0.90
0.80
2,000
4,000
6,000
2,000
4,000
6,000
2,000
4,000
6,000
Training Step
Figure 7: Cosine similarity between Va log TH. and Va log क on the TL:DR dataset for different preference optimization algorithms trained on Llama-3 8B and Mistral 7B.
20


Reviews:
[{"review_id": "LiQPLpRGEG", "reviewer": "Reviewer_Ftet", "summary": "This paper considers a common failure mode of RLHF algorithms like DPO, in which, although the difference between preferred and dispreferred outputs is guaranteed to increase, there is no guarantee that the log likelihood of the preferred option will increase, or that the log likelihood of the dispreferred option will decrease.  This paper derives a general criterion that can be used to identify pairs in which either both will increase, or both will decrease.", "questions": "Is there any strong argument that the problematic behavior addressed by this paper (both go up, or both go down) is important in real-world settings?\n\nI was intrigued by the claim that the log likelihood of a dispreferred output can go up, but this seems to not be a problem that has concerned any previous authors.  The alternative approaches discussed in the paper all address the problem that the log probability of the preferred option might go down, and that outcome is the outcome suggested by the last example analyzed in the paper.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The theoretical analysis was enjoyable to read. The examples demonstrate the proposed criterion quite clearly.", "weaknesses": "The provided examples are tremendously simplified, which is useful to demonstrate the applicability of the theoretical analysis, but which reduces confidence that the analysis will have much impact on real LLM practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dNG4jQ9Xue", "reviewer": "Reviewer_5B6v", "summary": "The paper investigates issues in margin-based preference optimization for language model alignment, with a focus on the Direct Preference Optimization (DPO) algorithm. It highlights that margin-based loss specifies only the ideal margin behavior, overlooking individual term behaviors, which often results in synchronized log probability changes between chosen and rejected responses. The authors theoretically derive conditions for this phenomenon starting with DPO, and extend their insights to other margin-based algorithms. They examine cases where the derived condition holds, noting that higher token similarity between chosen and rejected responses can intensify synchronized movement, and validate these findings empirically.", "questions": "1. The paper presents a clear delineation between preferred and dispreferred samples. \nHowever, some papers focus on distributional framework. This approaches insists the model to encounter a broader range of sample types to cover the distribution's full support, supporting the original RLHF sampling method.\n- Go, Dongyoung, et al. \"Aligning language models with preferences through f-divergence minimization.\" arXiv preprint arXiv:2302.08215 (2023).\n- Korbak, Tomasz, et al. \"On reinforcement learning and distribution matching for fine-tuning language models with no catastrophic forgetting.\" Advances in Neural Information Processing Systems 35 (2022): 16203-16220.\n\n Could the authors discuss the potential impacts of their idealized setup compared to a distributional approach? This clarification would help assess the implications of choosing one framework over the other in alignment training.\n\n2. Training with both high- and moderate-quality responses, where one is better but another may serve as a hard negative example, can push the model to better distinguish between fine-grained preference levels. The ideal condition that assume clear difference of the quality  between preferred and dispreferred samples, seems to take a different approach. Does this work incorporate hard negative samples, or is it constrained to cases with explicit distinctions?\n\n3. The findings on token-level gradients provide an intriguing direction for refining preference optimization. Could the authors elaborate on any potential methods to leverage token-level distinctions more effectively within their model, perhaps as a future research direction or via fine-grained adjustments?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "This paper stands out in its originality by addressing the under-explored limitations of margin-based preference optimization in RLHF, offering fresh insights into its impact on language model alignment. The theoretical contributions, specifically the gradient inner product conditions, demonstrate high quality, providing a clear, mathematically grounded framework that identifies when alignment failures may occur under existing loss structures. \nIn addition to an in-depth focus on the DPO algorithm, the paper comprehensively reviews existing methods in Table 2 and extends its findings in Section 3.2, which gives additional quality to the paper.\nThe suggested condition in DPO is further substantiated with its practical implication, with experimental validation in Section 4.", "weaknesses": "The paper centers primarily on DPO in Sections 3.1 and 4, with limited coverage of other margin-based optimization algorithms. To better align with reader expectations, it would be beneficial to introduce their main focus of DPO explicitly in the introduction and outline how subsequent sections will compare DPO against alternative approaches. Alternatively, devoting more space to a detailed examination of other algorithms could create a more balanced narrative.\n\nThe experimental results in Section 3.3 primarily observe phenomena regarding preferred and dispreferred response probabilities increasing or decreasing together—findings that have already been discussed in previous work, such as Pal et al. (2024), and does not fully investigate the conditions under which these probability trends occur. Furthermore, the experiment relies on a single dataset, limiting the depth of empirical validation. \n\nThe empirical observations are concise but lack important methodological details. For instance, it would be valuable to clarify how log probabilities are computed in Figure 1 (e.g., whether they are averaged over the training samples), and to include additional measures such as standard deviations or confidence intervals to improve the figure's reliability. Since Figure 1 is intended as a motivating visualization, providing a additional figure for dedicated to empirical results (Section 3.3) would allow for a more thorough presentation of the experimental findings without cluttering the introductory material.\n\nFor minor clarification, it would be better to change the interval of the grid to clearly show the trend of margin in Figure2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZWuPvPMlGa", "reviewer": "Reviewer_UXg4", "summary": "The paper identifies and analyzes a fundamental problem with margin-based preference optimization in language model alignment (like RLHF): these methods only specify how much better preferred responses should be compared to dispreferred ones, without specifying how the individual probabilities should behave. This leads to two unintended consequences: (1) sometimes the probability of dispreferred (e.g., unsafe) responses increases, and (2) when reducing dispreferred response probabilities, the probability of preferred responses often decreases too, even when they are ideal.", "questions": "N/A", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper addresses a fundamental issue in language model alignment through RLHF by analyzing the under-specification problem in margin-based preference optimization - this is an important contribution given how widely RLHF is used.", "weaknesses": "1  Presentation Issues:\nThe notation becomes dense and potentially confusing in Section 3.2, particularly around the general form analysis\nSome figures (like Figure 4b) would benefit from more detailed explanations of their implications\nThe connection between the theoretical results and practical implications could be more clearly articulated\n\n2  Limited Experimental Validation:  \nExperiments are conducted on only one main dataset (TL;DR) and a simple sentiment classification task\nOnly tested on two main models (Mistral 7B and Llama-3 8B)\nLacks validation across diverse preference optimization scenarios and task types\nMore extensive empirical validation across different datasets and model architectures would strengthen the claims\n\n3  Theoretical Assumptions:\nThe theoretical analysis in Section 4.1 relies on somewhat simplified model settings (learnable last linear layer, learnable logits)\nSome assumptions about token probability relationships between chosen/rejected responses may not hold in practice\nThe extension of single-token results to multi-token cases could be more rigorously justified", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lvEPIWVsQ4", "reviewer": "Reviewer_ai3q", "summary": "This paper criticizes margin-based preference optimization for LM alignment, which is a core method in RLHF. While increasing preference margins can reduce unsafe responses, it also often reduces preferred response probabilities, due to synchronized probability shifts between chosen and rejected responses. The authors reveal when these shifts occur and suggest that the under-specification of behaviour in current margin-based methods limits model safety and effectiveness. The work proposes fine-grained, token-level approaches as potential improvements, though this is not really pursued very deeply in this work.", "questions": "- While the theoretical analysis of gradient inner products and synchronized probability shifts is fairly rigorous, what actionable insights for real-world RLHF applications are achievable?\n- How would you evaluate the ‘safety-critical alignment tasks’ you mention in Sec 5 as being particularly ill-suited for margin-based preference optimization?\n- Can you clarify more directly how token-level gradient inner products between chosen and rejected responses provide a reliable indicator for alignment behaviour?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper provides a fairly nuanced and theoretically grounded critique of margin-based preference optimization, which is an increasingly relevant task (especially  in RL). Identifying conditions under which preference optimization can backfire is important for real-world use and may provide a relatively underdeveloped area of research\n- The proposal to address preference optimization problems at the token level is interesting (though perhaps not surprising).", "weaknesses": "- this is not meant to be a primarily empirical paper (apparently, it leans more on its theoretical aspects) but this work of course calls for broader empirical validation than it has. The proposed method may be evaluated across diverse tasks such as summarization, dialogue, and instruction-following. Other variants of the given LLMs (e.g., LLaMa) could have been explored, to demonstrate generalizability. \n- The analysis of Sec 4 focuses on synthetic, controlled scenarios. The kinds of complexities exist that in actual, diverse language data should be addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)