Learning Dynamics of LLM Finetuning
Authors: Yi Ren
Danica J. Sutherland
Venue: ICLR 2025 Oral (2025)
Keywords: Learning dynamics
LLM
finetuning
DPO
Primary area: foundation or frontier models, including LLMs
TL;DR: The paper propose a novel learning dynamics framework to understand LLM's behavior during finetuning (e.g., SFT, DPO, and other variants). Some counter-intuitive behavior can be well explained by the proposed framework.

Abstract: Learning dynamics, which describes how the learning of specific training examples influences the model's predictions on other examples, 
gives us a powerful tool for understanding the behavior of deep learning systems. We study the learning dynamics of large language models during different types of finetuning, by analyzing the step-wise decomposition of how influence accumulates among different potential responses. Our framework allows a uniform interpretation of many interesting observations about the training of popular algorithms for both instruction tuning and preference tuning. In particular, we propose a hypothetical explanation of why specific types of hallucination are strengthened after finetuning, e.g., the model might use phrases or facts in the response for question B to answer question A, or the model might keep repeating similar simple phrases when generating responses. We also extend our framework and highlight a unique ``squeezing effect'' to explain a previously observed phenomenon in off-policy direct preference optimization (DPO), where running DPO for too long makes even the desired outputs less likely. This framework also provides insights into where the benefits of on-policy DPO and other variants come from. The analysis not only provides a novel perspective of understanding LLM's finetuning but also inspires a simple, effective method to improve alignment performance.

Published as a conference paper ICLR 2025
LEARNING DYNAMICS OF LLM FINETUNING
Yi Ren
Danica J. Sutherland University of British Columbia & Amii dsuthêcs, cu
University of British Columbia reryi. joshua@gma i com
ABSTRACT
Learning dynamics, which describes how the learning of specific training examples influ- ences the model's predictions on other examples, gives us powerful tool for understanding the behavior of deep learning systems. We study the learning dynamics of large language models during different types of finetuning, by analyzing the step-wise decomposition of how influence accumulates among different potential responses. Our framework allows a uniform interpretation of many interesting observations about the training of popular algo- rithms for both instruction tuning and preference tuning. In particular, we propose hypo- thetical explanation of why specific types of hallucination e.g., the model might use phrases or facts in the response for question B to answer question A, or the model might keep repeating similar simple phrases when generating responses. We also extend our framework and highlight a unique "squeezing effect" to explain previously observed phenomenon in off- offlcol direct preference optimization (DPO), where running DPO for too long makes even the desired outputs less likely. This framework also provides insights into where the benefits of on-policy DPO and other variants come from. The analy- sis not only provides a novel perspective of understanding LLM'' finetuning but also inspires simple, effective method to improve alignment performance. Code for experiments is available at https: / /ggittttthbu/// com/ Joshua Ren/Leenaeeeee Ltw.rnaiggnndnaaaaaiaea. LLMMM.
strengthened after finetuning,
INTRODUCTION
Deep neural networks usually acquire new knowledge by updating their parameters via gradient descent (GD). This procedure can be described by learning dynamics, which links changes in the model's predictions the gradients generated by learning specific examples. With the help of learning dynamics, researchers have not only explained many interesting phenomena during training, e.g.. the "zig-zag" learning path (Ren et al. 2022) and the formation of compositional concept space (Park et al. 2024), but used these insights to propose novel, improved algorithms in different problems (e.g. Pruthi et al. 2020; Ren, S. Guo, al. 2023; Xia et al. 2024).
The study of large language models (LLM) is gaining popularity due to their surprising capabilities on various tasks. To ensure the L.Mss follow human instructions and align well with human preferences, finetuning has attracted much recent attention. Practitioners often start with instruction tuning, where the model learns extra knowledge necessary for the downstream task, and then preference tuning, where the model aligns its outputs to human preference (Ouyang et al. 2022). Various finetuning algorithms have been proposed to fit into this pipeline, with differing explanations as to why they improve the model's performance.
Contrary to most existing analyses of LLM finetuning, which use the perspective of their training targets, their status at the end of training, or their relationships to reinforcement learning (c.g. Ji et al. 2024; Rafailov al. 2024; Tajwar et al. 2024), this paper tries understand LLMs' evolution from dynamical perspective. Specifically, we formalize the learning dynamics of LLM finetuning by decomposing the change of the model's 's prediction into three terms which play different roles. This framework can be easily adapted to

---

Published as a conference paper at ICLR 2025
various finetuning algorithms with different goals, including supervised finetuning (SFT, Wei al. 2022), direct preference optimization (DPO, Rafailov et al. (2023), and its variants) and even reinforcement learning based methods (e.g., PPO, Schulman et al. 2017). This framework helps explain several interesting and counter-intuitive observations during training- including the "repeater" phenomenon after preference tuning (Holtzman et al. 2020), hallucination (L. Huang et al. 2023), the decay confidence of all responses during off-policy DPO (Rafailov et al. 2024), and more.
Moreover, we also provide a new perspective on understanding why off-policy DPO and other variants underperform their on-policy counterparts (S. Guo, B. Zhang, et al. 2024). Our explanation starts by observing an interesting "squeezing effect," which we demonstrate is a consequence of gradient ascent (as in DPO and similar algorithms) models with cross-entropy loss following softmax layer. In short, for each token's s prediction, the negative gradient will push down the model's predictions on (almost) all possible output labels, moving this probability mass to the most-likely labels. This can be detrimental the alignment we are trying to achieve. This effect is most serious when the negative gradient is imposed on an aalread--unikkele label, which is why the confidence of almost all responses decreases during off- off-polliy DPO, Inspired by this, we propose simple, counter- intuiirvitiiivee, but very effective method to further improve alignment performance.
2 DEFINITION OF LEARNING DYNAMICS AND AN MNIST EXAMPLE
Learning dynamics is usually an umbrella term describing how the change of specific factor influences the model's prediction. In this paper, we narrow down describe "how the change in model's parameter 0 influences the corresponding change in fe", i.e., the relationship between AO and Afe- When the model updates its parameters using gradient descent (GD), we have
AO A g++1 0² = =n VL xfo(
(x(x))
foc1 for (xo),
(I)
where the update of 9 during step é - + is given by one gradient update on the sample pair (Xux y.)) with learning rate v). In short, the learning dynamics in this paper address the question:
After an GD update on x., how does the model's prediction on x,, change?
Studying the learning dynamics shed light many important problems in deep learning and also help understand various counter- inttieviviiiieeeeeeee phenomena. Appendix A further discusses related work.
As warm-up, we first consider standard supervised learning problem, where the model learns map x to predictions y = (yi,..., y E V where V is the vocabulary of size V. The model usually outputs a probability distribution by first generating matrix of logits z he (x) R V and then takes the Softmax of each column. We can track the change in the model's confidence by observing log Te(y (y x).
Per-step influence decomposition. The learning dynamics of (1) become,
A log x' (y x,) log Tgr+1 (y x.) - log mg= (y xx).-
(2)
For simplicity, we start from the L = scenario, where the A0 and A log ा can be linked by the following result, version of result of Ren et al. (2022) proved and further discussed in Appendix B. For multi-label classification (L > 1). the updates separate; we can calculate L different A log τ' and stack them together. Proposition 1. Let Softmax(z) and hx(x). The one-step learning dynamics decompose A logn' n' (' xx) -1 A' K (Xoxx Xw) (x)) + [072
(3)
Vx1
VxV
VxV
Vx1
where Axxx = V. log Tg. (x.) = - 1I (xo). K. xx) = (())xx)) is the empirical
neural tangent kernel of the logit network and G (Xxxx V,L(x,, Y.)lz
2

---

Published as a conference paper ICLR 2025
(x, 4.%, Syuing500
-4) y 1i
Menical
Pheditton: tunter
Convergence Early siago
Ou)
Siniar: 502 9
1.2
Profitton of Fumtect
1:13
Opaning
Akge -qutto 201 xp Kil
(4-2
(9,0)
#0r 0
Prediction of sumber
00:9
Jpdaling cass
DV www.batttaeeeeeee...oo.ooo rws.adi (27
70 Orgen)
Cormirdion the acammdieeed
Figure 1:
The per-step learning dynamics and the accumulated influence in an MNIST experiment.
A'(x.) - 1 15 (x.) only depends on the model's 's current predicted probability. The matrix IC' is the empirical neural tangent kernel (eNTK, Jacot al. 2018 8) of the model, the product of the model's gradients with respect to X, and x The analysis in this paper relies on the following assumption:
During the training, the relative influence of learning zu on all other different To is relatively stable.
The common "lazy eNTK" assumption discussed in Arora al. (2019) is sufficient but not necessary condition for this paper. Appendix C provides a more detailed discussion and experimental verification for both MNIST and LLM settings. We can then think of K as a model-specific similarity measurement between different input samples: larger |AC*) || means the update of x likely influences the model's 's prediction on x more. Finally, G is determined by the loss function L, which provides the energy and direction for the model's adaptation. For example, for cross-entropy loss LCE - -y log (y xu). we have Ut Tg (y xx)) Ya. a length-V V vector that points from the model's current predictive distribution to the desired supervisory distribution. For typical "hard" labels, y. is a one-hot vector By.."
Accumulated influence and demonstration MNIST. Proposition I describes how the update of x changes the model's prediction on x, for each learning step. Since a real model updates its parameters for many steps, it is important to ask about accumulation of these per-step influences over time. We start by analyzing a simple example of training a LeNet on the MNIST dataset (LeCun et al. 1998).
training example (Xu: Y., = e.). The residual term C (X..Ya y..) )) is then represented by the red arrows, which all start from wg= (y] x..) and point to e4. We can then ask how the model's predictions this update. As in Figure 1-(b), for an x, in the same class with x, (i.e., the identical case), the predicted probability of this correct label is "pulled up" by this update, as expected. On the other hand, if this x, is similar to x (i.c.. ||A*|[ is reasonably large) but comes from another class, then the predicted probability on x. 's class (not the correct label of xx) would be "pulled up." as in the second panel of Figure 1-(b). Last, for examples that look dissimilar to x, (small [[)*)]]=)) this update will not change the model's 's prediction on x much as in the bottom panel in Figure 1-(b). The interactions among the updates of different inputs then form an interesting pattern for the learned predictions. As illustrated in Figure 1-(c), when making predictions on images coming from class 4, the model tends to assign higher confidence on class 9. That because the examples in class 9 average look more similar to class 4 than examples in other classes. Hence the update of examples in classes 4 and 9 will reinforce their mutual influence and lead to a bump in their predictions.
See Figure 1-(a), where the network The is updating its parameters using the loss calculated
different x., change after
To verify this, we plot the average of IT x) for from each of the classes in Figure 1-(d). The values of some off off-diagonal patches are significantly higher than others, which means the examples in those classes look more similar, like 4 and 9, 5 and and 5, etc.
3

---

Published as a conference paper ICLR 2025
3 LEARNING DYNAMICS OF LLM'S FINETUNING
Although learning dynamics have been applied to many deep learning systems, extending this framework to LLM finetuning is non-trivial. The first problem the high dimensionality and the sequence nature of both the input and output signals. The nigh-dimensional property makes it hard to observe the model's output, and the sequential nature makes the distributions on different tokens mutually dependent, which is more complicated than standard multi-label classification problem considered by most previous work. Furthermore, as there are many different algorithms for LLM finetuning SFT (Wei et al. 2022), RLHF (Ouyang et al. 2022). DPO (Rafailov al. 2023), etc. analyzing them under uniform framework is challenging. Finally, compared with the training-frome from-scratch scenario, where a roughly uniform distribution over all possible outputs is usually assumed, LLMs' finetuning dynamics heavily rely on the pretrained base model, which could make the analysis harder. For example, the pretrained model usually assigns little probability mass to unlikely tokens, which good for most applications but leads risk of the "squeezing effect" we show later. We now tackle these problems and propose a unified framework for different finetuning methods.
3.1 PER-STEP DECOMPOSITION OF THE SFT LOsS
The typical loss function used during supervised finetuning is the negative log-likelihood (NLL) of a given completion y 19'" "y......y". E V", conditioned the prompt x,:
(x. ytt)
log mly v/ y2 xx)
Σ wtt log 7(y Xus 21)-
(4)
Note that compared with the multi-label classification problem discussed before, where the joint distribution
of all labels can be factorized as Ily x) = II (M) x), the sequential nature of language modeling makes the analysis more complicated, because we must have r(y x) = I ((n y,)z))) To solve this
problem, we can merge this factorization into the definition of the backbone ha while keeping the format of
Proposition 1. Specifically, letting x be the concatenation of and y. the prediction of all tokens of is
z z ho (x): : π (y x) = Softmax_colum (z).
Here is V x L matrix where each column contains the logits of the prediction of the /th token. Our he, even though it takes the entire sequence X as its input, will force the model not to refer to the future tokens y>ı when making predictions
the 1-th token, commonly implemented via "causal masking" (proposed in Vaswani et al. (2017), details in Figure 10a of Appendix D). Then, we can calculate (Voz) (X.)le (Vle))((ee)) (Vez; (x.))))) xaa))))) T on each column of and stack them form a V X L tensor K K'(Xo, X.). The calculation of G and A also follows a similar procedure. Thanks to the causal mask implemented in hg. the resulting decomposition is almost identical to that in a multi-label classification problem. Assuming have a response y., of length associated with X., stacked into X.. and y. of length M associated with x., stacked into X.- The change of the model's prediction on the mnth token of yo can be represented as, when Z gradients have bounded norm,
A log ' xx) m
Σ n A' mm IC' X. Xw) (x'(x.))))) 0
(5)
VXM
VXVXM
VXVXL
VXL
where GSFT (x.) = Tar (y X.) Yu. Compared with Proposition 1, the main difference is that the eNTK term also depends on the responses ya and yo: which allows us to answer questions like
For prompt Xu, how does learning the response yt influence the model's belief about response y'
When tracking the model's confidence on different responses given the question xu, learning from ytt will impose strong "upwards" pressure yt, as illustrated in the first panel of Figure 2. At the same time,
4

---

Published as a conference paper ICLR 2025
the confidence of "similar responses" will also be slightly pulled up, like how learning a influences the prediction on 9 in the MNIST example. We will discuss how to understand the "similarity" between two sequences of responses in the next section.
3.2 PER-STEP DECOMPOSITION OF THE DPO Loss
Preference finetuning, which teaches the model to provide responses that align better with human preferences, is also an important phase in LLM finetuning pipelines. Different from the SFT stage above, where the training tells the model "what to do", many preference finetuning methods also teach the model "what not to do," which makes the learning dynamics more complex. For intuition, we start by analyzing a typical method, i.e., DPO (direct preference optimization, RL-free method), under similar framework. Following Rafailov et al. (2023), the loss function of off-policy DPO is
To (y xt) (y xx)
Blog Tge (y Teet
Xx X
(f) - (x.. yU)D
log a (Blog
(6)
where yt and y,, are pre-generated responses, and Tee is the reference model, typically the result of SFT. In the loss function, the Th. terms are also calculated using teacher forcing. Hence we can decompose the learning dynamics for DPO similarly to Equation (5),
log (y Xo)lm =- Σ O (IC" X² Mil99roo. - [CC'(xo Xu ).. + O17²)
Iro+ - 3(1 a) (T x yt) Iiro = B( - a) (m (y xxxxx)= y.))
(7)
where a the margin (i.c., the (-) value) for the I-th token, which represents how well the current policy separates yt and yu compared with the reference policy. Due to the monotonicity of o(-), larger margin leads to larger a, which in turn restrains the strength of Ppoo.. less energy on the examples that are already well separated. We then check the role of B8 which controls the regularizing effect on the KL distance between Tge and 7e in the original RL loss (Rafailov al. 2023). When the margin is negative, larger 3 leads to smaller a and hence provides stronger Opo for the model "eatch up" the separating ability of the reference model faster. But when the model is better and has a positive margin, inereasing 3 will increase and hence create negative influence 3 a). which makes the model update less. This aligns well with the claims of Rafailov et al. (2023): the stronger regularizing effect tends to "drag my back towards The when its predictions deviate from Nee too much. The derivation and the g' functions for other RL-free methods are given in Appendix B.2.2.
other words, Poo.. automatically provides
These analyses make assumptions on where yit and yı come from. Hence framework can be directly extended to on-policy RL-free methods, which often perform better than their off-policy counterparts (S. Guo, B. Zhang, al. 2024; Tajwar et al. 2024). The main difference between these algorithms is how the supervisory responses are generated. Off-policy methods typically use a fixed pre-collected dataset, with y and y. are generated by another LLM or humans. other words, it likely that both the chosen and rejected responses come from the "less likely" region of the model's 's prediction. On-policy responses, though, are more likely to have higher predicted probabilities under this model, as they were sampled from We will show next that imposing large negative pressure on unlikely prediction will lead to unexpected behavior, giving a new explanation for why on- on-policy sampling is important for algorithms with large negative gradients.
3.3 THE SQUEEZING EFFECT CAUSED BY NEGATIVE GRADIENT
As demonstrated by the first two panels in Figure 2, the use of large negative gradients is the main difference between the learning dynamics of SFT and DPO. We will show later that this difference is the key to understanding why the learning curves of SFT and DPO behave differently. For example, Pal et al. (2024),
5

---

Published as a conference paper ICLR 2025
On-poric DPO,
SPIN
SFT
Off-policy DPO, IPO
go (v
Caused squeezing effect
Yi
SUC
y
Yu
yu
36
Figure 2: The updating vector provided by the residual term G of different algorithms. The gray y are responses sampled from ı in an on-policy way. In the second panel, we demonstrate the "squeezing effect" caused by imposing a big negative gradient on "valley" region of distribution. For more details about this counter-intuitive effect, please refer to Section 3.3 and Appendix E. Other panels demonstrate on-policy DPO (and IPO), SPIN (Z. Chen et al. 2024), SPPO (Y. Wu et al. 2024), and SLiC (Y. Zhao al. 2023).
Rafailov et al. (2024), and Tajwar et al. (2024) (and our Figure reported that the confidence of both yt and yu gradually decreases when conducting DPO, while the confidence of yi: rarely drops during SFT. This trend becomes more serious if rggo is finetuned for longer before conducting DPO (reported in Figure 3 of Rafailov et al. (2024) and verified by our Figure 17). Furthermore, we also find that for all the vp (y X we track (various responses similar to yt or y details the next section), none of them increase during the DPO phase. This is different from SFT and quite counter-intuitive:
If everything we observe is becoming less confident, where has the probability mass gone?
To answer this question, we first identify a phenomenon we call the squeezing effect, which occurs when using negative gradients from any model outputting distribution with Softmax output heads, even in simple multi-class logistic regression task. Specifically, in the L = case when we impose a negative gradient on label y. describe the changing of model's predictive distribution gg++1 as follows:
Guarantee: the confidence of yu i.e., ma (y will decrease.
Guarantee: the decreased probability mass is largely "squeezed" into the output which was most confident before the update: if y argmax Ta (y i), then Tge+ (y y") will increase.
Trend: the rich get richer and the poor get poorer: generally, dimensions with high mge tend to increase, and those with low тạz tend decrease.
Trend: peakier wy= squeezes
If the probability is common for a pretrained model, all π (y * y") decrease (only y" is "rich").
concentrates few dimensions in mas which
Trend: smaller Te (ym) exacerbate the squeezing effect: if y. unlikely under Tæ., the probability mass of all other 1 (y * y²) will be more seriously decreased, and the Ty y increases more.
Appendix E illustrates the squeezing effect and analytically proves its existence for logistic regression models, by directly computing kg++1 in different situations. Section 4.2 also experimentally verifies the analysis above in real LLM experiments. Note that in practical settings, where both positive and negative pressures and the auto-regressive nature are strictly considered, the squeezing effect can become more complicated. The differences between the two eNTK terms in Equation (7) also influence the relative strength and direction of these two pressures. Razin et al. (2025) analyze similar problem in a token-level setting, and their conclusions align with ours well. We left more detailed analysis to our future work.
We can now get a high-level overview of the learning dynamics of a typical off-policy DPO algorithm. Since both y and yu are not sampled from the model's 's distribution, y" sometimes can be dissimilar to yt and the y, are likely located in a valley region of the model's prediction. Then its learning dynamics would look like the sketch in the second panel of Figure 2: the confidence almost all y pushed down. At the same time,
6

---

Published as a conference paper at ICLR 2025
all the decreased probability mass is squeezed to y", which might make the model keep generating repeated phrases, as reported by Holtzman et al. (2020). Variants of DPO algorithms often unintentionally mitigate this squeezing effect by constraining the strength of the negative gradient or the positions of y which partially explains their benefits. The last four panels in Figure 2 and Appendix B.2.2 have more details.
4 EXPERIMENTAL VERIFICATIONS
We now verify our analysis in practical settings. We first create the training set Dirain by randomly selecting 5000 examples from the training split of the dataset. We consider two common datasets, Ant Anpropic C-HH (Y. Bai et al. 2022) and SlltraFeedbaco (G. Cui et al. 2023), in all experiments. Each example in Drain contains three components: the prompt (or question) x. the preferred response yt. and the less preferred response y SFT finetunes with x and y*, while DPO uses all three (subscripts of x and are removed for conciseness). We repeat the experiments (Biderman et al. 2023) and Qwen1. 0- 5B/15 1. 8B (J. Bai et al. 2023).
two series of models: pythia-4 0M/ 1B 4B/2 8B
To get more detailed observation of the learning dynamics, by randomly selecting 500 examples from Drain. and generate several typical responses based on the corresponding x, y+, or y (We also study another probing dataset where all come from the test set in an ablation study in the appendix.) Then for each x in Pprob. we can observe how log Ter (y x) gradually changes on different types of y. For example, one extended response type can be a rephrase of yt, an irrelevant response answering another question x', or just randomly generated English sentence with the same number of words with y We explain why we need these extended responses and how they are generated in detail in Appendix D.1. In short, Ppnnh helps us to get more fine-grind inspection of the learning dynamics, which can not only support our analysis above, but also shed more light on how the model's s prediction evolves on the entire ) E TR. VXL, very sparse and huge space.
further create probing dataset D prob
4.1 LEARNING DYNAMICS OF SFT
The main lesson learn from the analysis in Section 3.1 is that learning from yt not only increases the model's 's confidence on yt. but also indirectly "pulls up" responses similar to y with a smaller strength (scaled roughly by [|A*[[]) similar to how learning "4" influences the prediction of "9" in the MNIST example. At the same time, the increase of Th- (yt X..) naturally "pushes down" all A y because the model's predicted probability all responses in V'-sacc must sum to one. The model's behavior on different y is mostly a trade-off off among these pressures. To verify this claim, we finetune the model for several epochs and evaluate the model's prediction on all responses in Porob every 25 updates (with a training batch size of 4, the probing occurs every 100 examples). For each type of response, we average the model's confidence on all 500 examples and report the mean value of their loeelho.
As demonstrated in the first panel of Figure 3, the model's confidence on yt keeps increasing throughout the whole learning process, which is straightforward because the main "pull-up" pressure is imposed directly on yt. However, the behavior of some responses similar to yt is non-trivial. For example, we draw the following types of responses in the same panel, i.e. the less preferred response for the same question (y ). two types of rephrases of y generated by ChatGPT (ytu. and ytru). another preferred response randomly selected from the test set (yitz), even randomly generated English sentence (Ynum) The model's confidence in these responses are all slightly increased at the beginning of training, and then gradually decrease as the training goes on, even though the model never sees them during SFT. This counter-intuitive behavior be well explained by the learning dynamics we discussed before. Since all these examples are "similar" to yi to some extent (at least, they all common "standard English" sentences), their ||A*|] are reasonably large. Then learning y will indirectly increase the model's confidence of these similar y. That is why the corresponding T (y) X..) are slightly increased at the beginning of training. However, the training goes
7

---

Published as a conference paper ICLR 2025
Chisones mitiee
Non man esarece
Wacte y wouee ol Juin raiity
3F1 2P0 parg sener Sppes
Accumbiizee Hiueee
14
Zim
122
-00
Yivo
-3
Yiea
Pur
Mamrer OEGONS
of opote
Winter of spocts
Furrber stects (<30 SED
Figure 3: First three: learning dynamics of SFT different response types. Fourth: SFT 10 epochs then DPO. Last: the accumulated influence when SFT using different y (full results in Appendix C and D).
on, the model's confidence on yu keeps increasing and the update energy, the norm of GEE in Equation (5), gradually decreases. That means the indirect "pull-up" pressures are also diminished accordingly. Then, the "push-down" pressure on all y # y becomes dominant and all the related curves start going down.
To verify the existence of this global "push-down" pressure, observe two types of responses; both have the same number of words as their yr: One is a purely random English word sequence Yond Another is random permutation of all the words in y which is called y yumdd Since both are not natural language, we expect the ||C||| between them and y to be very small, which means learning from y imposes almost no "pull-up" pressure on them; thus the "push-down" pressure will dominate through the whole training procedure. These analyses are well supported by the second panel in Figure 3, which we see these Te (y Xx)) all start from a very small value, and keep decreasing throughout the training.
Another interesting type of responses is Y)zu preferred response for another question xjztu in the training set. For these responses, the model's prediction on Thx (y ( Ix.) will be kept influenced by two "pull-up" pressures: one is from learning ( another is from learning (x y where the latter might be even stronger as the gradient is directly calculated by observing yo That explains why we see the confidence on you keeps increasing with a smaller rate compared with y the third panel. Because the "pull-up" pressure is always strong enough to counter the "push-down" one. These observations provide us with a unique explanation of why specific types of hallucinations are amplified after SFT. Specifically, the increase of ue. (y j (x.) means if we ask the model to answer a question X,, it might provide a response from (or partially from) another unrelated question xzzú in the training set.
Last, to further explore the "similarity" between different responses from the model's perspective. we SFT the model using more types of responses and observe how To X.)) changes accordingly. The results are demonstrated in Figure 3, where the blue and orange colors represent the positive and negative influence respectively. The x-axis the updating response while the y-axis denotes the observing response. Hence the first column resembles how different [xw;y'] changes when we SFT the model using [xui y.]. One interesting finding that all responses generated by ShatGPT are considered very similar to each other, regardless of how semantically different they are. Probably, LLM has its preferred idioms or phrases, which could be considered as a type of "fingerprint". We left this interesting problem for our future work.
4.2 LEARNING DYNAMICS OF OFF-POLICY DPO
To verify our framework also explains the model's behavior in preference tuning, we conduct similar experiments for DPO. Recall the residual term Gi introduces a pair of arrows on both ytt and yu with different directions. To show how these two pressures influence the model, we check two types of rephrases of y or y (y Egtss y Yegat Ygpts+ and y gott used in the previous experiment). See the three curves in the first panel in Figure 4, where the two rephrases decrease at a similar speed, faster than the decay of y That because the upward pressure is directly imposed on y rather than these rephrases. Similarly, in the second
8

---

Published as a conference paper ICLR 2025
Chesen v.s. its rephrases
Rejected v.s. rephrases
Rojected vs. its raphrases
Choser V.S. rejected
Where the pro bability mass gone?
-120
-12
Auvege bopprababllty
-147
-197
All other responses
-15
Yests
Yu
Yigts
-102
-122
Vists
Y
Yesr
-120
-22
Yeuu
-24
Yigu
-90
Y5
Argmax
Number spochs
Number epochs
Number
Number
Number spocts
Figure 4: Learning dynamics of off-policy DPO. The last panel verifies the existence of the squeezing effect.
panel, observe that y. decays faster than its rephrases, because Ppo directly imposes negative pressure on y., Then in the third panel, we find the rephrases of yt consistently decay slower than those of y although none of them ever occur during training. That is because these responses
close to Yı y, in Y, which means their || is relatively large. Hence the pressures imposed on yī and y. also introduce a non-negligible influence on them. Last, in the fourth panel, the margin Tg. (y (x)) -ep (yn Ixu) keeps increasing, which means the model is gaining the ability to separate y and yu as the training goes on.
Although Hro directly imposes a "pull-up" pressure on y the value of πT= (y (X.) does not increase a lot as it does in SFT. The downward arrow on yu indeed introduces a "push-down" pressure on responses that are similar to yu. but the influence is unlikely to be that strong (it will be weakened by ||A* ()) to make the confidence on almost every observing responses decrease so fast, as demonstrated in the last panel of Figure 3 where we use similar 77 for both SFT and DPO. Then, where has the probability mass gone during DPO? The key to answering this question is the squeezing effect discussed in Section 3.3: since the big negative gradient is imposed on Yu which is at this point probably a region of low Tg² Xx,), the confidence of most y will be decreased and ae (y"l (x.) will increase very fast.
To verify this, we report the log-likelihood of y chosen by greedy decoding: each token is chosen by maximizing the conditional probability given [y] in real-time, where y a sub-sequence of y As illustrated by the last panel of Figure 4, the confidence of this "teacher forcing" greedy y increases very fast (from -13 to -63), which is even faster than the increase of T&. (yt Ixx) during SFT (from-130 to -90), within 8 epochs. However, the tokens with the highest confidence do not necessarily form a preferred response: it will reinforce the prior bias in 0 This could be reasonable explanation of the "degeneration" reported in recent work (e.g. Holtzman et al. 2020): as т& becomes more peaky at its most confident predictions, it is casier to sample sequences with repeated phrases. Note that such behavior could also be understood as a special type of self-bias amplifying (Ren et al. 2024), which would bring more serious consequences if it is combined with a multiple-generation aelf-improving algorithm, e.g., self-reward (Yuan et al. 2024), iterative DPO (Xiong et al. 2024), etc.
summary, the behaviors of different types of responses all match our analyses well. More subtle trends of different responses support our story well (both for SFT and DuO).Due to space constraints, we explain these (and the full results on other models and datasets) in Appendix D.
4.3 MITIGATING THE SQUEEZING EFFECT BY AUGMENTING THE TRAINING SET FOR SFT
Since the "squeezing effect" caused by the big negative gradient on unlikely predictions can damage the model's performance during DPO, we can first train the model on both ix;; and [x.:Y.] during the SFT stage (making the negative response more likely), and then run the usual DPO. Following the analysis above, we can expect during this new SET stage, the region of those responses similar to y or y will be "pulled up" simultaneously. This what we want because in many cases, both y and y are reasonably good responses for the question x,; the new SFT design hence helps to pull up a larger region that contains more suitable responses compared with the baseline SFT. After that, the "push-down" pressure imposed during DPO can
9

---

Published as conference paper ICLR 2025
Chosen
Tie sctedtedd...
Other
Arginax
AI reshrases
F2 pesk,nar
Nomhn
Dinn se oPo
Number spochs
Number of epocts
Number of epochs
Number of epachs
Figure 5: Learning dynamics of the baseline and the proposed method with training data extension. Key trends to observe: 1.) Baseline and the extend method have similar behavior on y during SFT; 2.) The extend method considerably increase yu during SFT; The squeezing effect of the extend method is weaker (all other responses decay slower and the confidence on the "greedy-decoding" response increases slower).
efficiently decrease the model's confidence on yu and its similar responses. Since y. is no longer so unlikely before DPO, the squeezing effect should not be as strong in the baseline procedure.
We call our training pipeline "extend" and compare its learning dynam- ies with the baseline setting in Figure 5. It is clear that the squeezing effect is mitigated, because the confidence of other responses all decays slower during DPO, and we also observe a big drop the greedy- decoding response when DPO starts. To further show that mitigating the squeezing effect indeed brings benefits, we compare the responses generated by models trained using different methods by feeding them to Chat GPT and Cl Claude3. Specifically, we first SFT the model for two epochs using two methods discussed above and call the resulting policy network Thase and Textend- Then, we conduct identical DPO training both Thase and Textend for several epochs. The win rate of the proposed method against the baseline one is provided in Table 1. It is clear that before DPO, Thase is better, because Textend is explicitly trained on those y However, the Textend performs better after DPO several epochs since the squeezing effect efficiently mitigated. Please refer to Appendix F for more details. In the future, this simple method inspired by our analysis could be further improved by introducing more responses, e.g., rephrases y etc., during both stages, and also by combining with many existing RL-free methods we mentioned before.
Table 1: Win-rate against baseline.
DPO Ep.
ChatGPT
Claude
0.4729
0.4679
0.6518
0.5151
0.6928
0.6045
0.6667
0.5432
5 CONCLUSION
Learning dynamics, which depict how the model's prediction changes when it learns new examples, provide a powerful tool to analyze the behavior of models trained with gradient descent. To better utilize this tool the context of LLM finetuning, we first derive the step-wise decomposition of LLM finetuning for various common algorithms. Then, we propose unified framework for understanding LLM predictions' behaviors across different finetuning methods. The proposed analysis successfully explains various phenomena during LLM's instruction tuning and preference tuning, some of them are quite counter-intune intuitive. We also shed light how specific hallucinations 2024), and where the improvements of some new RL-free algorithms come from compared with the vanilla off-policy DPO. The analysis of the squeezing effect also has the potential be applied other deep learning systems which apply big negative gradients to already-unlikely outcomes. Finally, inspired by this analysis, we propose simple (but counter-intuitive method that effective improving the alignment of models.
introduced in the SFT stage, previously observed (Gekhman et al.
10

---

Published as conference paper ICLR 2025
ACKNOWLEDGEMENTS
This research was enabled in part by support provided by the Canada CIFAR AI Chairs program, WestGrid, and Compute Canada. We thank Shangmin Guo, Noam Razin, Wonho Bae, and Hamed Shirzad for their valuable discussions and feedback. We also appreciate the constructive comments from the anonymous reviewers, which helped improve this work.
REFERENCES
Sanjeev Arora, Simon S. Du, Wei Hu, Zhiyuan Li, Ruslan Salakhutdinov, and Ruosong Wang (2019). "On Exact Computation with an Infinitely Wide Neural Net." NewrlPS. arXiv: 1904. 11955.
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello (2024). "A general theoretical paradigm to understand learning from human preferences." International Conference on Artificial Intelligence and Statistics. PMLR, pages 4447-4455.
Jinze Bai, Shuai Bai, and et.al (2023). "Qwen Technical Report." " Xivi:: 2309.1660
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. (2022). "Training a helpful and harmless assistant with reinforcement learning from human feedback." " arXiv: 2204. 05862.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O' Brien,, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, al. (2023). "Pythia: A suite for analyzing large language models across training and scaling." ICML. PMLR, pages 2397-2430.
Massimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau, and Laurent Charlin (2020). "Language GANs Falling Short." International Conference on Learning Representations.
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu (2024). "Self-play fine-tuning converts weak language models to strong language models." arXiv: 2401. 01335.
Leshem Choshen, Lior Fox, Zohar Aizenbud, and Omri Abend (2020). "On the Weaknesses of Reinforcement Learning for Neural Machine Translation." International Conference on Learning Representations.
Paul FF Christiano, Jan Leike, Tom Brown, Miljan Martie, Shane Legg, and Dario Amodei (2017). "Deep reinforcement learning from human preferences." NewrIPS.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun (2023). "Ultraeeedback: Boosting Language Models with High-quality Feedback." arXiv: 2310.01377 [cs. CLL.
Dan Feldman (2020). "Introduction to core-sets: an updated survey." arXiv: 2011 2011.09384.
09384.
Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, and Jonathan Herzig (2024). "Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?" arXiv: 2405.05904.
Daya Guo, Dejian Yang, Haowei Zhang, Jumxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. (2025). "Deepseek-rl: Incentivizing reasoning capability llms via reinforcement learning." arXiv preprint
Shangmin Guo, Yi Ren, Stefano V Albrecht, and Kenny Smith (2024). "IpNTK: Better Generalisation with Less Data via Sample Interaction During Learning." ICLR.
11

---

Published as conference paper ICLR 2025
Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. (2024). "Direct language model alignment from online AI feedback." arXiv: 2402.04792.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean (2015). "Distilling the knowledge in neural network." arXiv:
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi (2020). "The Curious Case of Neural Text Degeneration." ICLR.
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong. Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et (2023). "A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions." arXiv: 2311.05232.
Arthur Jacot, Franck Gabriel, and Clément Hongler (2018). "Neural tangent kernel: Convergence and generalization in neural networks." NeurlPS.
Haozhe Ji, Cheng Lu, Yilin Niu, Pei Ke, Hongning Wang. Jun Zhu, Jie Tang, and Minlie Huang (2024). "Towards Efficient and Exact Optimization of Language Model Alignment." arXiv: 2402. 00856.
Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Perey Liang (2022). "Fine- Tuning can Distort Pretrained Features and Underperform Out-of-Distribution ICLR.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner (1998). "Gradient-based learning applied document recognition." Proceedings of the IEEE 86.11, pages 2278-2324.
Sheng Liu, Jonathan Niles-Weed, Narges Razavian, and Carlos Fernandez-Granda (2020). "Farly-learning regularization prevents memorization of noisy labels." NeurIPS.
Mohamad Amin Mohamadi, Wonho Bae, and Danica J. Sutherland (2022). "Making Look-Ahead Active Learning Strategies Feasible with Neural Tangent Kernels." NeurIPS. arXiv: 2206. 12569.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. (2022). "Training language models to follow instruc- tions with human feedback." NeurlPS, pages 17730-27744.
Arka Pal. Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White (2024). "Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive." arXiv: 2402. 3228.
Core Francisco Park, Maya Okawa, Andrew Lee, Ekdeep Singh Lubana, and Hidenori Tanaka (2024). "Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Space." The Thirty-eighth Annual Conference on Neural Information Processing Systems.
Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan (2020). "Estimating training data influence by tracing gradient descent." NeurIPS.
Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn (2024). "From to Q": Your Language Model is Secretly a Q-Function." arXiv: 2404. 12358.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn (2023). "Direct preference optimization: Your language model is secretly a reward model." NeurIPS.
12

---

Published as a conference paper ICLR 2025
Noam Razin, Sadhika Malladi, Adithya Bhaskar, Danqi Chen, Sanjeev Arora, and Boris Hanin (2025). "Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization." The Thirteenth International Conference on Learning Representations.
Yi Ren, Shangmin Guo, Wonho Bae, and Danica J. Sutherland (2023). "How to prepare your task head for finetuning." " ICLR.
Yi Ren, Shangmin Guo, Matthieu Labeau, Shay B. Cohen, and Simon Kirby (2020). "Compositional languages emerge in a neural iterated learning model." ICLR.
Yi Ren, Shangmin Guo, Linlu Qiu, Bailin Wang, and Danica J Sutherland (2024). "Bias Amplification in Language Model Evolution: An Iterated Learning Perspective." NeurIPS.
Yi Ren, Shangmin Guo, and Danica J. Sutherland (2022). "Better Supervisory Signals by Observing Learning Paths." ICL.R
Yi Ren, Samuel Lavoie, Michael Galkin, Danica J Sutherland, and Aaron C Courville (2023). "Improving compositional generalization using iterated learning and simplicial embeddings." NeurIPS.
Yi Ren and Danica J. Sutherland (2024). "Understanding Simplicity Bias towards Compositional Mappings via Learning Dynamics." NeurIPS 2024 Workshop on Compositional Learning: Perspectives, Methods, and Paths Forward.
Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo (2024). "Are emergent abilities of large language models mirage?" Advances in Neural Information Processing Systems 36.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov (2017). "Proximal policy optimization algorithms." arXiv preprint arXiv: 1707.06777
Burr Settles (2009).** "Active learning literature survey."
Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Stefano Ermon, Chelsea Finn, and Aviral Kumar (2024). "Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data." arXiv: 2404. 14367 [Cs.LG].
Yunhao Tang, Daniel Zhaohan Guo, Zeyu Zheng, Daniele Calandriello, Yuan Cao, Eugene Tarassov, Rémi Munos, Bernardo Ávila Pires, Michal Valko, Yong Cheng, al. (2024). "Understanding the performance gap between online and offline alignment algorithms." arXiv preprint 2rXiv:2405.08448,
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin (2017). "Attention Is All You Need." NeurIPS. arXiv: 1 1706. 03762.
Tongzhou Wang, Jun- Yan Zhu, Antonio Torralba, and Alexei A Efros (2018). "Dataset distillation." arXiv: 1811 10959.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe Le (2022). "Finetuned Language Models are Zero-Shot Learners." ICLR.
Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu (2024). "Self-play preference optimization for language model alignment." arXiv: 2405 00675.
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen (2024). "Less: Selecting influential data for targeted instruction tuiig." arXiv: 2402. 04333.
13

---

Published as conference paper ICLR 2025
Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang (2024). "Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint." ICML.
Zhaorui Yang, Qian Liu, Tianyu Pang. Han Wang, Haozhe Feng. Minfeng Zhu. and Wei Chen (2024). "Self-Distillation Bridges Distribution Gap in Language Model Tinenn. arXiv: 2402 13669.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston (2024). "Self-rewarding language models." " arXiv: 1441111022
Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei (2024). "Negative preference optimization: From catastrophic collapse effective unlearning." arXiv preprint arXiv:2404.05866
Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. (2023). "Siren's 's song the Al ocean: survey on hallucination in large language models." arXiv preprint arXiv:2309.01219
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter Liu (2023). "Slic-hf: Sequence likelihood calibration with human feedback." arXiv: 2305. 10425.
14

---

Published as a conference paper ICLR 2025
A MORE RELATED WORKS
A.1 MORE ABOUT LEARNING DYNAMICS
Instead of emphasizing the model's 's convergence status and guarantees, learning dynamics focus more on its relative behavior during training. Since many LLMs exhibit "emergent ability" during training in terms of the amount of training data or inference time (Schaeffer et al. 2024), like the "aha moment" mentioned in D. Guo et al. (2025), hence believe the analysis of learning dynamics can provide novel perspective on understanding the LLM system.
Beyond their application LLMs, learning dynamics are widely utilized in analyzing various machine learning problems. For example, if we consider x from the training set, and x, from the test set, this form of learning dynamics provides a new perspective on generalization: the model generalizes better if the loss of fx(x.) keeps decreasing when it learns from X By studying the influence of different X at different stages during supervised learning, Ren et al. (2022) explain a "zigzag" pattern of the learning path, which sheds light on why the model can spontaneously pursue better supervisory signals and correct noisy labels in the early stage of training (see also S. Liu et al. 2020). Kumar et al. (2022) and Ren, S. Guo, et al. (2023) apply learning dynamies explain why directly finetuning well-trained backbone with randomly initialized task head might harm the out-o--distribution generalization ability. Ren et al. (2020), Ren, Lavoie, et al. (2023), and Ren and Sutherland (2024) also explains where the simplicity bias favoring compositional representations comes from during knowledge distillation (Hinton et al. 2015), providing a new perspective of understanding why successive knowledge transferring can improve the model's systematic generalization ability.
Besides explaining the model's hehavior, learning dynamics is also helpful for evaluating the quality or the effectiveness of different training samples. For example, Pruthi et al. (2020) propose quantitative metric called TracIn to compute the influence of a training example on the predictions made by the model. This metric is then applied by Xia al. (2024) search for the most influential examples in LLM instruction finetuning. By expanding Equation (1) in the neural tangent kernel (NTK) regime, S. Guo, Ren, et al. (2024) propose a metric called 1pNTK to measure the relative difficulty among different training samples. These metrics and analyses inspired by learning dynamics are expected to be helpful in many related fields, like coreset selection (Feldman 2020), active learning (Settles 2009) (see, e.g., Mohamadi et al. 2022), and dataset distillation (T. Wang et al. 2018).
A.2 MORE ABOUT LLM'S FINETUNING
In this paper, we broadly define finetuning as any in-weight learning on top of a pretrained base model, including supervised finetuning (SFT), direct policy optimization (DPO, Rafailov et al. 2023) and its variants, etc. Since the analysis throughout this paper relies on the "teacher forcing" mechanism and the relatively stable eNTK assumption, our framework cannot be directly applied to algorithms with token-wise supervision like reinforcement learning with human feedback (RLHF, Ouyang et al. 2022) and proximal policy optimization (PPO, Schulman et al. 2017)!. We leave the study of the token-wise learning dynamics, which aligns better with the "squeezing effect" in real settings, to future work.
We also identify several related works that report similar observations on the phenomena discussed in this paper. For example, Gekhman et al. (2024) and Yue Zhang et al. (2023) mentioned that learning new facts during SFT tends to make the model hallucinate more, which aligns with our finding that the model tends to use y y when answering question i. Holtzman al. (2020) related the peakiness of the model's distribution to LLM's "repeater phenomena", which also indirectly supports our claims well: more DPO leads to more serious squeezing effect, hence the model's prediction becomes peakier on most tokens, which makes the aforementioned phenomena more common.
IIf the implementation indeed applies the teacher forcing mechanism, our analysis still works.
15

---

Published as a conference paper ICLR 2025
Furthermore, the "confidence decaying on y attracts more attention the community, because is quite counter-intuitive and the vanilla off- policy DPO algorithm works reasonably well in most cases. Many related works study this phenomenon by analyzing the major discrepancy between off- off-policy DPO and PPO, i.e. where the samples used to train the model comes from, e.g., S. Guo, B. Zhang, et al. (2024), Rafailov et al. (2024), and Tang et al. (2024). They showed that when the responses are off-policy sampled, the learning process may fail to benefit from the contrastive information in the data. In other words, we should be more careful when working on the "valley" region of the model's distribution. Other works try to analyze this problem by inspecting the token-level influence between responses. For example, Pal al. (2024) assumes yt and yu are identical expect one token. Under this assumption, the model's confidence of yit after the identical token is guaranteed to decrease. They propose solution by significantly enhancing the learning rate (roughly x50 larger when their À = 50) of the positive part when detecting y., located in a low-confidence region. Razin et al. (2025) takes the similarity between the hidden embeddings and the geometry of the readout layer of different responses into account. Most of the conclusions of their paper align with ours well. The main discrepancy lies in the squeezing effect part, which we will discuss our future work (they do not contradict each other, but need a more detailed analysis to understand the whole story).
A.3 BENIGN AND HARMFUL NEGATIVE GRADIENT
The "squeezing effect" can negatively impact our analysis when it is strongly imposed in a valley region of the model. However, well-regulated negative gradient is both beneficial and commonly observed in many deep-learning systems. For example, it is common in many "machine unlearning" algorithms, e.g., in Ruiqi Zhang et al. (2024). Moreover, even in the field of LLM finetuning, we can find many mechanisms in different popular algorithms that can mitigate this effect. For example, the typical learning rate of DPO is usually smaller than that used in SFT, which unintentionally mitigates the harmful squeezing effect. The on-policy counterpart of the DPO-like algorithms is shown to perform better than their off-policy counterparts, which also supports our claims. Furthermore, we find the PPO loss automatically avoids imposing a big negative gradient (when its Å, is negative) on the valley region (when its my is small).
On the other hand, the effect that negative gradients make the model's distribution peakier is independently reported in many related works. For example, Equation 1 in Caccia et al. (2020) shows that we are minimizing a negative thing in a standard GAN loss, which might explain why peakiness occurs. Furthermore, in Table I and Table 2 of Choshen et al. (2020), we see the peakiness (measured by Aprop Apmode) of the "PG-average" method is stronger than the standard PG method. Note that the "PG-average" method will map reward ranging from 0 to 1 to centered one ranging from -0,5 to 0.5. Since the negative reward can introduce a negative gradient, the peakiness increases.
B PROOF OF PROPOSITIONS AND RESIDUAL TERM FOR DIFFERENT LOSSES
B.1 PROOF OF PROPOSITION
Proposition 1. Let π = Softmax(z) and = ho(x). The one-step learning dynamics decompose as
A log (y Xx) -1 A'(x)) 10' (x., X G (x) + +(r21Voz((x)113)
(3)
Vx1
VXV
VXV
VX1
where A Axxx = V. log Tg. (x.) =I I- (xo). K. Xx) = ((o(xx))l:(((o2z())) is the empirical
neural tangent kernel of the logit network and G (xxxx Yu) V,L(x,, Y.)lz
16

---

Published as a conference paper ICLR 2025
Proof. Suppose we want to observe the model's prediction on an "observing example" Xa. Starting from Equation (2), we first approximate log xf+1(y xo) using first-order Taylor expansion (we use x² to represent
Tee interchangeably for notation conciseness):
log x xx) log x (y x.) (Vlog ny Xo); 0 0*) -
Then, assuming the model updates its parameters using SGD calculated by "updating example" (Xa; Yu). we can rearrange the terms in the above equation to get the following expression:
A log o (y xx) log π y xx) log x. (y Xx) Vo log n' xx) (0'+1 l*)
0110011-0011
Vx1
Vx1
Vxd
dx1
where d is the number of parameters of the model. To evaluate the leading term, we plug in the definition of SGD and repeatedly use the chain rule:
V log n' (y Ko)lo=
(0'+1 0²)
Vz log T (x))222
Voz (xo)lor ) Vol(xx)lor)
Vxd
dx1
VXV
Vxd
1xd
v log (x,)
Voz Voz' (xa) (-11 (1/VVLL(xx))) Vaz
(x,) (xx))or
VXV
Vxd
1XV
Vxd
Vzlog T'' (xo)les
Voz (xo)le
(Voz' (2((xx))z)
VxV
Vxd
Vx1
= (x. )x' (x,; x Yx)
(8)
For the higher-order term, using as above that
g++1 8r = - (x (x..))
and noting that, since the residual term G is usually bounded (and the practical algorithms will also use gradient clip to avoid too large gradient), we have that
0 8* || = Or² (V
(xw)l:)) (Vozz(xx)lo))) 1lap ||g (x., 1)pp) = 01
In the decomposition, using { TV to represent the model's prediction on different dimensions, we can write our A' as:
T
-TT 1 - 12
1 22
(9)
TV
TV
TT
The second term in this decomposition, (X.: x,), ) is the product of gradients at x, and X,- Intuitively, if their gradients have similar directions, the Frobenius norm of this matrix is large, and vice versa. This matrix is known as the empirical neural tangent kernel, and it can change through the course of training as the network's 's notion of "similarityy" evolves. For appropriately initialized very wide networks trained with very small learning rates, K remains almost constant during the course of training, the kernel it converges to is known the neural tangent kernel (Arora et al. 2019; Jacot et al. 2018). Note that the assumption that IC' (x,,,. x,) is unchanged (usually used in theoretical analysis) might be too strong in the LLM's finetuning. Hence as stated in the main context, our qualitative analysis only assumes that "during the training, the relative influence of learning x,, on all other different x, is relatively stable".. We will validate this assumption using experiments in Appendix C.
NNte that this proposition assumes L = 1. For L > different Equation (3). The V matrix log x¹ can then be achieved by stacking them.
1 case, we will have multiple task heads which leads L
17

---

Published as a conference paper ICLR 2025
B.2 RESIDUAL TERM FOR DIFFERENT LLM FINETUNING ALGORITHMS
As stated in Section 3, one of the conundrums of decomposing the learning dynamics of LLM is its auto- regression nature of the output sequence. Different from the multi-label classification problem, where y for different I is independently generated as long the shared network is fixed, the yr for the LLM's output depends on Y<r. which is usually sampled from the model's prediction iteratively. However, in most of the finetuning cases where the supervisory signal yu is given, the model will apply the so-called "teacher forcing" mechanism when calculating the predicting probabilities. In other words, when generating the oulput of each yı. the Yeı is given rather than sampled on-policy. This mechanism makes it possible for us to define xx; y) and hence merge the auto-regressive nature of the sequence prediction into the shared K. (Xo: X.). After this step, the decomposition of L.M's finetuning learning dynamies then becomes similar to multi-label classification task.
B.2.1 INSTRUCTION FINETUNING USING AUTO-REGRESSINN L.SS (SFT)
Here we derive the residual term, i.c. G for different algorithms in LLM's finetuning. We first rewrite Equation (5) here:
A log (y Xo)lm
Σ nl A (x.) Im (x:
O(1²).
VxM
VxVxM VXVXL
VXL
where m {I...., M); L {I,...,L, and G' v=((x,)le is a V L matrix. As the auto- regression nature of the SFT loss is already encoded in the causal mask used in ha, as demonstrated in Figure 10a. the columns in (X..) are independent of each other, which can separately calculated. Plus, the summation over i can also be achieved by left-multiplying a length-L all-one vector 1. Specifically, the SFT loss for each / is:
[LrT((X.)l = log (w= = vt Xx.) = -
T- log ग (yr xx.) = -e b
log (So (Softmax(z:))
where vit is for the 1-th dimension of yt: The gradient of L on z can be then calculated as:
(GSPT(X.)]1 = Vz, (x.))
V V
V ग VXV
1XV
Vx1
=- (e mm)
Vz x Xx)
e
(10)
where 2 is clement-wist division.
To calculate the equation above, we first recall the NLL loss of the 1-th token is [CsFT] e L = - TT) wit) = ./- log rT, where = Softmax(z). Then, VzL = V2T. For each dimension of V,C,, we
ixV 1XV VXv
have ec 0f n. # yt and Oc - if vt. By writing it in vector form, have v.C |. -e @ T) V2T. For VzH, we have:
T - mT)
T22
TV
T2
T2272
TT
(T TV
722
TV
18

---

Published as conference paper ICLR 2025
Combining this matrix and the 1 V vector (ey! 2 TT) where the only non-zero term at the k k= y/" position. So, left multiplying by this vector is actually first selecting the k-th row of VzT, and then multiplying to it. In summary, have:
Fi.
Tik
J,L- T T - TT ..)...., =KTV]" [TT, ................................ Thh 1,..., TV = π ex
Tk
By stacking the terms with different I [L], can get
(xr( = VxLSFT(Xu)le Tor (y Xx) y
(11)
B.2.2 DIFFERENT PREFERENCE FINETUNING ALGORITHMS
Direct Preference Optimization (DPO, Rafailov et al. (2023)) is usually considered the first RL-free alignment algorithm for preference finetuning. Different from the standard RLHF (reinforcement learning with human feedback (Christiano al. 2017)), the training of off- off-policy DPO is more similar to SFT, where the model keeps learning from pre-generated preference dataset. Hence, we start from DPO to analyze the learning dynamies of different preference finetuning algorithms (the on-policy versions of these algorithms could also be explained using the proposed framework).
Following Rafailov et al. 2023, the training loss of DPO is:
To (y
X x
Tg. (y
Xxi
(0) = (x. yTN logo a Boog
-
Blog
(12)
Tea (y
ref
Before calculating the residual term DPo- we need to re-calculate the learning dynamics decomposi- tion, because the loss term now depends on both The (y two different Z terms. Specifically, we define np (y X = Softmax Softmaxxcclumn(22 column(z*) and To. (y
| Xx) and Tp. (y Xv ). which involves
X.:)
Softmax_ Softmax_column( column(z where zt (x( and
hg(Xm) respectively (x: [x,;y.
and X
y..). [
Then, starting from L 1, the decomposition for the DPO loss (similar to Equation (8)
for SFT) could be written as:
Ve log n (x.) A0'
(Z 1ooo''(X Vez (x.)lor) ) Vez'(x))or
17 Vg y yu )oo T
Vxd
dxl
VxV
Vxd
1xd
Vzlog rt' 2z Voz'
-n V (2t Slz
Voz (x Voz (x. lee
VxV
1x2V
21
Vologno r' (xo)
Vozz (xo)lor
([Voz* (x.)
V9z (x
lo.)
Vitto Clet) (le))
VxV
Vxd
dx2v
2Vx1
= =-1A' [K' x.t ); )x (x.r X. (Vistz-111z))
-nn (K* (x. Xx xt) Popro. 10' 1C'( X Ibpo.
(13)
19

---

Published as conference paper ICLR 2025
where [-- are concatenation of two vectors matrices, Boo. (x:) & Vzi CDpO. and (x.) - V, LDPO. To calculate the residual terms, we decompose the loss into:
(xro yt,, Vyu
() - -oog(a)
_ (bb)
_ (log Th. (y Xx log To: (Y Xxi)
3 (CSFT (xtr(xx))))) CsFT(Xu))
_ (log Tr xiit) - log (y Xx.)
(14)
where e. is not function of 0. Using the chain rule, the 1-th column of the residual term Ooo+ can be calculated (the calculate of Pro. is similar):
Ibo+ @LDro da
Da Öb Vz bl2
1
aa(1 0) V bilet
8(1 a) (Te (y
X Yu
By stacking values with different we can get the residual term of DPO as
Ii = = - a) (Ta. (y
x
y
IDP = =,((1 - a) (TT xw - yyu)
Te (y
X
Trrrly x
= Bog
Blog
(15)
Xu
Tert
Xw
Similarly, we can calculate the residual terms for other off-policy preference optimization methods, like Identity-preference Optimization (IPO (Azar et al. 2024)):
[[[o8
TAi (y
X
Tag (y
Xx )
1
Luo -E
-X (x. WDD
log
(16)
T
X
Tret Ya
Xu
2,3
Tp (y
X
T +
X
1
GLoo = Sro a log
log
(17)
T (y
Xu
neet Yvu
Xa
2,3
For the Sequence Likelihood Calibration (SLiC (Y. Zhao et al. 2023)), we have:
Toa (y X
LSLiC - (x yt DD max 0,, -log
B -..log mg. (Yret Xxet)
(18)
Teg (y
X
[8 JDD mmxx [0, 8 + LSFT - LSFr(x..) )] + 3LL (Xret)
(19)
(5 llo
n (y
SSuC+ Ioo. (TT Xx.) yeet) ;
0
(20)
T (y
In summary, these RL-free algorithms all relate to the SFT loss extent. For the DPO and IPO loss, the directions of the updating signals are identical. A scalar controls the strength of this update, which usually correlated with the confidence gap between the model's current confidence y and y i.e.,
20

---

Published as conference paper ICLR 2025
Gap(TT:) ) - log For (x.
Fat (yū lxx) Generally, larger this value leads to bigger making the norm of ge smaller.
In other words, we see "regularizing" effect in this term, where the model should not make Gap( too large. The SLiC loss can be considered
combination of SFT adaptation and preference adaptation. Similarly, we can also see a hard version of the regularization effect mentioned above. If GTp( > 5, the indicator function will become zero, and the model stops pushing क (y and T away when already separates y and y well.
Recently, authors of (Y. Wu et al. 2024) propose another interesting self-play alignment algorithm called SPPO, which further improves the alignment performance on top of many on-policy DPO methods. Our framework could also give an interesting explanation why this method works so well. Specifically, the loss function of SPPO can be written
ESPOO = -E E
(x waND
(10g
Toi (y
X
2
2
+
+logg
log
Tel (Yu
Xu
+
27
(21)
Teel Yi
Xi
Teell y
Xu
Tgc (y
Xx Xii
Tg. (y
Xx )
SSPO 2 log
(Tg: - -yt) +t)++3 +2
log
+
(mT - -yu). 2
(22)
Teef yu
2
Teef Yo
X
This loss looks similar to the IPO one, but the main difference between SPPO and other methods (e.g.. DPO, KTO, IPO, SPIN, etc.) is that there is negative sign in front of ma (y Xx or To: (y X ). From its residual term SSPPO it more convenient understand this algorithm imposing two positive vectors on both y and yu but the former has longer norm, as illustrated in Figure 2. By doing so, the big negative
gradient no longer exists, and so does the squeezing effect. That partly why this method more stable and performs better.
C THE "RELATIVE STABLE" ENTK ASSUMPTION
We use this appendix to verify the core assumption of our analysis during the training, the relative influence of learning ve on all other different v, is relatively stable on both MNIST and LLM finetuning settings. To make the notation concise, we use k to reprsent AC' (C*(x,,, x,), KC' (Xo= X.,) and other related variants.
Step1: Create grobing dataset (#200) 20 examples numoer
Step3: Ca Calculate the NTK-rorm tensor each epach Epoch, 200, 200, 1 1|pp0 xx)l,,)
Stepf: Visualize CoNTK{ u. for each in one figure.
Align with trainyteet loss, Ex. [xo||
and Yu, | xx,,' togetner jobserve different phases).
Step4: Average NTK-narm of different CoNTK + Epoch, 10, 10]
l:athougg eNTK might change during training,
Step2 Calculate the tensor each epoch Epoch, 200, 200, 10,
Σ wol.
relative relatianship of CoNTK u. i] on different i stable after several epachs ing, epach! and 19", and "8" clear. Fact chchang eNTK the late training (similar finetuning) way smaller than the beginning phase.
2ach pairing effect
(0.0)
x (H
Figure 6: Experimental design of verifying the relative stability ||Cttoll for fixed x,, on different Eo.
C.I RELATIVE STABLE ENTK ASSUMPTION MNIST EXPERIMENTS
For the MNIST example, we directly calculate the eNTK term using a pipeline demonstrated in Figure The results showed in Figure where the key findings are:
21

---

Published as conference paper ICLR 2025
2526
46
1500
1020
505
2002
7010
79
1002
1020
vuls
y, is
5.
9
3002
2002
8
2002
9
3020
2020
1000
2-0
109
Iran los
0.8
1595
Namm alrichurgg
300
3002
1:0
Thstiing Spoets
Training Eacete
Training
Figure 7: Results showing the relative stability of |AC r for fixed x., different z, (labeled by the colorful digits near the lines).
1. The last three panels roughly indicate different phases throughout the training, where the first several epochs (0 ~ 30) are bit messy, and the last several epochs (800 ~ 1000) behave similarly to the finetuning stage;
2. Although the norm of eNTK (Eu, [le) and the norm of eNTK's adaptation
(Eu,o [|C - 1 [[e]) changes a lot after 30 epochs, the ranking between | |e on different o are relatively stable, as demonstrated by the upper panels;
3. The pairing effect between the "similar" inputs is clear, e.g., "4" and "9", "5" and "8", etc;
4. The pairing effect between the "dis-similar" inputs are also clear, e.g., "6" and "7", "2" and "5", etc.
5. The pairing effect mentioned previously is not strictly symmetry, which is because the inconsistent A and terms;
6. The accumulated influence demonstrated in the third panel of Figure is strongly correlated to the integral of all these curves.
C.2 RELATIVE STABLE ENTK ASSUMPTION LLM EXPERIMENTS
Directly caleulating | |r for the LLM experiment requires huge amount of computation, because for each token in each example, we need to multiply a V x matrix to a V one, where d the number of parameters of the LLM. However, since only care about the relative relationship between ||K[, on
22

---

Published as conference paper ICLR 2025
different Xo. where Xu fixed, based on the basic decomposition in Proposition 1, we can get a lower-bound as follows (ignoring superscript for conciseness, ignoring the influence of ))7²)))
A log π -oo
(23)
| log π T| = Aoloo
(24)
I 11 19
(25)
We hence define quantitive measurements have better understanding of Kno. they are:
|| log ग T | G. P
LBKua
S K/ool
(+1 SignDelta A Em,,llog म.,!
log T',
(26)
A.
where the subscript v. here represent the I-th token and w-th dimension for the prediction. In later experiments, we will observe both LBK and SignDeltauo to have better understanding of the strength (norm) and the direction (sign) of the relative influence imposed via IC..-
Regarding the calculation of LBK showed log x' for different responses.
A log || is easy track because, in the main context, already |9. = || - yt Il where yi is defined as a stacking of I. one-hot
vectors. The |/ is a bit complex. Recall the definition that A. = 1x we can have:
|A..l Trace Ao)
(27)
Trace (1-113 1m (1 (1-1-1))
(28)
Trace ( -1= m1 11
(29)
Trace ( 22Trace11 T () + VTrace(TT T
(30)
= =V - + V||
(31)
which is also trackable in our setting. Note that intuitively, the value of || is inversely correlated to the Shannon entropy of the distribution || 1f π is one-hot; || VV if is uniform. Hence we
can also interoperate ||A.o] as the peakiness of x Xo). In the following experiment, we track the value of LBKuo for different types of responses during SFT and DPO to show that the relative influence between different response types is relatively stable. We show the experimental results in Figure 8, in which the key findings are:
1. In both SFT and DPO under different supervisory signals, the change of these metries are relatively stable, similar to those in Figure 7;
2. The clear pairing effect between y (blue curve) and ffu. (red curve) exist;
3. In LBK learning any natural language sequences (i.e., y Yu yepss: Yepss) influence the non- language sequence (y Ympa a lot, especially at the end finetuning. However, from Sign know such influence is negative, which is caused by the pushing down pressure;
4. An interesting "similarity pattern" occurs: by observing SignDeltauo- we see SFT using Yapss or ypss imposes more influence the sequence generated using ChatGPT other than their original response (i.e., yt y.). which might be interesting phenomenon to explore further;
5. By observing the last row, where the model is trained using DPO, it is clear that the push-down pressure dominant. Because almost all SignDelta terms have big negative values, and the only positive one is yt (roughly 0.5, much smaller than other positive values in the SFT cases).
We also provide some intermediate quantities in Figure to further validate our analysis. The key trends are provided in its caption for ease of reading.
23

---

Published as conference paper ICLR 2025
Traciing aMIX LBC
Accuratated afun
Tradiig AigGnnt
AcouTABited
3F 10
(h)
Figure 8: Tracking the relative stability of K by observing LBK.. (a) and Sign (b) under different settings. The accumulated influence is the integral of the corresponding curve and z-axis (smoothed using exponential moving average).
D MORE ABOUT EXPERIMENTS
This section provides more experimental details and results about the learning dynamics to support claim. We will first discuss how different types responses are selected in our probing dataset Ppob These responses fit into 2-D space where one dimension is semantical relevance of the response ytt We then provide more results and discussions on different models and settings. The subtle differences between the responses all support our story well.
24

---

Published as conference paper ICLR 2025
Peakiness
lognly"
lognty
-40
-00
-25
203
-102
except aoompe
-100
Erear (iccllogggg
Figure 9: Other metrics related to LLM's learning dynamics. The first panel demonstrates how || changes during SFT (higher means peakier π). It clear that the peakiness y i... the supervisory signal, increases fastest. The last two panels demonstrate the average log x myy") and its complementary (denoted by log iyy") which measures how many probability masses are left for other possible tokens). The second one is for SFT and the third one is for DPO. It is clear that log (y") and log Tyy) changes faster in the DPO case, which matches our observations in the fourth panel of Figure 3 well. The linear fit extrapolates the log (pp") ) values because suffer an underflow issue when estimating this term. We will fix them in the next version. However, the trend of their changing speed is consistent across different settings.
I (o
KOvi Yrai mask)
reponse
Suseritsion
12 reghrase semantics leessnn
CE Loss
GPT-ephasee
hrdictions
Cassal mask with prels
Yewer
23 GPT-rephraee forma
2,
from
In event from tos:
Random sentence
Randoriy muned yt Pure sandom takano
forma cilancc
(b)
Figure 10: (a). How causal mask implementation helps convert auto-regression modeling to multi-label
modeling. (b). The 2-D plane y by considering the distance in both format and semantics.
D.1 THE SELECTION OF RESPONSE TYPES THE PROBING DATASET
Besides the sequential nature of the loss function, another conundrum in analyzing LLM learning dynamics is the huge response space y: the number of possible y E y is VL, but the vast majority of possible sequences look nothing like natural language, and we expect the model to generate only subset of natural language-like responses. These properties prevent us from observing the changes of all possible y like what we did for MNIST. Instead, we define several interesting regions of ) and select corresponding typical responses to observe. Intuitively, we can use the semantic relevance between y and x., as a heuristic, Such a measurement be understood as "how suitable this y is response to X,, compared to ytt Then, starting from the structure of common preference optimization datasets such as Ant ropic c-H (Y. Bai et al. 2022) and Ul UltraFes SaaFeebacc ck (G. Cui et al. 2023),
divide ) into three sub-spaces and evaluate the following types of responses (as in Figure 10b). The prompt templates used to generate them are illustrated in Figure 11. We also provide examples of all 14 types of responses in Figure 12.
25

---

Published as conference paper ICLR 2025
ViF reasonable responses following the instruction X
O. y,o, the initial response generated by feeding x L.MM before finetuning:
1. y the chosen (i.e., the preferred) response Xu.
1.1 Yillfr rephrase yı using y,,, algorithm from Z. Yang et al. 2024;
1.2 rrphrraee yt using ChatGPT, keep the semantics while changing the format;
1.3 y Yapr- rephrase yu using ChatGPT, keep the format while changing the semantics;
2. Yu, the rejected (i.e., the less preferred, but still reasonable) response to X
2.1 Yseler rephrase yı using y,,, algorithm from Yang al. 2024;
2.2 ypess rephrase yı using ChatGPT, keep the semantics while changing the format;
2.3 Y yppt- rephrase yu using ChatGPT, keep the format while changing the semantics;
Dec--i:: irrelevant responses to x, that are still recognizably human language (in these datasets, roughly "internet-standard English):
3. jzu the chosen response for a different question Xjya selected from the training set.
4. Visto the chosen response of question Xest selected from the test set.
5. Yhum "random" English sentence generated by Chat with many words y
y Yee humi token sequences that do not form meaningful human language:
6. y ytmad a random permutation the words (space separate strings) of y
7. Yna- random permutation of the words of a generated sentence as in Yhum
Furthermore, we also create another probing dataset (named D prohtess) where all x comes from the test set. Compared with D probiest that we used the main context, all the prompts and responses in Ppmittss are never exposed to the model during finetuning. By comparing the learning curves of these two probing datasets, we can figure out the difference between the model's ' prediction those directly influenced responses (y appears during training) and the indirectly influenced ones (y that the model never sees during training). Finally, we believe the level of the "on-policy" property (which very important for the preference finetuning, as discussed in Tajwar et al. (2024)) could also be introduced the second axis in our 2-D plane. We left the exploration of this interesting direction in our future work.
Prompt for self-rephrase: Sselfr
Below instruction on that describes task along with reference answer. Using the reference answer as guide, write your own
AEU Instruction: (x)
Reference
AHV Response:
Prompt for GPT-rephrase semantics keeping Yspts
Given reference sentence, please generate an output sentence. use different as possible while keeping meaning of the renceence sentence unchanged. Please only return the output sentence. Reference sentence: -y")
#tl Response:
Prompt for GPT-rephrase format keeping y
Given reference sentence, please generate an output sentence, Please change the meaning the reference sentence as much possible while keeping the formet of Please return the output sentence. asd Reference sentence: :)
ANW Response:
Figure 11: The prompts used generate Y y yps and yt you The rephrases of rejected samples generated similarly. The self-rephrase template comes from Z. Yang al. (2024).
26

---

Published as a conference paper ICLR 2025
Prompt:x
Rejected response
Haman: hat will remove siganiite ooer from a racm? Assistani: bellove there are products that will work at removing odors, Can you tell me woee's on?
you tell me about your stuation, I con try to give you more help.
2.1 Self-rophrase
Humon: What you meon?
I'm not sure what meon. I'm nutsure what yourre zokiin..
2.2 GPT-rephrase, semantics keeping
Initial response
Should you share details of your ciruumtances with me, er endeaver to provide fürther resetance.
I'm chemist but I'm guansing hat most ciganette oder comes the burning tobrcco. Hnow we can buy products that reduce the smell smikee, and that what most in arrffere anufacuures deim Would that who werk to remoie rignnette odor in your apartment?
2.3 GPT-rophrase, format keeping
your diruumst noes me, my best to provide you acdriceel atanseence
3. Irrelavent from train set
Chosen response
at nec manitr's release schwdule Are there any games you're part culletly exctte for?
Let's
You me't oned something about digaratie smell, can you tel me what hoppaned?
4. Inreavent from test set
Self-rephrase
You's 've sew serched through quite fow results and heven't come across a recipe you like yat.
believe there are products that will work removing cigarette odor.
5. Random sentence
GPT-rephrase, semantics keeping
You brought up the tssu of digorette odor, could you previde me with
The purple cst dencec under the starry night sky joyjj abandon.
more detail??
Random permuted ya
me about mentined can smell, tell happened? You cigarette something you a what
1.3 GPT-rephrase, format keeping
7. Puro random tokens
Can you provide datalls on the oder you notked corlier?
grrhghh few You'se recipe and wcrows ate searched come haven't resutts
Figure 12: Example of all possible responses for one x in our probing dataset. Note that the pure random token is generated by first creating random sentence, then randomly permuting its tokens.
D.2 MORE RESULTS ON DIFFERENT SETTINGS: SFT CASE
Consistent learning dynamics for different models. In this subsection, we provide more results to support analysis on SFT in Section 4.1. The first thing verify is the consistency of the trends of learning dynamics across different settings. As illustrated in Figure 14, we conduct SFT on five models with different sizes pretrained using different recipes. Note that Pythia-4 41 OM/10/1 4B/2. 8B are pretrained 1athiaa4400/1/111111
using exactly the same dataset and pipeline (Biderman et al. 2023), while Qwenl. 5- 5B are pretrained differently. Hence we can observe a slight difference between the curves from Pythia series and Owen series, e.g., those in Yhum However, the trends demonstrated in Figure 3 consistently hold for all models.
Compare the rephrases of yt and y. See Figure 15, where we put the rephrases of the same response into the same figure. We can treat the red curve, i.e., the one of y generated by π xx(x), as baseline, whose decaying suggests the policy model is deviating from the initial point. The first observation is that after several updates, y is the only one that keeps increasing fast, which means the "pull up" pressure generated by [xu:Vi] do not have that strong influence on these rephrases compared to [xui johul even though these y are good rephrases of y (recall the curve yjen always increase in Figure 14). Furthermore, by carefully comparing the decreasing speed of g and other curves, we find those rephrases decays slower than y in the chosen case, but not the case for the rejected responses. This phenomenon also supports our analysis well: because we train the model using y their rephrases are "pulled up" more than the rephrases of y.. Such a claim is also verified by the experiment in the last column of this figure, where we train the model using [xu; yw] rather than y In these two panels, we see the decaying speed of rephrases of yt is now identical to that of y while the decaying speed of rephrases for y, is slightly slower. Last, compare the green and orange curves (i.e., the Kormat- and semantics-keeping GPT rephrases), we find the predicting probabilities of those format-keeping curves are usually larger than their semantic-keeping counterparts. This is a sign that the model during SET might care more about the format rather than the semantics of one sentence. We will delve into this interesting phenomenon in our future work.
Compare D Prob and Prrobtest. To isolate the influence of the "pull up" pressure introduced by the training updates, we also create another probing dataset D Probtes using the same pipeline as Dprsb. The only difference between them is that all x in D proutees comes from the test set, and hence neither the prompts nor the responses ever occur during training. See Figure 16, where the solid curves and dotted curves represent the learning
27

---

Published as conference paper ICLR 2025
dynamics of responses in Proo and Prrobtest respectively. The color of the curves represents the model we are finetuning. By qualitatively comparing the trend difference between curves coming from D prob treed_dd
and D probtess we roughly observe that trend_di trend_ f gy which aligns well with our hypothesis about how strong the "pull up" pressure influence different responses.
brend_diffi gpts
Crean -ttccee
35
Viet
Nasker so0s
Matter depeei
(a) Result SFT Antropic
Ocese 24. seeceddd
haman rupptis
Whathr scourd durgirass traeeg
-pdata bufore mredice
22
25
Age
7
agedd
(b) Result of SFT on UltraFeedback
Figure 13: The learning dynamics of responses in different groups in the proposed probing dataset. Trends to observe: 1.) yt increase and y first increase then decrease; 2.) both yund and Y'md decrease and very small; 3.) Yo increases with a smaller rate than y although the [xui y never occurs during training; 4.) both Yust and Ynum has bell-shape curve; 5.) the inflection of hum is earlier. Because find that most sentences in Yhum are descriptive ones while those in yieu are question-answer style sentences. This suggest that the y are semantically more similar to yt than Ynum (i.e., larger [|K*]]]). Hence in general, the "pull-up" pressure on yust is larger, and hence its inflection point is later than Yuum
28

---

Published as conference paper ICLR 2025
YV
Yeer
gnario
petrast
Yeer
y.
Pram
110
Mamoer
(a) Result SFT Antropic
grtalice
Number aoocte
Mamber of S00
Rincer speche
Sambbr of apocns
Nimorral spoche
Number of
Yuy
-40
Number
Mamber
Number spechs
Mumber
Numbere
Number olaschh
(b) Result of SFT on UltraFeedback.
Figure Trend to observe: curves different models exhibit similar trends.
29

---

Published as conference paper ICLR 2025
Tumber ofnderefspoaa
Harbe
(a) Result of SET on Antropi C- HH.
.erraaL bothyy me
(b) Result of SFT on UltraFeedbacke
Compare different rephrases of ytt and y, under different models. Key trend to observe: 1.) For the first row, the decaying speed of y yps. and Youur are smaller than y,o, which means the pull-up pressure exists; 2.) For the second row, the decaying speed of Ygnts and Ygpe are similar to that of y,,, because the pull-up pressures rejected samples are smaller; 3.) For the last column, since we SFT the model using the rejected sample rather than the chosen one, the trend in (1) and (2) reverses.
Figure 15:
201
2.5
220
Number ebotth
Number of apocte
Tumber #00cMs
Number of apacht
Number sprcht
Number of spochs
-14
Yiar
-11
-04
-40
Number
Number
Figure 16: Compare the learning dynamics of examples from Dpros and Drrohtess- Key trend to observe: for Dprub. since many responses and prompts Curves of y Ygnss y you and y , shows clear trend. (Antropic HH, SFT)
during training, the pull-up pressure is generally stronger.
30

---

Published as a conference paper ICLR 2025
D.3 MORE RESULTS ON DIFFERENT SETTINGS: OFF-POLICY DPO CASE
Similar to Appendix D.2, we also provide extra experiments for DPO in this part using the same probing dataset. Note that as the responses of on-policy DPO change pnneration-by-generate it is hard to observe the dynamics of pre-collected probing dataset. We left the exploration of how to effectively probe other DPO variants in our future work.
Consistent learning dynamics for different models. Compare Figure 4 in the main context and Figure 18, where we provide the results on many different models (Pythia- 410M/ 1B/2. 8B and Owen1. 5-0. 5B). Their trends on different Tg. (y) are quite consistent:
1.) in the first column, the margin No. (ym) -o (yw) keeps increasing. The To (yt) first increase and then decrease, always with a smaller decay speed than that of Tg. (ym):
2.) in the second column, Tg. (yt)) decreases slower than the other rephrases, verifying the "pull up" pressure and the influence on other responses via KC;
3.) in the third column, To. (y.) decreases faster than the other rephrases, verifying the "push down" pressure and the influence other y:
4.) in the fourth column, the rephrases of y decay slower than those of yu supporting the claims that the rephrases near the chosen responses are influenced by the "pull up" pressure while the rephrases of the rejected ones are influenced by the "push down" pressure.
Learning dynamics of' conducting SFT first, then DPO. As stated in (Ouyang et al. 2022), conducting SFT before DPO is a common pipeline for alignment. Using {x,; the SFT dataset is also a common practice in many existing works. Hence in this part, we plot the curves of different Tgx (y) in both two stages to demonstrate their differences. See Figure 17, where the difference between the experiments in these three rows is how long the model is trained using SFT before DPO. The learning rate of both SFT and DPO are controlled to be the same (i.c., 5 x 10 the default value in (Tajwar et al. 2024)). All the curves are aligned by the 10th epoch on the x-axis (i.e., the starting time for the DPO training) for the convenience of comparing the trends across different settings.
We first check the curves of SFT and DPO parts separately and find that all the above relative trends still hold in these experiments. We then compare the model's behavior in these two phases respectively. In the last two rows of Figure 17, where the epoch for SFT is non-zero, it is clear that the decaying speed of most observing u (y) is much larger in DPO than those in SFT. The main reason for this is the existence of a big negative gradient introduced in DPO. This gradient, especially conducted on "valley" region of the model's prediction, will "push down" the whole curve significantly, except the one with the highest confidence before updating. This non-trivial trend is named "squeezing effect",, which is elaborated on in Appendix E. Furthermore, a more peaky Tga (y) and a smaller Tgo (ym) ) will lead to a stronger "squeezing effect", which can be verified by comparing the curves of the last two panels: longer SET makes the model's prediction peakier when DPO is conducted, which leads to larger decay on all (y) during DPO.
31

---

Published as conference paper ICLR 2025
Chougn Specled
Crosen wh.b sstraa
Rejedee v2, regrrsaee
Peested K.. Iaphaae
Nen-human 123200013
130
10
Are
Daa
Pro
X
Kin
X-
12
Numbar cpects
Nimber oloppera
Mamzer
humber
Namrer of
bumber oleppets
The learning dynamics of conducting DPO after SFT the model for several epochs. We align the starting point of DPO (i.e., the 10th epoch from the x-axis) to better compare the curves. Key trend observe: 1.) Confidence of all responses decays way faster when DPO starts, which caused by the squeezing effect introduced via big negative gradient; 2.) The more epochs we SFT the model, the more serious the squeezing effect is (confidence decays faster). (Antropic- HH, SFT - DPO)
Figure 17:
(a) Result on Antropic- EH.
(b) Result UltraFeedback.
Figure 18: The learning dynamics DPO on different models. Key trends to observe: 1.) Confidence of yt decays slower than that of ; 2.) Confidence of y decays slower than those of y and y Epp because the pull-up pressure is directly imposed on ; 3.) Confidence of y decays faster than those of ygpts and y gott because the push-down pressure is directly imposed on : 4.) Confidence of the rephrases of rejected responses decays faster than the rephrases of chosen responses.
32

---

Published as a conference paper ICLR 2025
E THE SQUBEZING EFFECT INTRODUCED BY BIG NEGATIVE GRADIENT
DPO, the model gradually learns how to separate the chosen and rejected responses by imposing one positive and one negative adaptation vector centered at yt and y. respectively, illustrated in the second panel Figure 2. These two opposite pressures ensure the margin reward ne (y x -e (y Xx keep increasing, which makes the model align better with human preferences. However, if we go deeper and consider no (y XT.) and To XT, separately (actually we should, because their X., are usually different), very interesting phenomenon occurs. See the first column of Figure 18, find although DPO also contains a strong positive adaptation vector, the curve of no ( X all goes down after several updates, which is very different from Te X in the SFT case. Such an observation is also reported in many related works (Pal et al. 2024; Rafailov et al. 2024; Razin et al. 2025; Tajwar et al. 2024), but clear-cut explanation of is still missing. Furthermore, although the relative behaviors of various rephrases matches our analysis of learning dynamics well, merely the two pressures on y and y, cannot explain why all these observed To (y) keeps decreasing during training. So, is natural to ask:
Where has the probability mass gone?
E.1 THE SQUEEZING EFFECT AND WHY IT EXISTS
To answer the above question, we can start from the properties of the basic Softmax function by analyzing a simple multi-class logistic regression problem. Because no matter how complex the LLM is, its predictions are made by converting the logits into probabilities using Softmax heads. Note that the analysis here only considers the negative gradient, i.e., the one imposed by y, in LLM's finetuning. As also pointed by Razin et al. (2025), the pull up pressure imposed by yt will cancel the influence imposed by yı when their X are identical. However, when x and X are dissimilar, the squeezing effect discussed in this paper still dominates. We left analyzing this intricate interaction between these two pressures is left to our future work.
Consider a simple V-class logistic regression problem where each high-dimensional input data x converted to a length-d feature vector via deep neural network 6. In other words, we have o(x) E IRdx1 The model uses linear read-out layer w E R to convert the feature vector logits w (xx) and then generate the probability prediction vector p using a Softmax head. We consider a common cross- entsopy loss function for each input pair (x. y). In summary, we have:
CCE(p' .v) = _ log p"; p' = Softmax(zz); z'= (w')) T
(32)
where t is the index of the step during training and e, is length-V V one-hot vector determined by the ground truth label y. To simplify our analysis, we assume a fixed @ and only update the parameters of the read-out layer using stochastic gradient descent:
w w - V =w--=w--- w' - e,y)
(33)
where 17 is the learning rate which can be negative if we consider a negative gradient during training. With Equation (32) and (33), we can write down each dimension of p' and p*+1 after some calculations. To
quantitatively analyze how the model's confidence in each class changes, we define ratio aq e pf and use the following lemma to describe its behavior:
Lemma 1. The ratio of confudence change for each i can be represented as:
P H+
E/= et
(34)
pf
8503
33

---

Published as conference paper ICLR 2025
Note that the values of By also depends on whether equals y, hence for Case (i y) and Case (i y). we have (1/ - x11(6())1 112 is the equivalent learning rate):
fe v'-''(1-p--
(1 if y
ifi y if 1
Case 1: 83
Case 2: Bj = (0-1609-0111) =
(35)
Proof. To derive Equation (34), we need have the analytical expression of each p p and p/. As p Softmax(z), we need to link z't ++1 and z' first. With Equation (32) and (33), zt+1 can be recursively written down as:
z1+ = (1 T x((x)
(w' - -no(x) - -e,) ) o(x)
(p' -e,))) (p'-)))))) o(x)
=
z' -11(p(x)11((pp e,,)
z' 1/'(p' ewy
(36)
where '/ è ]]]{(x)] is the equivalent learning rate that depends on the norm of feature representation. Note that z, p and ey all length-V vectors and is integer ranging from I to V. Then write down each 2;
2
+
if==
(37)
n'pi
i
Then, we can combine the definition of Softmax function and write down different p ++1 Case 1 where i y. we have:
case-by-case. For
e cki-n'ri+n
(38)
Pimg
Σ Ejst E -'1'' + e -
pD + e Σ Ejjy e²} of (1+p)
combining the fact that p
we can derive and By as the left part of Equation (35). Similarly,
D E) o3
when + y, we have:
e
# 10 -/ (39)
Ejny e -
Vp' + -
Eney e
(pp)
which leads to the right part of Equation (35).
We can now better understand how each p; changes after this update. Specifically, if > 1, the corresponding P: increases, and vice versa. To determine the value of a,, we can treat any 33 > 1 as contributing to the
conclusion that a; while any 33 against it. The value of the corresponding e and [8, 1 controls how strong the contribution is. With the preparations above, we derive the following observations on how the confidence evolves when gradient ascent (i.e., 7) 0) is imposed on class y.
34

---

Published as a conference paper at ICLR 2025
Claim 1: The value of Py is guaranteed to decrease, ie., at 1. We start from the value of B in Case 1 as illustrated in Equation (35). It is clear that for any j # we have B3 > 1, because 1 + p. pf > 0. Combining with By 1, it straightforward to have Claim 1.
The value of p:- where 1" = [VV p is guaranteed to increase, ie., ax- 1. We now use the value of in Case 2, since i cannot equal y by definition. When j * y. we have pj - -pp-- S 0 for all possible j. because p/. the largest among all p'zy of p'. Hence all 3y must be smaller than one. Combining with the fact that B, < (because pl, ps. 1 must be negative), we can prove that a,- > 1.
Claim 2:
The two claims above demonstrate that the parameter update can be imagined as taking the probability mass from Py and redistributing that to other dimensions. From Claim 2, we know some of the mass is guaranteed be "squeezed" into the dimension with the highest p¹-- (if py is the highest value, then p/. is the second highest in p'). But how other p: changes still not clear yet. Will the probability mass from Py is also split into other ps (i.e., other p; increases)? Or will Pr- absorb the mass not only from Py but also from other dimensions (i.e., other p; decreases)? To get a clearer picture, we need to track the adaptations of each p;- To achieve this, we now must scrutinize the distribution of p', because it controls the value of e for different j. We chose three typical scenarios where p' is strictly uniform, slightly non-uniform, and extremely peaky, and leads to the following claims.
Claim 3A: When p' is a uniform distribution, the probability mass decreased from class y is uniformly distributed to all other A y/, i.e., all Pizy increase the same value. With the uniform p' assumption,
Equation (34) can be simplified to a; = - "3-1 Note that the first two claims hold for any distribution
p', hence we only check the values of City here to verify the "uniformly distributed mass" hypothesis.
v Substituting the values of B3 to this new leads O₂ v-I+e" for all i * y. Since r/ < 0 and e" <1,
we must have City > 1. Combined with the fact that all p are the same, this claim can be proved.
Claim 3B: When p' is slightly non-uniform, D3 with smaller p tend to decrease, and vice versa. This claim is a general trend and might not have any guarantees. However, analyzing such scenario helps us to understand the influence p' better. Assume we are observing ay where i' is not y nor i". We consider two subsets of [V] {v). i.e., B, which contains all j with p S p and S that contains all j with p³. > pj.. Now consider Case 2 in Equation (35), we have:
< < 1;
3JEB > 1.
(40)
Note that misuse the < notation to highlight the fact that 9j=y would be much smaller than 3jes. because there is a negative one term in the exponential. With the above expression, we can imagine that if ph, is relatively small, the size of B would be large, which means there will be more 8y contributing the conclusion that og < 1. If the influence of AJEB is strong enough to override the influence of other B (especially 85=y which is way smaller than other 3), ag would be smaller than one and hence py decreases. On the contrary, for those i' with relatively large pl,, the 3 < 1 terms becomes dominant and hence lead to ay 1, i... Pe increases.
In the analysis above, we assume p' is only slightly non-uniform (i.e., not so peaky), which means the values of different e are relatively comparable. However, in practical machine learning systems like LLM's finetuning, the distribution p' would be very non non-uniform, which means most of the probability mass is obtained by few dimensions. That is because the LLM''s vocabulary size is usually very large and the reasonable choice of the next word is only a small portion of the whole vocabulary. Thus we have the following claim to describe this practical scenario.
35

---

Published as a conference paper ICLR 2025
Claim 3C: When p' is very peaky, which means most of the probability mass is obtained by i", then all other P: will decrease. In other words, the probability mass of all other pi is squeezed to pr We continue the analysis in Claim 3B but consider more extreme influence on e* For this peaky p', might have very large ex! that dominates a. In other words, 0z 0 B - observe, the ag 2 3.- with the highest confidence in p' will decrease.
of
Then for any i' we want
1 < 1. In other words, the model's predictions on all dimensions other than the one
Last, we analyze the influence of Py to explain why "imposing a large negative gradient on the valley region" makes the squeezing effect more serious.
Claim 4: Smaller p makes those non-max p; easier to decay, f.e., a stronger squeezing effect. This is also a general trend that is observed in the experiments in Figure 20, Intuitively, since the model is already confident that y cannot be the correct label (i.e., Py is very small), letting the model further decrease the prediction on Py does not make sense. We can also use the analysis above to understand how it happens. As illustrated in Equation (40), where the value of / is decomposed into three subgroups. Recall the definition of 04,, we know all 8, < contribute to the hypothesis that p; increases after this update, where the strength of this contribution is controlled by e Since P small means small e the influence of 3 Bj-y < 1 is significantly weakened under this scenario. In other words, a, < is more likely to occur for all possible i, which means the squeezing effect (all Pjzy decreases) becomes more serious.
The learning rate with larger absolute value ly) and a larger feature norm [x)()) will amplify all the trends, maybe more serious than our expectation. Throughout our analysis, the equivalent learning rate nf < 0 is shared scalar in all By. Hence larger [r/| can amplify all the trends aforementioned. Furthermore, recall the shape of an exponential function ex,, where a small change of x (especially when x > 1) will make ex changes a lot. Then the terms Bitv e- (1-pp in Case 1 and ßj=y = e e (+)-r(-1) in Case 2 will play a stronger role if we use larger learning rate |n) or the norm of features is larger.
Claim 5:
E.2 VERIFY THE SQUEEZING EFFECT USING A SIMPLE EXPERIMENT
Postitre tradlen (M
P: Big necathre pradient for flst mrg
C: Ule nepet ve credient on the "pesk" rrge pol (PO)
D. Big neguttiv: grafient the "viliey"
Push down evenytiing but the
Yu
yu
Yu
Yu
Figure 19: Illustration of how big positive and negative gradients influence the model's prediction.
Let us analyze a simple example to get an intuition. We set V 50, 5, |n| 0.5, and randomly generated dxx). In the first row of Figure 20, we consider the model updates its parameters using standard SGD assuming the label of this x is 21. Specifically, we randomly generate w by sampling each parameter from standard Gaussian distribution and calculate w using Equation (33). The two curves in each panel demonstrate the model's predicted distribution before and after this update, As we expected, the positive vector the 21st class "pull up" pa 21) and "push down" all other p (v) at the same time. This trend is quite consistent under different settings (i.e., different choices of V, d, x, 7) w etc.), which can be depicted by the first panel in Figure 19.
36

---

Published as conference paper ICLR 2025
Probabilly change
Probability By change in log scale
R10
36-4
Yu
31-
02-
31-1
Yo
n
10-4
38"1
Ne paaky
Yo
20-
22
Y5
20-0
3.-
200
42B
800
306
1200
200
1089
Index vocabulary
Index vocabulary
Figure 20: Experimental verification of the "squeezing effect" illustrated Figure using simple multi-
class logistic regression task.
We then 17 0.5 to simulate the negative gradient in DPO and consider three different settings. First, we assume the model's prediction on x is relatively flat, as demonstrated in the second row of Figure 20, where the predicting probability of every class is around 0.02. The negative gradient is imposed on y = 42, a randomly selected number. We see the negative adaptation vector "push down" p (y = 42) heavily and re-assign those decreased probability mass evenly to other classes, as illustrated in the second panel in Figure 19.
Although the behavior described above follows our intuitions well, a flat pd not common in LLM's finetuning. Because finetuning usually starts from a pre-trained w, where the model's prediction would likely be non-uniform. So in the third row of Figure 20, we consider a more practical w that leads to multi-mode pº. In this example, the model has relatively high confidence in classes 5 to 11 and low confidence in all other dimensions. We set the target label as 11 (i.e., the one in the model has the highest confidence) and -0.5 to "push down" the model's prediction this class. As demonstrated by the blue curve,
37

---

Published as a conference paper at ICLR 2025
(' 111 decreases lot as we expected. However, different from the flat p" case, where the model evenly assigns the reduced probability mass to all other y. the model in this example "squeezes" the mass to those confident predictions, i.e., classes 6, 9, and 10, leaving the confidence of other classes almost unchanged. Such trend is consistent when the negative gradient is imposed on the "peaky" region of non-uniform distribution, as illustrated in the third panel in Figure 19.
The previous setting simulates the on-policy DPO well, where the rejected examples yū are sampled from the high confidence region of the model's 's predictions. Then, what will happen if we conduct off-policy DPO and impose big negative gradient on those classes that already have very low confidence? See the fourth row of Figure 20, where we use the same w and 1) as in the previous case. The only difference is that we change the label of to 42, where p (1yy 42) is very small (roughly 10 10-6) before training. The behavior in this setting is quite interesting: we first observe a big increase on p (y = 11), which means the model "squeezes" the probability mass to the most confident one in pD, similar to the previous setting. More interesting, the predictions on all other y are heavily "pushed down", even including classes 6, 9, and 10, whose confidence relatively high before training. In the last two panels of Figure 20, we set V 1000 and find this trend is more obvious (that might be because the absolute value of the efficient learning rate, which depends on [})()] becomes larger). Since the vocabulary size of a common LLM is usually more than 50k, the squeezing effect in real systems would be non-negligible even if the learning rate is small. Such trend is also quite consistent as long as we impose a big negative gradient on the "valley" region of the model's s prediction, as illustrated in the last panel in Figure 19. Now we can answer the question of why all observing Tge (y) decreases and where the probability mass has gone:
For each token, the probability mass is squeezed to the one with the highest confidence.
Note that the tokens with the highest confidence do not necessarily form a preferred response: it just reinforces the prior knowledge contained 0º, which could be a drawback for off-policy DPO.
The hypothesis above is not only supported by this simple logistic regression problem but also by many consistent trends in LLM's finetuning experiments. First, by comparing the average decaying speed of the To: (y) when the model SFT different epochs before DPO (in Figure 17), we notice that longer SFT leads to a more peaky To (y) and hence leads to a faster decaying speed of all non-argmax responses. That is because the longer SFT stage will eventually push down Tge (y.)) more. Hence in the DPO stage, the big negative gradient is imposed on deeper valley region, which makes the squeezing effect stronger. Second, to directly verify this hypothesis, we track the sum of the log-liiel of the tokens with the largest confidence and call x, Y1a-1). As illustrated in the last panel in Figure 4, the argmax confidence keeps increasing while all other Tax (y) decreases: the missing probability mass is found! Last, in the dataset-extension method we proposed in Section 4.3 and Appendix F, we train the model using both [x. y and [x. yu] during SFT to also "pull up" the yu region before conducting DPO. Then, we observe compared with the standard training flow, i.c., SFT using [x; y first and then DPO, the proposed flow has a lower "argmax confidence" during DPO. That is because SFT stage, the big negative gradient is then imposed on the peaky region rather than the valley region of the model's prediction. Such a change in turn weakens the squeezing effect, as illustrated in Figure 5.
it "argmax confidence", i.e., Σ, Ter (argmax,,
pulled up Tgo (ym) during the modified
F A SIMPLE METHOD TO IMPROVE ALIGNMENT
F. 1 PINPOINTING THE DRAWBACK OF OFF-POLICY DPO
Based on our observations and analysis above, we speculate that "imposing big negative gradients on the valley region" is one of the bottlenecks of off- RL-free methods. Starting from this hypothesis, we believe introducing on-polin on-policy sampling has the potential to mitigate this problem, as demonstrated in SPIN (Z. Chen et al. 2024) and other online algorithms (S. Guo, B. Zhang, et al. 2024). However, also speculate
38

---

Published as a conference paper ICLR 2025
that these methods improve the model's performance not only by mitigating the squeezing effect. Hence to figure out to what extent the squeezing effect can harm the model's performance, we propose simple yet effective method to isolate its influence. As this method can directly mitigate this effect, it can also be considered as an ablation study of this interesting phenomenon.
B B--2 DPO 2 epochs
B
Bz-4 DPO epochs
SET using (x,yt)
DRD my (x,y*,7)
B2-6 DPC epochs
Pretrained LLM
Dffferent
Ideitial
B = Eg
SFT using (x,y*) AND (x,y")
Eg 2 DPG 2 epochs
DPO using (cyt,y
Identicall
E
Ez-4 DPG 4 spochs
E2-6 DPO epachs
Figure 21: Illustration of the proposed method and baseline. "E" is short for the "dataset extension".
F.2 A SIMPLE METHOD INSPIRED BY LEARNING DYNAMICS
As illustrated in Figure 21, where the baseline method is a standard SFT-then-DPO pipeline. The proposed method is very simple. We only need to augment the dataset used in SFT by adding (x. yu) pairs for each sample into it. All other settings are unchanged. The motivation for this method is also quite simple: as SFT can pull up the region of supervised y and we don't 't want the model to impose big negative gradients on a can just pull up those y. before DPO. Furthermore, demonstrated in the third panel in Figure 19 and Equation (15), the negative gradient in DPO would be strong enough to push down Tp. (y yy)) ), because the gradient will be large if the model cannot separate y and yı well. In other words, under DPO's loss, there is no need to worry about the model overfitting those y, during SFT.
valley region,
F.3 EXPERIMENTAL VERIFICATION
To verify our analysis, we conduct experiments by finetuning a pretrained Qwen1. 5--1. 8B (J. Bai et al. 2023) model using Ant ratiropion C-HH dataset (Y. Bai et al. 2022) (we use a subset containing 5000 random examples from the training split). The pipelines of different methods are demonstrated in Figure 21. In this experiment, we call the pretrained model Bo (and Eg. which is identical to Bp), which is an official checkpoint pretrained by J. Bai al. (2023). Model B, and E₁ are the ones after SFT, which are different for these two methods. Model B2 2/4/6 and E 2 are the models finetuned using DPO for 2/4/6 epochs. All the settings (except the starting model) of the DPO stage are the same for these two methods.
We first observe the learning dynamics of these two methods in Figure 5, where all the trends support our analysis quite well. See the first two panels that compare wai (yt) and xa. (ym) respectively. It is clear that these two methods have an almost identical curve on T (yt) in the SFT stage but behave quite differently on Fa. (y.)) ): because we directly train the model using (x, y..) the proposed method. Then, after the SFT stage, we conduct DPO using identical settings for these two methods. From the first three panels, we can observe the decay speed of all curves the proposed method smaller than its counterpart in the baseline. That is the benefit introduced by "pulling up" the Tgo Too (y,) region before conducting DPO. With this specific design, the big negative gradients in DPO are imposed on the peaky region (the behavior is like the third panel in Figure 19) rather than the valley region (see the fourth panel), hence the squeezing effect is successfully restrained. The results in the last panel of Figure 5 are also a strong verification of the whole picture. During the SFT stage, the observed nargmax-probabilng of the proposed method is higher than the baseline, because
39

---

Published as a conference paper ICLR 2025
Cleaeny?
Other meccnnet
Argrax PRESOTER
reptrases
X.
Masiumaa
Deip attee
extord
Number oTedoms
Number stepehh
Nimberof specte
Winber of speete
(a) Result on antropic-ti
Argres toppate
Shw the
baralre
sobend
Rrough new
Sesueene
Namber spocts
humber of socdhs
Number of soccha
Numoer of azecch
(b) Result on ULt raE eedback.
Figure 22: Learning dynamics of the baseline and the proposed method with training data extension. The one for SFT is the same as Figure 5 in the main context. Key trends to observe: 1.) Baseline and the extend method have similar behavior on y during SFT; 2.) The extend method considerably increases yu during SFT; 3.) The squeezing effect of the extend method is weaker (all other responses decay slower and the confidence on argmax response increases slower).
we impose twice "pull up" pressure, i.e., those for (x. y. ) compared with the baseline. However, the beginning of DPO, we observe a clear drop in the orange curve. That is because the negative gradients are exactly imposed on those yu (in the second panel of Figure 5, Ago (y,) already very high). Furthermore, at the end of DPO, we see the "argmax- "egeax-probabiltt"" of the proposed method is significantly lower than the baseline setting, which implies that the squeezing effect is restrained our setting.
In order to figure out whether the model trained using the proposed flow, which successfully restrains the squeezing effect, indeed does alignment better, we conduct pair-wise comparisons of these models' responses and report their win rate as in (Rafailov et al. 2023). Specifically, we first randomly select 1000 test questions from the test split of Ant ropic- eopic--Hi HH and generate 1000 responses by feeding the prompts to each of these models (we use the default sampling setting provided in (Rafailov et al. 2023)). Then, with the prompt template provided in Figure 23, we evaluate the win rate of the responses pairs using GPT3, OPT3.5-Tu0 5-Turbo and Claude3 Ha iuu Here we report the average win rate of different comparisons (the degenerated responses are not compared, so the number of compared examples is slightly smaller than 1000). Note that a win rate greater than 0.5 means the method that comes first is preferred by the evaluator.
1. Compare models after SFT: Ej v.s. B1, win rate is 0.4729 and 0.4679;
2. Demonstrate benefits of DPO:
Ba-4 v.s. BI, win rate is 0.6727 and 0.6411;
E--4 v.s. Ej. win rate is 0.6898 and 0.7321;
3. Compare the proposed method and baseline after DPO for different epochs:
a. E2-2 V.S. B2-2, win rate 0.6518 and 0.5151;
b. Eg-1 v.S. B2-4, win rate is 0.6928 and 0.6045;
Eg-6 V.S. B2-6, win rate is 0.6667 and 0.5432;
4. Compare the best Ez_4 with other 2 checkpoints:
40

---

Published as a conference paper at ICLR 2025
E2-4 v.s. E2-2, win rate 0.6853 and 0.5517; E2-4 v.S. E2-6, win rate 0.6324 and 0.5316;
In the first comparison, we find the model trained using both (x, yu) and (x, Y.) loses more (win rate is smaller than 0.5), which makes sense because Ej assigns higher probabilities on those less preferred responses. In the second comparison, the model fine-tuned using DPO indeed aligns with human value better. The win rate of the proposed method is slightly higher, which might also be explained as E1 leaving more space for improvement. Hence we then directly compare the models after DPO these two methods in the third group. In this group, all models in the proposed method win the baseline counterparts by a large margin, which demonstrates the effectiveness of our proposed method. Furthermore, we find the evaluation made by Claude is more reserved compared with GPT (the numbers are smaller). However, the trends among the comparisons in this group are consistent: E2-4 brings the largest improvement, which is potentially the best model. This fact is verified in the fourth group comparison, where evaluate E2-1 against Eg-2 and E2-6- The results demonstrate that both a too-long or too-short finetuning stage using DPO is not the best choice.
Prompt for judgement:
Example of feedback:
Given the history of multi-round chat, which response more helpfu/?
History: (promat)
Compar son;
Response A: Iresgonse_ A]
Response provides clear and cancisa explanation of why turkeys became the official food Thankssgivigg focusing on historical co steet and signtf can tee,, while Response ricludes more detailed but somawhat scattaras information that may be overwhelming for the reader,
Response fresponse B]
FIRST provide mee-eencnnce comparison of the two responses and explain which you feel is more helpful. SCOOND, a line, state only "A" "Q" to indicate which response more helçful. Your resginis should use the format:
Comparisen: sonn sentence comparison and explanctionn
Mure helppul: < A' or "Bts'
Mora helpfull: A
Example degenerate response (repester effect]:
txomple 3:
Yos'ee vary see come, Haue spocdidyy liy rose
you cding mel 3 ape n. ttank you sxing (i) hade suger day dh) and thank you for caing taal re)) and piess saic pleaee minn
3 thnk you fhrusing
3 pleace dorting you
you using.
do for
(1]
mel [I]
things for faturs (11
Fomple:: 2
crsta you right wond by seedding your coler more oftenn You 3lo try teingwoigs nts colo bodywcienn woro lie cot releserjummpp porpueuu Aec you AUF YOL wast thot? not surt iPoegeed thes meal people aheays wiitte serrent that no mouily therggh lape epecteent aoot Ase you sure
to try What iiteriigg righ your Whe mimbeeeye hartt you ongr??
vou poct)
maver
whete
you secee?
naver ereyy?
Pitmokkss sodF Whatht
you Sepeorsod7
wouwomeed What i- mabes ted? What makes posstesedd? whua fill miikesss woried? truttl makes you hustratdd? Whitt males you frustrate)) WT mates soudls tted? What il meiio you chappeintvt Whald moker pourpperty Whet maires you upuety Whet inll makes hatoes? de moom hurmase Whee i mebos ernoyes?? What it ma you annoped?? what masecyyu angry? Whel IT's mave you angly? Sena tt make you snnoyed wheve trokeeyn you and gett what It maker you stropeed Wate mokkes ghoannnge whete makes you tncopedd What reabi you annayed?? Whall makes you amnoyed? What Fitmmbbs ereuyod?? Whatif mave annoyed? What Linki arnopes? What ma arroyyat)....
what 2 makes anolwast Whet makes pernnoitf What makes worres? What you apolstensnes what (T mares
scardd Whe titmmless heamut Westt
Figure 23: Prompt used for evaluating model's response (from (Rafailov al. 2023)), an example feedback from GPT3. 5 -turbo, and two examples of the "degenerate" effect described in Holtzman et al. 2020, Although both B2 and Eg inevitably generate such degenerate responses, we find this phenomenon is less common in the proposed method.
41


Reviews:
[{"review_id": "QiHWk89hT3", "reviewer": "Reviewer_5a38", "summary": "This work studies the learning dynamics of LLM finetuning by analyzing the way in which likelihoods assigned by the model to different completions of training prompts evolve throughout the finetuning process. By explicitly writing out the single step update rules in terms of the change in model parameters and change in the functional output of the model, and then leveraging the empirical NTK for the LM head/logit layer of the model, they show that one can analytically describe the sample wise influence dynamics of both the SFT and DPO objectives for LLMs. Corroborating prior observations of model distributions degenerating during (off-policy) DPO, their analysis describes a \"squeezing effect\" due to the performing of gradient _ascent_ during standard preference tuning, and they validate this and other qualitative predictions via the finetuning of 1B-3B parameter LLMs on carefully augmented preference tuning data.", "questions": "1. Can you discuss the analytic or empirical argument for \"Peakier πθt suffer a more serious squeezing effect.\" in a bit more detail? I see arguments titles Claim 3A and 3B on pg 27, but they seem to be more intuitive that formal. It could be useful to separate the claims in the Sec 3.3 enumeration into those with direct proof/analytic evidence and those that are more informal or empirical -- this would not weaken the arguments, but rather making these subtle distinctions clear would increase the soundness of the presentation.\n\n2. The color coding of the variables in section 2 and 3 is non-standard but fine if the authors like this (and no other reviewers mind it). However, it seems inconsistent to not continue this into Sec 4 and 5? Additionally, if there is a way to continue the scheme into the figures, especially Fig 2, that would make more full use of the fact that  the x and y's are colored in the related written sections.", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. They motivate and ground the study of training dynamics for LLMs well and highlight key difficulties in applying standard influence function style analyses to LLM finetuning and preference optimization, a highlight of the preliminary material.\n2. The formalism and analysis in Section 3 is generally clear and well written. Figures 2 and the bullet point enumeration in 3.3 are concise presentations of the core claims of the analysis.\n3. The experimental design of the empirical verification section is well done and cleanly visualized, particularly the use of completions $y^{+}$ gpts, gptf, test, hum.\n4. Proposal for a mitigation to the \"squeezing effect\" (while not the main contribution of the paper) is well motivated by the analysis and simple to implement.", "weaknesses": "1. The clarity of Sec 3.1 when discussing the causal masking and teacher-forced production of the full next-token logits set could be improved, though this is admittedly tricky.  It is possible that for some readers, a diagram of the matrix structure here could be helpful since most papers don't tackle the more complex formulation of the influence problem and so readers may not be clear on it. (The reviewer imagines a teacher-forced causally masked model forward on input and label sequences X,Y as a forward on an augmented view of X,Y according to causal mask M, such that if the length of X+Y=N, the input has X' now has dim NxN (lower triangular consisting of all prefixes of the sequence X+Y) and we consider the output to be labels Y_i for each row of X' corresponding to the next token following the tokens in row X'_i.)\n\n2. Additionally, while the analysis relies on the kernel values between the feature maps of $\\mathcal{X_0}$ and $\\mathcal{\\tilde{X_u}}$ to implement the pressure transfer between various samples during learning, these intermediate values are not actually experimentally estimated. To concretely tie the analysis to the empirical observations, it would be useful to compute examples of the decomposition values for the terms in Eq 5 and Eq 7. If this intermediary relationship can be shown as clearly as the loss trends described in Figures 3 and 4, a stronger assertion of causality between the theoretical model described by Eq 5 and Eq 7 and the empirical learning dynamics can be made, strengthening the impact of the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cRgKzTN1Ym", "reviewer": "Reviewer_yrnd", "summary": "The paper extends the existing learning dynamics for deep learning systems to LLMs and  argues that doing so is a non-trivial problem due to high dimensionality and sequential nature of both inputs and outputs. The paper talks about a “squeezing effect” where running DPO for too many epochs causes outputs to deviate from ideal behavior.", "questions": "How does the squeezing effect differ from overfitting with DPO?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1) The paper has strong theoretical and experimental backing for their analysis.\n2) First paper to propose a framework extending learning dynamics to LLMs.", "weaknesses": "1) Did not find any specific weaknesses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eh6FffGGwF", "reviewer": "Reviewer_4pPy", "summary": "The learning dynamics of a linear-softmax classifier on top of fixed features is analyzed. The method is applied to the settings of LLM finetuning, by considering particular loss/objective functions specific to this settings and analyzing their effect on learning dynamics. The results helps elucidating previously observed phenomenon in LLM finetuning, and constructing new methods for overcoming potential \"bottlenecks\" in the process.", "questions": "Line 31: \"Many interesting phenomena\" -- such as? (not saying there aren't, but this is an odd statement to have without providing at least some refs)\n\nLine 61: I feel that this is conflating different things, i.e some notion of \"interpretability\" (some \"compact\" understanding of how model predictions/behavior are determined by the parameters), and notions of learning dynamics (how model parameters evolve during training). One can in principle discuss the one without the other, and it would be better to spell out more clearly / explicitly the current motivation.\n\nFigure 1: The red arrows on panel a, top do not actually represent $\\Delta\\theta$ but rather $\\Delta \\pi$. It's also not clear why a continuous curve is plotted, given the fact that $\\pi$ is discrete -- why not show all 10 digits as bars (as is later done in panels (c) and (d))? Perhaps most importantly, it's not clear whether or not this figure should be understood only qualitatively, or is this the result of an actual computation.\nThe same comments apply for all panels in Fig.1 (b). If this is an actual result then many details are missing, both in presentation (bars; scale of the y-axis) and perhaps more crucially, at what stage of learning this is done. The entire argument relies on some implicit assumption that \"4\" images are similar to \"9\" images, but this crucially depends on the current learned representation of the model (since this is a \"feature-learning\" rather than \"lazy\" regime). If this is just an illustration, then it is highly misleading presenting it along with panels (c) and (d) which seems to be actual numeric results.\nI'm also unable to understand how to interpret panel (c). The caption says \"accumulated change\" but the the charts seems to indicate absolute probabilities. It's also unclear what is being tracked, over how many epochs, and what are the examples being presented. The main text (lines 134--138) is unhelpful and reads as another explanation of what's actually illustrated in panels (a)-(b) (i.e., the single-step influence)\nOverall, the entire discussion of the motivating example in Section 2 is hard to follow. I think the overall intention of illustrating the idea in a simple example is a good one, but the example itself should be made much more clear.\n\n\nLine 233: \"Any no assumptions\" -> \"no assumptions\" \n\n\nLine 253: Calling this \"unexpected behavior\" might be too strong, see previous literature\nLine 314: \"Fine grind\" -> \"Fine grained\"", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is overall well written, clearly explains the motivation and the approach. The use of simple \"pedagogical\" examples also helps in communicating the main message of the paper.\n\nThe main contribution seems to be in the application of the theory to the different loss functions used in LLM finetuning, making explicit use of the \"decomposition\" of the effective learning dynamics into 3 different terms, and identify which one directly depends on the loss function being used. This is demonstrated to be productive as it can provide (at least a preliminary) explanation for different observed behaviors.\n\nOverall, this is an interesting paper and the results are likely to be of interest to the community.", "weaknesses": "There are two major limitations here that, while are (somewhat) acknowledged by the authors, can benefit from a more careful discussion and/or analysis.\n\n1. The first is the obvious limitation that the entire analysis is being done under the assumption that the \"feature map\" is held fixed, and only the classification layer is changing during learning, but the extent to which this assumption holds in practice is not being evaluated at all. Without such an evaluation (for example, either by quantifying/tracking the change in $\\mathcal{K}^t$, or by repeating some experiments when the weights are *actually* frozen) it is hard to say whether the theoretical explanation about the squeezing effect is in fact an important part of the observed phenomena (or whether it is largely / to some extent driven by more complicated dynamics that involves changes in the feature map as well). I think this should, at the very least, be acknowledged more explicitly in the text.\n   \n2. The second limitation is more nuanced, and has to do with the relying of some intuitive understanding of \"similarity\" (between prompts/examples), that may or may not be valid. Again no evaluation is offered -- despite the fact that the authors explicitly specify how this similarity should be understood (i.e., it is measured by the eNTK, $\\mathcal{K}$ ). It is far from being clear a-priori that the learned features agree with some intuitive judgement of similarity. Moreover, what \"dissimilar\" means, in the extremely vast space of strings, is also non-trivial. For example (and as pointed out by the authors), it might be that all \"well-formed\"/grammatical sentences are somewhat similar, and the model is able to push probability mass to entirely non-grammatical/nonsensical strings. This leads to some arguments being made only on the basis of some intuition. For example, \"the model’s confidence on y+u keeps increasing and the update energy... gradually decreases\"  (Line 350) -- this seems to be inconsistent with the fact that we see no slow-down of the updates for $\\mathbf{y}^+_{u}$ itself.\n\nFinally I would also like to point out that peakiness (/\"squeezing\") effect in text generating models followed by RL tuning is a phenomena that has, in fact, been observed and to some reason explained before, including the observation that the on/off policy choice might play an important role in dealing with the \"exposure bias\". The paper will benefit from referring to some of these previous works, for example:\n- Caccia et al., Language GANs Falling Short, (arxiv 2018 / ICLR 2020)\n- Choshen et al., On the Weaknesses of Reinforcement Learning for Neural Machine Translation (arxiv 2019 / ICLR 2020)\n- Kiegeland and Kreutzer, Revisiting the Weaknesses of Reinforcement Learning for Neural Machine Translation (naacl 2021)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1jVMYiBBVW", "reviewer": "Reviewer_W188", "summary": "This paper analyzed the learning dynamics of LLM fine-tuning, in particular, the change of the loss function for other examples delta f(x_o) during training time parameter updates (from gradient descent with respect to input examples x_i), for both SFT and DPO. For SFT, the research used MNIST classification as an illustrative example to indicate that if x_o is similar to x_u but classified differently, the predicted probability of the x_u's class for x_o also got pulled up. For DPO, the research found that log likelihoods of both the positive examples y_u^+ and negative examples y_u^- all decreased, and the only example with log likelihood increase is the greedy decoding result y_u^*. This phenomenon could potentially explain the increased repetitiveness of DPO generated results towards the end of fine-tuning, and the increase of halluncinations during SFT. The researcher proposed a method to mitigate this effect by first SFT-ing on both positive examples and negative examples, and then perform DPO.", "questions": "1. In your page 5, \"make any no assumptions\" -> \"make no assumptions\"\n2. In your appendix A, add more derivations to show that -(e_y_u^+ / pi)^T [A(x_u)]_l = pi(y_l | x_u) - e_y_u^+. All parts in Appendix A until this point is easy to follow.\n3. In your page 16, lines 712-713, are there some nablas that are left behind?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Analyzed the theoretical process behind DPO gradient updates. Explained deeply how the effect of gradient updates with respect to the training examples affect the model's performance on unrelated examples. Provided a theoretical framework for (potentially) understanding the origin of hallucinations.\n2. Explained a common problem in DPO that was not illustrated well in prior work. For example, Rafael et.al. 2024 only vaguely claimed that \"DPO decreases the likelihood of all data in favor of extrapolated responses\", but this paper clearly indicated that the decreased likelihood all (or mostly) added to the greedy decoding sample (y^*).\n3. The paper is well written and easy to follow.", "weaknesses": "Disclaimer: Only checked the main text and the Appendix A. Did not check Appendices B,C,D.\n\n1. Need a \"related work\" section. The paper in its current form does not explain the relations between \"learning dynamics\" with related literature, for example, preventing negative transfers during transfer learning, neural tangent kernels, or loss landscape that enabled adversarial attacks. Offering a \"related work\" section whereas reducing the theoretical focus a little bit would help ICLR readers better position your work.\n\n2. Need to indicate assumptions in your prop 1 and section 3.2 clearer. As far as I could see, there are at least 2 assumptions: \n- Higher order terms has little to no impact to learning dynamics\n- K changes slowly during the initial training stages and thus has a limited impact to training dynamics\n\nPlease list out these assumptions (potentially what I did not listed out) so that users will follow and understand the limitations of your analysis.\n\n3. Should design more experiments that \n\n- (a) During off-policy DPO, show that no other sentence's log probability significantly increased other than y^*. Although your theoretical analysis seems to have proved this, we still need to show this point in experiments due to your analysis assumptions. The sampling should be much more extensive to show the non-existence of such examples (and statistically significant)\n- (b) For SFT, quantitatively show the causal effect of increased hallucinations due to the \"pull up\" effect of similar but negative examples (scaling up more than MNIST)\n- (c) For both experiments, show that neuron tangent kernels (your K-term) did not change much, and thus the learning dynamics can be fully attributed to your G-term\n- (d) Qualitatively demonstrate the repetitiveness of off-policy DPO towards the end of fine-tuning\n- (e) Quantitatively measures how your proposed approach reduced hallucinations / answer repetitiveness via more rigorous benchmarks, other than just a measurement of the win rate", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)