Learning Dynamics of LLM Finetuning

## Abstract

Abstract Learning dynamics, which describes how the learning of specific training examples influences the model’s prediction of other examples,
give us a powerful tool for understanding the behavior of deep learning systems.
We study the learning dynamics of large language models during finetuning,
by analyzing the step-wise decomposition and accumulated influence among different responses.
Our framework allows a uniform interpretation of many interesting observations about the training of popular algorithms for both instruction tuning and preference tuning.
The analysis not only explains where the benefits of these methods come from
but also inspires a simple, effective method to further improve the alignment performance.
Code for experiments is available at https://github.com/Joshua-Ren/Learning_dynamics_LLM .

## 1 Introduction

Deep neural networks usually acquire new knowledge by updating their parameters via gradient descent (GD).
This procedure can be described by learning dynamics,
which links changes in the model’s predictions to the gradients generated by learning from specific examples.
With the help of learning dynamics,
researchers have not only explained many interesting phenomena during training,
but used these insights to propose novel, improved algorithms [, , ].

The study of large language models (LLM) is gaining popularity due to their surprising capabilities on various tasks.
To ensure the LLMs follow human instructions and align well with human values,
finetuning has attracted much recent attention.
Practitioners often start with instruction tuning,
where the model learns extra knowledge necessary for the downstream task (such as instruction-following),
and then preference tuning, where the model aligns its outputs to human preference [].
Various finetuning algorithms have been proposed to fit into this pipeline,
with differing explanations as to why they improve the model’s performance.

Different from most existing analyses of LLM finetuning, which use the perspective of their training targets,
their status at the end of training, or their relationships to reinforcement learning [[, e.g.]]ji2024towards, rafailov2024r, tajwar2024preference,
this paper tries to understand LLMs’ evolution from a dynamical perspective.
Specifically,
we formalize the learning dynamics of LLMs’ finetuning
using a similar decomposition of gradient updates to that of [].
Surprisingly, this framework can be easily adapted to various finetuning algorithms with different goals,
including supervised finetuning [[, SFT,]]wei2022finetuned, self-play finetuning [[, SPIN,]]chen2024self, direct preference optimization [[, DPO,]]rafailov2024direct, and their variants.
This framework
has the potential to explain several interesting and counter-intuitive observations during training
– including the “repeater” phenomenon after preference tuning [], hallucination(^1^11Especially the case that the model uses facts or phrases that appear in the responses of question B to answer question A.) [], the decay in confidence of *all* responses during off-policy DPO [], and more.

Moreover, we also provide a new perspective on understanding why off-policy DPO and other variants underperform their on-policy counterpart [].
Our explanation starts by observing an interesting “squeezing effect,”
which we demonstrate is a consequence of gradient *ascent* (as in DPO and similar algorithms)
on models with cross-entropy loss following a softmax layer.
In short,
for each token’s prediction,
the negative gradient will push down the model’s predictions on (almost) all possible output labels,
moving this probability mass to the most-likely labels.
This can be detrimental to the alignment we are trying to achieve.
This effect is most serious when the negative gradient is imposed
on an already-unlikely label,
which is why the confidence of almost all responses decreases during off-policy DPO.
Inspired by our analysis,
we propose a simple, counter-intuitive, but very effective method to further improve alignment performance.

## 2 Background and Related Works

### 2.1 Learning dynamics of deep learning

When studying a practical machine learning algorithm,
we usually care about how the learned model $f_{\theta}$ is influenced by different factors,
e.g., the model’s parameter $\theta$, the training dataset, the learning hyperparameters, etc.
Broadly speaking, “learning dynamics” is usually used as an umbrella term to describe how the change of a specific factor influences the model’s prediction.
In this paper, we narrow down this term to describe “how the change in $\theta$ influences the corresponding change in $f_{\theta}$”,
i.e., the relationship between $\Delta\theta$ and $\Delta f_{\theta}$.

For a simple model like $f_{a,b}(x)=ax^{2}+b$,
it is straightforward to directly manipulate $\Delta\theta$ and observe the resulting $\Delta f_{\theta}$ accordingly.
However, for a deep neural network,
such an observation is impractical for many reasons.
First, a deep model usually contains an enormous number of parameters,
individual parameters are usually very difficult to interpret.
Second, the input and output spaces of the model $f_{\theta}(x)$ are usually high-dimensional,
making it hard to observe $\Delta f_{\theta}$ directly.
Combining the fact that modern deep models are usually trained via different variants of stochastic gradient descent (SGD),
where all parameters change simultaneously in each update,
we usually use the following definition to study learning dynamics:

$$ $\quad\Delta\theta\triangleq\theta^{t+1}-\theta^{t}=-\eta\cdot\nabla\mathcal{L}\left(f_{\theta}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})\right);\quad\Delta f({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}})\triangleq f_{\theta^{t+1}}({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}})-f_{\theta^{t}}({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}}).$ (1) $$

Here the update of $\theta$ during step $t\rightarrow t+1$ is given by learning the sample ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}}$ using SGD with learning rate $\eta$.
In short, the learning dynamics in this paper address the question:

After an SGD update on ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}}$, how does the model’s prediction on ${\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}}$ change?

Studying the learning dynamics defined above can shed light on many important problems in deep learning and also help to understand various counter-intuitive phenomena. Here are a few.

Understanding generalization.
If we consider ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}}$ from the training set, and ${\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}}$ from the test set, this form of learning dynamics addresses generalization:
the model generalizes better if the loss of $f_{\theta}({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}})$ keeps decreasing when it learns from ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}}$.
By studying the influence of different ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}}$ at different stages during supervised learning,
[] explain a “zigzag” pattern of the learning path,
which sheds light on why the model can spontaneously pursue better supervisory signals and correct noisy labels in the early stage of training [[, see also]]liu2020early.
[, ] apply learning dynamics to explain why directly finetuning a well-trained backbone with a randomly initialized task head might harm the out-of-distribution generalization ability.
[, ] also explains where the simplicity bias favoring compositional representations comes from during knowledge distillation [], providing a new perspective of understanding why successive knowledge transferring can improve the model’s systematic generalization ability.

Measuring the “quality” or “influence” of training samples.
Besides explaining the model’s behavior,
learning dynamics is also helpful for evaluating the quality or the effectiveness of different training samples.
For example, [] propose a quantitative metric called TracIn to compute the influence of a training example on the predictions made by the model.
This metric is then applied by [] to search for the most influential examples in LLM instruction finetuning.
By expanding [Equation 1](#S2.E1) in the neural tangent kernel (NTK) regime,
[] propose a metric called lpNTK to measure the relative difficulty among different training samples.
These metrics and analyses inspired by learning dynamics are expected to be helpful in many related fields,
like coreset selection [], active learning [] (see, e.g., []), and dataset distillation [].

### 2.2 Challenges of Analyzing the Learning Dynamics of LLM’s Finetuning

The first conundrum for analyzing the learning dynamics of LLM’s finetuning is the high dimensionality and the sequence nature of both the input and output signals.
The high-dimensional property makes it hard to observe the model’s output,
and the sequence nature makes the distributions on different tokens mutually dependent,
which is more complicated than a standard multi-label classification problem considered by most previous works.
Furthermore, as there are many different algorithms for LLM finetuning
– SFT [], RLHF [], DPO [], etc. –
analyzing them under a uniform framework is challenging.
Finally, compared with the training-from-scratch scenario, where a roughly uniform distribution over all possible outputs is usually assumed at the beginning of training,
LLMs’ finetuning dynamics heavily rely on the pretrained base model,
which could make the analysis harder.
For example,
the pretrained model usually assigns little probability mass to unlikely tokens,
which is good for most practical applications but leads to risk of the “squeezing effect” we show later.

In the remainder of the paper,
we will tackle the aforementioned problems and propose a unified framework to describe different finetuning algorithms.
We believe studying the learning dynamics provides a complementary perspective on LLM finetuning behaviors
to the usual analyses of where different algorithms eventually converge.

## 3 Learning Dynamics under Per-step and Accumulated Perspectives

We first consider a standard supervised learning problem,
where the model takes a high-dimensional input $\bm{\mathsf{x}}$ and generates a sequence response $\bm{\mathsf{y}}=\{y_{1},\dots,y_{L}\}\in\mathcal{V}^{L}$,
where $\mathcal{V}$ is the vocabulary of size $V$.
We assume we actually define a distribution over $\bm{\mathsf{y}}$,
i.e., $\pi_{\theta}(\bm{\mathsf{y}}\mid\bm{\mathsf{x}})$.
To get this probabilistic distribution,
the model first generates a logits matrix $\bm{\mathsf{z}}=h_{\theta}(\bm{\mathsf{x}})\in\mathbb{R}^{V\times L}$ and then takes the $\operatorname{\mathsf{Softmax}}$ operation on each column.
To make the analysis for the LLM case easier,
we track the change of $\log\pi_{\theta^{t}}$ instead of $\pi_{\theta^{t}}$,
slightly different from the results of [].

##### Per-step influence decomposition.

The learning dynamics of ([1](#S2.E1)) become

$$ $\displaystyle\Delta\theta\triangleq\theta^{t+1}-\theta^{t}=-\eta\cdot\nabla\mathcal{L}\left(\pi_{\theta}(\bm{\mathsf{y}}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}}),{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{y}}_{u}}\right)$ (2) $\displaystyle\Delta\log\pi^{t}(\bm{\mathsf{y}}\mid{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}})\triangleq\log\pi_{\theta^{t+1}}(\bm{\mathsf{y}}\mid{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}})-\log\pi_{\theta^{t}}(\bm{\mathsf{y}}\mid{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}}),$ (3) $$

where ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{y}}_{u}}\in\mathbb{R}^{V}$ (likely one-hot) is the supervisory signal.
To get a better intuition,
we start from the $L=1$ scenario,
where the $\Delta\theta$ and $\Delta\pi$ can be linked by the following proposition.

[Similar to Proposition 1 of []]propdynamicsdecompose
Let $\pi=\operatorname{\mathsf{Softmax}}(\bm{\mathsf{z}})$ and $\bm{\mathsf{z}}=h_{\theta}(\bm{\mathsf{x}})$. The one-step learning dynamics decompose as

$$ $\underbrace{\Delta\log\pi^{t}(\bm{\mathsf{y}}\mid{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}})}_{V\times 1}=-\eta\underbrace{\mathcal{A}^{t}({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}})}_{V\times V}\underbrace{\mathcal{K}^{t}({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}},{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})}_{V\times V}\underbrace{\mathcal{G}^{t}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}},{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{y}}_{u}})}_{V\times 1}{}+O(\eta^{2}\|\nabla_{\theta}\bm{\mathsf{z}}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})\|^{2}_{\mathrm{op}}),$ (4) $$

where $\mathcal{A}^{t}({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}})=\nabla_{\bm{\mathsf{z}}}\log\pi_{\theta^{t}}({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}})=I-\bm{\mathsf{1}}\pi_{\theta^{t}}^{\top}({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}})$, $\mathcal{K}^{t}({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}},{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})=(\nabla_{\theta}\bm{\mathsf{z}}({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}})|_{\theta^{t}})(\nabla_{\theta}\bm{\mathsf{z}}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})|_{\theta^{t}})^{\top}$ is the empirical neural tangent kernel of the logit network
and $\mathcal{G}^{t}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}},{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{y}}_{u}})=\nabla_{\bm{\mathsf{z}}}\mathcal{L}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}},{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{y}}_{u}})|_{\bm{\mathsf{z}}^{t}}$,
which for cross-entropy loss is $\pi_{\theta^{t}}(\bm{\mathsf{y}}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})-{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{y}}_{u}}$.
The proof and more discussion can be found in [Appendix A](#A1).

In this decomposition,
$\mathcal{A}^{t}({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}})=I-\bm{\mathsf{1}}\pi_{\theta^{t}}^{\top}({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}})$,
which only depends on the model’s predicted probability at time $t$.
Intuitively,
for any length-$V$ vector $\bm{\mathsf{b}}$, left-multiplying this $V\times V$ matrix $\mathcal{A}^{t}$ centers the vector with respect to the probability distribution $\pi_{\theta^{t}}({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}})$,
because $\mathcal{A}^{t}\bm{\mathsf{b}}=(I-\bm{\mathsf{1}}\pi^{\top})\bm{\mathsf{b}}=\bm{\mathsf{b}}-\bm{\mathsf{1}}(\pi^{\top}\bm{\mathsf{b}})$,
and hence the mean of $\mathcal{A}^{t}\bm{\mathsf{b}}$ under $\pi$ is $\pi^{\top}\bm{\mathsf{b}}=0$.

The matrix $\mathcal{K}^{t}$ is the empirical neural tangent kernel [[, eNTK,]]NTK,linntk2019lee of the model,
i.e., the product of the model’s gradients with respect to ${\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}}$ and ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}}$.
Since the eNTK usually changes slowly (or is even nearly invariant) when finetuning with a small learning rate [, , , ],
we can treat $\mathcal{K}^{t}$ as a relatively stable and model-specific similarity measurement between different input samples:
larger $\|\mathcal{K}^{t}\|$ means the update of ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}}$ can influence model’s prediction on ${\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}}$ more.
The term $\mathcal{G}^{t}$ is determined by the loss function $\mathcal{L}$,
which usually provides the energy and direction for the model’s adaptation.
For example,
consider a cross-entropy loss $\mathcal{L}_{\text{CE}}\triangleq-{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{y}}_{u}}\cdot\log\pi(\bm{\mathsf{y}}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})$.
Then we have $\mathcal{G}_{\text{CE}}^{t}=\pi_{\theta^{t}}(\bm{\mathsf{y}}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})-{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{y}}_{u}}$,
a length-$V$ vector that points from the model’s current predictive distribution to the desired supervisory distribution.
When considering one-hot labels,
the above term can also be written as $\mathcal{G}_{\text{CE}}^{t}=\pi_{\theta^{t}}(\bm{\mathsf{y}}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})-\bm{\mathsf{e}}_{y_{u}}$,
which is more common in practice.

For a multi-label classification problem where $L>1$,
to this first-order approximation,
we only need to calculate $L$ different $\Delta\log\pi^{t}$ and stack them together.

##### Accumulated influence: a simple example on MNIST.

[Equation 4](#S3.E4),
which states how the update of ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}}$ changes the model’s prediction on ${\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}}$,
provides us with a powerful tool for understanding the mutual influence between the update of different samples during training.

As a warm-up,
we first analyze an example of training a LeNet on the MNIST dataset [].
See [Figure 1](#S3.F1)-(a),
where the network $\pi_{\theta^{t}}$ is updating its parameters using the loss calculated on one training example $({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}},{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{y}}_{u}}=\bm{\mathsf{e}}_{\texttt{4}})$, where $\bm{\mathsf{e}}_{\texttt{4}}$ is the one-hot vector corresponding to a label of class 4.
The residual term $\mathcal{G}^{t}_{\text{CE}}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}},{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{y}}_{u}})$ is then represented by the red arrows,
which all start from $\pi_{\theta^{t}}(\bm{\mathsf{y}}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})$ and point to ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{y}}_{u}}$.
We can then ask how the model’s prediction on different ${\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}}$ changes after this update.
As illustrated in [Figure 1](#S3.F1)-(b),
for an ${\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}}$ in the same class with ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}}$ (i.e., the identical case),
the predicted probability of this correct label is “pulled up” by this update, as expected.
Supposing this ${\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}}$ is from the test set,
this update makes the model more confident in correctly predicting an unseen sample.
On the other hand,
if this ${\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}}$ is similar to ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}}$ (i.e., $\|\mathcal{K}^{t}\|$ is reasonably large) but comes from another class,
then the predicted probability on ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}}$’s class (which is not the correct label of ${\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}}$) would be “pulled up.”
Last, for examples that look dissimilar to ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}}$ ($\lVert\mathcal{K}^{t}\rVert$ is small),
this update will not change the model’s prediction on ${\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}}$ much,
as illustrated in the bottom panel in [Figure 1](#S3.F1)-(b).

Figure: Figure 1: The per-step learning dynamics and the accumulated influence in an MNIST experiment.
Refer to caption: /html/2407.10490/assets/x1.png

The interactions among the updates of different inputs then form an interesting pattern for the learned predictions.
As illustrated in [Figure 1](#S3.F1)-(c),
when making predictions on images coming from class 4,
the model tends to assign higher confidence on class 9.
That is because the examples in class 9 on average look more similar to class 4 than examples in other classes.
Hence the update of examples in classes 4 and 9 will reinforce their mutual influence and lead to a bump in their predictions.
To further verify this,
we plot the average value of $\pi(\bm{\mathsf{y}}\mid\bm{\mathsf{x}})$ for $\bm{\mathsf{x}}$ from each of the classes in [Figure 1](#S3.F1)-(d).
In this heatmap,
the indexes of the rows denote which class the images come from.
The values of some off-diagonal patches are significantly higher than others,
which means the examples in the corresponding classes look more similar,
like 4 and 9, 5 and 3, 8 and 5, etc.

## 4 Learning Dynamics of LLM Supervised Finetuning

##### Per-step decomposition of the SFT loss.

We are now ready to tackle the supervised finetuning (SFT) of an LLM.
The typical loss function used in this stage is the negative log-likelihood (NLL) of a given completion $\bm{\mathsf{y}}^{+}_{u}=(y^{+}_{1},\dots,y^{+}_{L})\in\mathcal{V}^{L}$, conditioned on the prompt ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}}$:

$$ $\mathcal{L}_{\text{SFT}}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}},\bm{\mathsf{y}}_{u}^{+})\triangleq-\sum_{l=1}^{L}\log\pi(y=\bm{\mathsf{y}}_{l}^{+}\mid\bm{\mathsf{y}}^{+}_{1:l-1},{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})=-\sum_{l=1}^{L}\bm{\mathsf{y}}_{l}^{+}\cdot\log\pi(\bm{\mathsf{y}}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}},\bm{\mathsf{y}}^{+}_{1:l-1}).$ (5) $$

Note that compared with the multi-label classification problem discussed before,
where the joint distribution of all labels can be factorized as $\pi(\bm{\mathsf{y}}\mid\bm{\mathsf{x}})=\prod_{l}\pi(y_{l}\mid\bm{\mathsf{x}})$,
the sequential nature of language modeling makes the analysis more complicated,
because we must have $\pi(\bm{\mathsf{y}}\mid\bm{\mathsf{x}})=\prod_{l}\pi(y_{l}\mid\bm{\mathsf{x}},\bm{\mathsf{y}}_{1:l-1})$.
Note that this is the correct form, and not one conditioning on the previous model predictions,
due to the “teacher forcing” scheme
typically used in training sequence models.
To solve this problem,
we can merge this factorization into the definition of the backbone $h_{\theta}$
while keeping the decomposition format of [Equation 4](#S3.E4).
Specifically,
define ${\bm{\mathsf{\chi}}}\triangleq[\bm{\mathsf{x}};\bm{\mathsf{y}}]$.
Then for the prediction of all tokens of $\bm{\mathsf{y}}$,
we have

$$ $\bm{\mathsf{z}}=h_{\theta}\left(\bm{\mathsf{\chi}}\right);\quad\pi\left(\bm{\mathsf{y}}\mid\bm{\mathsf{\chi}}\right)=\operatorname{\mathsf{Softmax_column}}\left(\bm{\mathsf{z}}\right).$ $$

Note that $\bm{\mathsf{z}}$ is a $V\times L$ matrix where each column contains the logits of the prediction of the $l$-th token.
Our $h_{\theta}$,
even taking the entire sequence ${\bm{\mathsf{\chi}}}$ as its input,
will force the model not to refer to the future tokens $\bm{\mathsf{y}}_{>l}$ when making predictions on the $l$-th token,
which is commonly implemented via “causal masking” [[, e.g.]]allyouneed.
Then,
we can calculate $(\nabla_{\theta}\bm{\mathsf{z}}_{l}({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{\chi}}_{o}})|_{\theta^{t}})(\nabla_{\theta}\bm{\mathsf{z}}_{l}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{\chi}}_{u}})|_{\theta^{t}})^{\top}$ on each column of $\bm{\mathsf{z}}$ and then stack them to form a $V\times V\times L$ tensor $\mathcal{K}^{t}({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{\chi}}_{o}},{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{\chi}}_{u}})$.
The calculation of $\mathcal{G}^{t}$ and $\mathcal{A}^{t}$ also follows a similar procedure.
Thanks to the causal mask implemented in $h_{\theta}$,
the resulting decomposition is almost identical to that in a multi-label classification problem.

Assume now we have a response ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{y}}_{u}}$ of length $L$ associated with ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}}$,
stacked into ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{\chi}}_{u}}$,
and ${\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{y}}_{o}}$ of length $M$ associated with ${\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}}$, stacked into ${\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{\chi}}_{o}}$.
The change of the model’s prediction on the $m$-th token of $\bm{\mathsf{y}}_{o}$
can be represented as

$$ $[\underbrace{\Delta\log\pi^{t}(\bm{\mathsf{y}}\mid{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{\chi}}_{o}})}_{V\times M}]_{m}=-\sum_{l=1}^{L}\eta[\underbrace{\mathcal{A}^{t}({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{\chi}}_{o}})}_{V\times V\times M}]_{m}[\underbrace{\mathcal{K}^{t}({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{\chi}}_{o}},{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{\chi}}_{u}})}_{V\times V\times L}]_{l}[\underbrace{\mathcal{G}^{t}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{\chi}}_{u}})}_{V\times L}]_{l}+O(\eta^{2})$ (6) $$

Compared with [Equation 4](#S3.E4),
the main difference is that the eNTK term also depends on the responses ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{y}}_{u}}$ and ${\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{y}}_{o}}$,
which allows us to answer questions like “how learning $(\bm{\mathsf{x}}_{u},\bm{\mathsf{y}}_{u}^{+})$ influences the model’s belief about $(\bm{\mathsf{x}}_{u},\bm{\mathsf{y}}_{u}^{-})$”.
Combined with the SFT loss,
the residual term is

$$ $\mathcal{G}^{t}_{\text{SFT}}\left({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{\chi}}_{u}}\right)=\nabla_{\bm{\mathsf{z}}}\mathcal{L}_{\text{SFT}}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{\chi}}_{u}})|_{\bm{\mathsf{z}}^{t}}=\underbrace{\pi_{\theta^{t}}(\bm{\mathsf{y}}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{\chi}}_{u}})}_{V\times L}-\underbrace{{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{y}}_{u}}}_{V\times L},$ (7) $$

almost identical to that of a multi-label classification problem.

### 4.1 Response Space and the Probing Dataset

Besides the sequential nature of the loss function,
another conundrum in analyzing LLM learning dynamics is the huge response space $\mathcal{Y}$:
the number of possible $\bm{\mathsf{y}}\in\mathcal{Y}$ is $V^{L}$,
but the vast majority of possible sequences look nothing like natural language,
and we expect the model to generate only a subset of natural language-like responses.
These properties prevent us from observing the changes of all possible $\bm{\mathsf{y}}$ like what we did for MNIST.
Instead, we define several interesting regions of $\mathcal{Y}$, and select corresponding typical responses to observe.
Intuitively, we can use the semantic relevance between $\bm{\mathsf{y}}$ and ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}}$ as a heuristic.
Such a measurement can be understood as “how suitable this $\bm{\mathsf{y}}$ is as a response to ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}}$, compared to $\bm{\mathsf{y}}_{u}^{+}$.”
Then, starting from the structure of common preference optimization datasets
such as
Antropic-HH []
and UltraFeedback [],
we can roughly divide $\mathcal{Y}$ into three sub-spaces and propose seven typical responses to evaluate (as in [Figure 2](#S4.F2)):

- •
$\mathcal{Y}_{\text{IF}}$: reasonable responses following the instruction ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}}$:
1.
$\bm{\mathsf{y}}_{u}^{+}$, the chosen (i.e., the preferred) response to ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}}$.
2.
$\bm{\mathsf{y}}_{u}^{-}$, the rejected (i.e., the less preferred, but still reasonable) response to ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}}$.
- •
$\mathcal{Y}_{\text{non-IF}}$: irrelevant responses to ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}}$ that are still recognizably human language (in these datasets, roughly “internet-standard” English):
3.
$\bm{\mathsf{y}}_{j\neq u}^{+}$, the chosen response for a different question $\bm{\mathsf{x}}_{j\neq u}$ selected from the training set.
4.
$\bm{\mathsf{y}}_{\text{test}}^{+}$, the chosen response of a question $\bm{\mathsf{x}}_{\text{test}}$ selected from the test set.
5.
$\bm{\mathsf{y}}_{\text{hum}}$, a “random” English sentence generated by GPT4 with as many words as $\bm{\mathsf{y}}_{u}^{+}$.
- •
$\mathcal{Y}_{\text{non-hum}}$: token sequences that do not form meaningful human language:
6.
$\bm{\mathsf{y}}_{\text{urnd}}^{+}$, a random permutation of the words (space-separated strings) of $\bm{\mathsf{y}}_{u}^{+}$.
7.
$\bm{\mathsf{y}}^{\prime}_{\text{rnd}}$, a random permutation of the words of a generated sentence as in $\bm{\mathsf{y}}_{\text{hum}}$.

Figure: Figure 2: The division of $\mathcal{Y}$ and seven typical selected responses. The small panels demonstrate how the average predicted probability of specific responses evolves during SFT. The three curves with different colors roughly represent the model’s prediction of the three stages of finetuning: before SFT ($\pi_{\theta^{0}}$), early stopping after training a few epochs ($\pi_{\theta^{\text{ES}}}$), and the overfitted case ($\pi_{\theta^{\text{T}}}$).
Refer to caption: /html/2407.10490/assets/x2.png

In [Section B.1](#A2.SS1), we also consider the Levenshtein (edit) distance to $\bm{\mathsf{y}}_{u}^{+}$ as another axis,
and track more responses accordingly.
As shown by [],
the influence between samples (roughly speaking, $\lVert\mathcal{K}^{t}\rVert$)
is better captured by semantic similarity than the superficial edit distance,
so we do not consider this metric in most of our results.

### 4.2 Experimental verification

We now verify our analysis of the learning dynamics using experiments.
We first create an SFT training set by randomly selecting 5000 examples from the training split of Antropic-HH;
we give similar results for UltraFeedback in [Section B.2](#A2.SS2).
The model will be finetuned using these $[\bm{\mathsf{x}};\bm{\mathsf{y}}^{+}]\in\mathcal{D}_{\text{train}}$ for several epochs.
To observe the learning dynamics,
we create a probing dataset by first selecting 500 prompts from $\mathcal{D}_{\text{train}}$
and then generate seven different responses for each $\bm{\mathsf{x}}$ following [Figure 2](#S4.F2).
We evaluate the log-likelihood of these responses every 25 updates (with a training batch size of 4, the probing occurs every 100 examples).
The model is finetuned for $8$ epochs,
where we call the first several epochs from $\pi_{\theta^{0}}$ to $\pi_{\theta^{\text{ES}}}$ fitting stage and the last several epochs from $\pi_{\theta^{\text{ES}}}$ to $\pi_{\theta^{\text{T}}}$ overfitting stage.
We verify the generalizability of our findings by using six models:
pythia-410M/1B/1.4B/2.8B []
and Qwen1.5-0.5B/1.8B [].

We emphasize beforehand that while the behaviors seen in this section are perhaps not particularly surprising,
they validate that our learning dynamics explanations can explain practical behavior in LLM finetuning,
and that these techniques *will* yield a surprising “squeezing” effect when applying them to preference finetuning in the next section.

Behaviors of $\bm{\mathsf{y}}\in\mathcal{Y}_{\text{IF}}$.
As illustrated in the first panel in [Figure 3](#S4.F3),
in the fitting stage,
we see SFT “pulls up” the curve of this region.
That is because their $\mathcal{K}^{t}([{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{u},\bm{\mathsf{y}}}],[{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}},{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{y}}_{u}^{+}}])$ are usually large enough,
which is similar to the pairing effect of 4 and 9 in the MNIST experiment.
As the training goes on,
the model enters into the overfitting stage,
where $\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{-})$ starts to plateau and then decrease while $\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{+})$ keeps increasing.
That is because the energy provided by residual term $\mathcal{G}^{t}_{\text{SFT}}$ always points to $\bm{\mathsf{y}}_{u}^{+}$ while deviating from other $\bm{\mathsf{y}}$.
As the training goes on and the energy in the main direction decreases (because the model’s confidence in the target increases, which makes the gap between the current prediction and the one-hot supervisions becomes smaller),
the deviating energy will gradually dominate and “push down” those regions.

Behaviors of $\bm{\mathsf{y}}\in\mathcal{Y}_{\text{non-hum}}$.
The learning dynamics for this region is simpler:
the predictions start from a very low log-likelihood and keep decreasing throughout training,
as illustrated in the last two panels in [Figure 2](#S4.F2) and the second panel in [Figure 3](#S4.F3).
The responses in this region play a similar role to the 0 examples in the MNIST experiment:
as they usually look dissimilar to 4,
hence the norms of the corresponding $\mathcal{K}^{t}$ are small,
which leads to only tiny pressure “upwards” when the model learns.
On the other hand,
as the responses of all possible $\bm{\mathsf{y}}$ given ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}}$ must sum to one,
increasing of $\pi_{\theta^{t}}(\bm{\mathsf{y}})$ on other groups must naturally “push down” the curve in this region.
Another interesting finding is that $\pi_{\theta^{t}}(\bm{\mathsf{y}}^{\prime}_{\text{rnd}})$ is always bigger than $\pi_{\theta^{t}}(\bm{\mathsf{y}}_{\text{urnd}}^{+})$,
where the latter contains all the words from $\bm{\mathsf{y}}_{u}^{+}$.
That is because learning the chosen response increases the likelihood of specific phrases, e.g., $[y_{1},y_{2}]$.
Then, in $\pi_{\theta^{t}}(\bm{\mathsf{y}}_{\text{urnd}}^{+})$, if $y_{1}$ is followed by $y_{3}\neq y_{2}$,
the resulting prediction would be very small.

Behaviors of $\bm{\mathsf{y}}\in\mathcal{Y}_{\text{non-IF}}$.
This group is the most complex one because first,
we must consider the influence coming from the update of another training example, e.g., $[\bm{\mathsf{x}}_{j\neq u};\bm{\mathsf{y}}_{j\neq u}^{+}]$.
Furthermore, although the responses in this group are irrelevant to ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}}$,
they are reasonable human language and the model will assign non-negligible probability mass to them.
That means if we want to sample a response from the model given ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}}$ (e.g., when deploying the model or doing on-policy training),
the probability that the model provides a response from (or partially from) $\mathcal{Y}_{\text{non-IF}}$ in non-negligible,
which might cause hallucinations (i.e., using the fact in problem $j$’s response to answer question $i$).

We first compare the curves of $\pi_{\theta^{t}}(\bm{\mathsf{y}}_{j\neq u}^{+})$ and $\pi_{\theta^{t}}(\bm{\mathsf{y}}_{\text{test}}^{+})$,
as illustrated in the third panel in [Figure 3](#S4.F3).
The main difference between them is that the response $\bm{\mathsf{y}}_{j\neq u}^{+}$ does occur during training, while $\bm{\mathsf{y}}_{\text{test}}^{+}$ does not.
Then based on [Equation 6](#S4.E6),
the update of $[{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{j\neq u};\bm{\mathsf{y}}_{j\neq u}^{+}}]$ will influence model’s prediction on the combined $[{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{u};\bm{\mathsf{y}}_{j\neq u}^{+}}]$.
That is why $\pi_{\theta^{t}}(\bm{\mathsf{y}}_{j\neq u}^{+})$ keeps increasing with a slower speed than $\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{+})$ all the time,
even when the model becomes overfitting.
On the contrary,
the curve of $\pi_{\theta^{t}}(\bm{\mathsf{y}}_{\text{test}}^{+})$ first increases for several updates and then begins to decrease as the model keeps training,
showing a similar trend as $\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{-})$.
As these responses never occur during training,
there is no consistent upward pressure to counter the downward pressure generated from increasing other responses.
That is why these curves go down in the overfitting stage.

Figure: Figure 3: Comparison of the learning dynamics of responses in different groups in the proposed probing dataset (the curve demonstrates the average of the likelihood of all corresponding responses). The last panel compares the inflection time of curves on $\pi_{\theta^{t}}(\bm{\mathsf{y}}_{\text{hum}})$ and $\pi_{\theta^{t}}(\bm{\mathsf{y}}_{\text{test}}^{+})$ under 10 different settings. Results for UltraFeedback are in [Figure 9](#A2.F9); results under more settings are in [Section B.2](#A2.SS2).
Refer to caption: /html/2407.10490/assets/x3.png

We now compare the nuanced difference between $\pi_{\theta^{t}}(\bm{\mathsf{y}}_{\text{test}}^{+})$ and $\pi_{\theta^{t}}(\bm{\mathsf{y}}_{\text{hum}})$,
neither of which appears during training.
They both have a reverse bell-shaped curve,
but the inflection point of $\pi_{\theta^{t}}(\bm{\mathsf{y}}_{\text{hum}})$ is consistently earlier than that of $\pi_{\theta^{t}}(\bm{\mathsf{y}}_{\text{test}}^{+})$,
as illustrated in the last two panels of [Figure 3](#S4.F3).
This phenomenon can also support our analysis:
since the strength of their downward pressure is at the same level,
reaching the plateau earlier means the upward pressure diminishes faster.
Recall that $\bm{\mathsf{y}}_{\text{test}}^{+}$ also comes from the dataset, while $\bm{\mathsf{y}}_{\text{hum}}$ is randomly generated by GPT4.
By examining these responses in $\bm{\mathsf{y}}_{\text{hum}}$,
we find that $\bm{\mathsf{y}}_{\text{test}}^{+}$ is more likely to be a “multi-turn dialog response,”
while the $\bm{\mathsf{y}}_{\text{hum}}$ is usually a descriptive sentence on a random topic (as demonstrated in [Figure 8](#A2.F8)).
In other words,
$\mathcal{K}^{t}$ for the former response is likely to be larger,
explaining the aforementioned observations.

In summary,
the learning dynamics of SFT on different responses could be roughly illustrated by the trend depicted in [Figure 2](#S4.F2):
the model’s prediction on most $\bm{\mathsf{y}}\in\mathcal{Y}_{\text{IF}}$ will be pulled up slightly at the beginning of SFT.
Then after several updates,
all the responses that never appear in the training set will gradually be pushed down.
Note that $\bm{\mathsf{y}}_{j\neq u}^{+}\in\mathcal{Y}_{\text{non-IF}}$ also consistently increases during SFT,
which might be a source of hallucination.
A more fine-grind analysis considering different rephrases of these responses and the results on different models can be found in [Appendix B](#A2).

## 5 Learning Dynamics of LLM Preference Finetuning

Instruction tuning as in the SFT stage above improves the model’s instruction-following ability.
In order to better align LLM’s output with human preferences,
typically instruction tuning is followed by preference tuning [].
The first widespread preference tuning method comes from RLHF (reinforcement learning with human feedback []),
which requires substantial expert annotation of LLM responses.
RLAIF [] offload this hard work by collecting preference signals from AI,
but a large reward model is still required.
To avoid this and mitigate the high-variance nature of the RL-based training,
[] propose DPO (direct preference optimization, an RL-free method),
and show it has the same optimization target as RLHF under the Bradley-Terry reward model [].
RL-free preference-tuning methods like DPO and variants such as IPO [], SLiC [], KTO [], and so on have gained popularity due to their good performance and robustness during training.
Significant effort [[, e.g.]]rafailov2024r, tajwar2024preference, pal2024smaug, ji2024towards has gone to theoretical explanations of the equivalence and differences between them and the RL-based methods.
This paper provides a different perspective, understanding these RL-free methods from the gradient space and the learning dynamics.
The behavior of all of these methods – even on-policy versions and the instruction tuning method SPIN [] – can be well explained by a similar pipeline as we used for analyzing SFT.
We hope the proposed framework can shed more light on how these RL-free methods work and further inspire more efficient algorithms.

### 5.1 Learning Dynamics of RL-free Preference Tuning: DPO as an Example

We start from the off-policy DPO with the following loss function:

$$ $\mathcal{L}_{\text{DPO}}(\theta)=-\sum_{({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}},{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{y}}_{u}^{+},\bm{\mathsf{y}}_{u}^{-}})}\log\sigma\left(\beta\log\frac{\pi_{\theta^{t}}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{y}}_{u}^{+}}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})}{\pi_{\text{ref}}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{y}}_{u}^{+}}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})}-\beta\log\frac{\pi_{\theta^{t}}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{y}}_{u}^{-}}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})}{\pi_{\text{ref}}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{y}}_{u}^{-}}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})}\right),$ (8) $$

where $\bm{\mathsf{y}}_{u}^{+}$ and $\bm{\mathsf{y}}_{u}^{-}$ are pre-generated responses (both in $\mathcal{Y}_{\text{IF}}$),
and $\pi_{\text{ref}}$ is the reference model, typically the result of SFT.
In the loss function,
the $\pi_{\theta^{t}}$ terms are also calculated using the “teacher forcing” mechanism,
which is identical to the SFT case.
Hence we decompose the learning dynamics for DPO similarly to [Equation 6](#S4.E6):

$$ $\displaystyle[\Delta\log\pi^{t}(\bm{\mathsf{y}}\mid{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{\chi}}_{o}})]_{m}=-\sum_{l=1}^{L}\eta[\mathcal{A}^{t}({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{\chi}}_{o}})]_{m}[\mathcal{K}^{t}({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{\chi}}_{o}},{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\tilde{\bm{\mathsf{\chi}}}_{u}})]_{l}[\mathcal{G}_{\text{DPO}}^{t}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}},{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{y}}_{u}^{+},\bm{\mathsf{y}}_{u}^{-}})]_{l}+O(\eta^{2})$ (9) $\displaystyle\mathcal{G}^{t}_{\text{DPO}}\approx\beta(1-a)\left({\bm{\mathsf{y}}_{u}^{-}}-{\bm{\mathsf{y}}_{u}^{+}}\right);\qquad a=\sigma\left(\beta\log\frac{\pi_{\theta^{t}}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{y}}_{u}^{+}}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})}{\pi_{\theta^{t}}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{y}}_{u}^{-}}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})}-\beta\log\frac{\pi_{\text{ref}}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{y}}_{u}^{+}}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})}{\pi_{\text{ref}}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{y}}_{u}^{-}}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})}\right)\in\mathbb{R}$ (10) $$

where $\mathcal{K}^{t}({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{\chi}}_{o}},{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\tilde{\bm{\mathsf{\chi}}}_{u}})$ is the equivalent eNTK between the updating sequence $[{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u};\bm{\mathsf{y}}_{u}^{+},\bm{\mathsf{y}}_{u}^{-}}]$ and the observing sequence $[{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o};\bm{\mathsf{y}}_{o}}]$.
The derivation, which is similar to before,
and the $\mathcal{G}^{t}$ functions for other RL-free methods are given in
[Section A.2.2](#A1.SS2.SSS2).

### 5.2 Comparison of SFT, DPO, and other Variants

By comparing the learning dynamics of SFT and DPO,
we find they have identical $\mathcal{A}^{t}$ and similar $\mathcal{K}^{t}$.
The main difference lies in $\mathcal{G}^{t}$,
which determines the direction and strength of the evolution of $\pi_{\theta^{t}}$.
Recall the learning dynamics of SFT in the first panel of [Figure 4](#S5.F4),
the pressures controlled by $\mathcal{G}^{t}$ contain a big positive vector on $\bm{\mathsf{y}}_{u}^{+}$ and many small negative vectors on other $\bm{\mathsf{y}}$,
where the positive vector will gradually diminish with the increase of $\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{+}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})$ during training.

The DPO loss, on the other hand,
consistently generates a pair of vectors (a positive $\bm{\mathsf{y}}_{u}^{+}$ and a negative $\bm{\mathsf{y}}_{u}^{-}$) regardless of the predictions of any policy network(^2^22Strictly, there should be some small adapting vectors (both positive and negative) on other dimensions in DPO, because the precise $\mathcal{G}^{t}_{\text{DPO}}$ should contain $(\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{+})-\bm{\mathsf{y}}_{u}^{+})-(\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{-})-\bm{\mathsf{y}}_{u}^{-})$. However, we canceled the $\pi$ terms to make the expression more concise, using the fact that the model usually pays more attention to response pairs that are not well separated yet.).
However,
as illustrated in [Equation 10](#S5.E10),
the norm of $\mathcal{G}^{t}_{\text{DPO}}$ is controlled by a scalar $\beta(1-a)$,
which is influenced by both the current policy $\pi_{\theta^{t}}$ and the reference policy $\pi_{\text{ref}}$.
We first analyze the role of $a$,
which is controlled by the margin (i.e., the value inside the parentheses of $\sigma(\cdot)$) that represents how well the current policy separates $\bm{\mathsf{y}}_{u}^{+}$ and $\bm{\mathsf{y}}_{u}^{-}$ compared with the reference policy.
Due to the monotonicity of $\sigma(\cdot)$,
a larger margin leads to larger $a$,
which in turn restrains the strength of $\mathcal{G}^{t}_{\text{DPO}}$.
In other words,
$\mathcal{G}^{t}_{\text{DPO}}$ automatically provides less energy on the examples that are already well separated.
We then check the role of $\beta$,
which controls the regularizing effect on the KL distance between $\pi_{\theta^{t}}$ and $\pi_{\text{ref}}$ in the original RL loss [].
When the margin is smaller than zero,
larger $\beta$ leads to a smaller $a$ and hence provides stronger $\mathcal{G}^{t}_{\text{DPO}}$ for the model to “catch up” the separating ability of the reference model faster.
But when the model is good enough (the margin is positive),
increasing $\beta$ will increase $a$ and hence create a negative influence on $\beta(1-a)$,
which makes the model update less.
This behavior aligns well with the claims of []:
the stronger regularizing effect tends to “drag $\pi_{\theta}$ back towards $\pi_{\text{ref}}$” when it deviates from $\pi_{\text{ref}}$ too much in terms of the separating capability.

Figure: Figure 4: Illustration of the updating vector provided by the residual term $\mathcal{G}^{t}$ of different algorithms. The gray $\bm{\mathsf{y}}$ are responses sampled from $\pi$ in an on-policy way. In the second panel, we demonstrate the “squeezing effect” caused by imposing a big negative gradient on a “valley” region of a distribution. For more details about this counter-intuitive effect, please refer to [Section 5.3](#S5.SS3) and [Appendix C](#A3). For other panels, we demonstrate SPIN [], SPPO [], and SLiC [].
Refer to caption: /html/2407.10490/assets/x4.png

Note that the analyses above don’t make any assumptions on where $\bm{\mathsf{y}}_{u}^{+}$ and $\bm{\mathsf{y}}_{u}^{-}$ come from.
Hence our framework can also be extended to on-policy RL-free algorithms,
which often perform better than their off-policy counterparts [, ].
By definition,
the main difference between off-policy and on-policy algorithms is how the supervisory responses are generated.
Off-policy methods typically use a fixed pre-collected dataset, where $\bm{\mathsf{y}}_{u}^{+}$ and $\bm{\mathsf{y}}_{u}^{-}$ are usually generated by another LLM or humans.
In other words,
it is likely that both the chosen and rejected responses come from the “less likely” region of the model’s prediction,
as illustrated in the second panel in [Figure 4](#S5.F4).
On-policy responses,
on the other hand,
are more likely to have higher predicted probabilities under this model,
as they were sampled from it.
We will show soon that imposing large negative pressure on an unlikely prediction will lead to unexpected behaviors.
This leads to a unique explanation of why on-policy sampling is so important for algorithms with large negative gradients.

### 5.3 The Squeezing Effect of the Negative Gradient

The existence of negative gradients (the $\bm{\mathsf{y}}_{u}^{-}$ term in [Equation 10](#S5.E10)) is the key to understanding the difference of learning dynamics between SFT and DPO.
We find this negative gradient will impose a non-trivial “squeezing effect” on any models outputting the probabilistic distribution using $\operatorname{\mathsf{Softmax}}$ output heads,
even in a simple multi-class logistic regression task.
Specifically,
consider the $L=1$ case and assume our algorithm imposes a negative gradient on label $\bar{y}$,
the model’s predictive distribution $\pi_{\theta^{t+1}}$ will change as follows:

- •
The negative gradient will reliably decrease the confidence of $\bar{y}$, i.e., $\pi_{\theta^{t+1}}(y=\bar{y})$ is guaranteed to decrease.
- •
The decreased probability mass is “squeezed” into the most confident dimension before the update, i.e., $\pi_{\theta^{t+1}}(y=y^{*})$, where $y^{*}=\operatorname*{argmax}_{i\in[V]\setminus\{\bar{y}\}}\pi_{\theta^{t}}(y=i)$ is guaranteed to increase.
- •
The rich get richer and the poor get poorer: generally, dimensions with high $\pi_{\theta^{t}}$ tend to increase, and those with low $\pi_{\theta^{t}}$ tend to decrease.
- •
Peakier $\pi_{\theta^{t}}$ suffer a more serious squeezing effect. If the probability mass concentrates on few dimensions in $\pi_{\theta^{t}}$, which is common for a pretrained model, all $\pi_{\theta^{t+1}}(y\neq y^{*})$ decrease (only $y^{*}$ is considered “rich”).
- •
Smaller $\pi_{\theta^{t}}(\bar{y})$ make the squeezing effect stronger. If $\bar{y}$ is unlikely under $\pi_{\theta^{t}}$, the probability mass of all other $\pi_{\theta^{t+1}}(y\neq y^{*})$ will be more seriously decreased, and the $\pi_{\theta^{t+1}}(y=y^{*})$ increases more. That is what we observed in the off-policy DPO scenario.

[Appendix C](#A3) proves these observations analytically for linear models,
by directly computing $\pi_{\theta^{t+1}}/\pi_{\theta^{t}}$ in different situations.
Intuitively, recall that
the cross-entropy loss for a one-hot label $\bm{\mathsf{e}}_{i}$ is just
$-\log\pi_{i}=-z_{i}+\log\sum_{j}\exp(z_{j})$,
which has gradient $-\bm{\mathsf{e}}_{i}+\pi$.
Suppose there is one $k\neq i$ such that $\pi_{k}$ is rather large,
say $0.8$.
Then gradient ascent will decrease $z_{i}$, but increase $z_{k}$ almost as much.
Since $\pi_{k}$ is dominant, increasing its logit results in a large absolute increase to $\pi_{k}$,
which necessarily reduces all other probabilities to account for it.

In the next subsection, we will verify these claims by experiments on real LLM finetuning problems.
With the help of these analyses,
we figure out why the model’s confidence in all responses decreases during off-policy DPO training and successfully find where the decreased probability mass has gone.

Figure: Figure 5: Learning dynamics of off-policy DPO. For experiments on different settings and the results of conducting DPO after SFT, please refer to [Section B.3](#A2.SS3).

### 5.4 Experimental Analysis of DPO’s Learning Dynamics

To verify our framework also explains the learning dynamics of preference tuning well,
we conduct similar experiments for DPO as we do for SFT.
Recall the residual term $\mathcal{G}^{t}_{\text{DPO}}$ introduces a positive arrow on $\bm{\mathsf{y}}_{u}^{+}$,
which means the learning tries to “pull up” the model’s prediction on $\bm{\mathsf{y}}_{u}^{+}$.
To verify this,
we create two types of rephrases of $\bm{\mathsf{y}}_{u}^{+}$ using GPT4 and call them $\bm{\mathsf{y}}_{\text{gpts}}^{+}$ and $\bm{\mathsf{y}}_{\text{gptf}}^{+}$ (the first tend to keep the semantics while the latter tends to keep the format of $\bm{\mathsf{y}}_{u}^{+}$; details in [Figure 6](#A2.F6)).
See the three curves in the first panel in [Figure 5](#S5.F5),
where the two rephrases decrease at a similar speed, faster than the decay of $\bm{\mathsf{y}}_{u}^{+}$.
That is because the upward pressure is directly imposed on $\bm{\mathsf{y}}_{u}^{+}$ rather than these rephrases.
Similarly,
in the second panel,
we observe that $\bm{\mathsf{y}}_{u}^{-}$ decays faster than its rephrases,
because $\mathcal{G}^{t}_{\text{DPO}}$ directly imposes a negative pressure on $\bm{\mathsf{y}}_{u}^{-}$.
Then in the third panel, we find the rephrases of $\bm{\mathsf{y}}_{u}^{+}$ consistently decay slower than those of $\bm{\mathsf{y}}_{u}^{-}$,
although none of them ever occur during training.
That is because these responses are close to $\bm{\mathsf{y}}_{u}^{+}$ or $\bm{\mathsf{y}}_{u}^{-}$ in $\mathcal{Y}$,
which means the norm of their $\mathcal{K}^{t}$ is relatively large.
Hence the pressures imposed on $\bm{\mathsf{y}}_{u}^{+}$ and $\bm{\mathsf{y}}_{u}^{-}$ also introduce a non-negligible influence on them.
Last, by observing the fourth panel,
we find the margin, i.e., $\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{+})-\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{-})$, keeps increasing,
which means the model is gaining the ability to separate $\bm{\mathsf{y}}_{u}^{+}$ and $\bm{\mathsf{y}}_{u}^{-}$ as the training goes on.

However,
although the margin keeps increasing
as training continues,
the model becomes less confident on both $\bm{\mathsf{y}}_{u}^{+}$ and $\bm{\mathsf{y}}_{u}^{-}$,
as well as on all other responses in the probing dataset.
This is different from SFT and quite counter-intuitive:
*if everything we observe is becoming less confident, where has the probability mass gone?*
The observation has been made by several recent papers [, , ](^3^33Particularly in Figure 3 of [], where $\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{+})$ and $\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{-})$ decays even faster if we SFT the model before DPO. This phenomenon can be well explained by the “squeezing effect” discussed in [Appendix C](#A3).) and can be well explained by analyzing a “squeezing effect” of a large negative gradient on $\bm{\mathsf{y}}_{u}^{-}$.
In short,
when the negative gradient is imposed on an unlikely prediction under the model (which is very likely in off-policy DPO),
the probability mass on almost all responses will be significantly pulled down,
and is “squeezed” to the token with the highest confidence.
This effect is a basic property of the cross-entropy loss and the $\operatorname{\mathsf{Softmax}}$ function,
which applies to any LLM with softmax heads.
[Section 5.3](#S5.SS3) and [Appendix C](#A3) has a full discussion.

To verify this hypothesis,
we report the log-likelihood of $\bm{\mathsf{y}}$
chosen by greedy decoding:
each token is chosen by maximizing the conditional probability given $[{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}};\bm{\mathsf{y}}_{1:l-1}^{+}]$,
where $\bm{\mathsf{y}}_{1:l-1}^{+}$ is a sub-sequence of $\bm{\mathsf{y}}_{u}^{+}$.
As illustrated by the last panel of [Figure 5](#S5.F5),
the confidence of this “teacher forcing” greedy $\bm{\mathsf{y}}$ increases very fast (from -113 to -63),
which is even faster than the increase of $\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{+})$ during SFT (from -130 to -90).
However, the tokens with the highest confidence do not necessarily form a preferred response:
it will reinforce the prior bias contained in $\theta^{0}$.
This could be a reasonable explanation of the “degeneration” reported in recent work [[, e.g.]]Holtzman2020The:
as $\pi_{\theta^{t}}$ becomes more peaky at its most confident predictions,
it is easier to sample sequences with repeated phrases.
Note that such behavior could also be understood as a special type of self-bias amplifying [],
which would bring more serious consequences if it is combined with a multiple-generation self-improving algorithm,
e.g., self-reward [], iterative DPO [], etc.

## 6 Inspirations from Learning Dynamics

##### Explaining the success of various related algorithms.

The proposed learning dynamics explain many interesting observations during SFT and DPO;
this perspective also helps explain the success of many practical algorithms.
For example,
the literature mostly credits on-policy RL-free methods with better aligning the current policy’s distribution to the reward model [],
but we can extend this explanation from the learning dynamics perspective.
As illustrated in the third panel in [Figures 4](#S5.F4) and [18](#A3.F18),
we see $\bm{\mathsf{y}}_{u}^{-}$ is sampled from the current $\pi_{\theta^{t}}$,
which is less likely to be in a valley.
Hence the squeezing effect introduced via the big negative gradient is reduced.
The benefits of SPIN [],
which is inspired by treating the LLM itself as a GAN-style system [],
can also be explained similarly:
as the negative response $\bm{\mathsf{y}}_{u}^{-}$ always comes from the model’s policy in the previous generation,
its $\pi_{\theta^{t}}$ is less likely to be very small.
The SPPO method [] even avoids imposing the negative gradient on the model by creating two updating vectors with different lengths (i.e., it improves the confidence on both $\bm{\mathsf{y}}_{u}^{+}$ and $\bm{\mathsf{y}}_{u}^{-}$, but at different strength).
The success of SLiC [],
whose loss is more complicated than DPO,
can also be explained in our framework
because its residual term $\mathcal{G}^{t}_{\text{SLiC}}$ has a similar form as $\mathcal{G}^{t}_{\text{DPO}}$,
as in [Equation 24](#A1.E24).
The threshold $\delta$ not only guarantees the $\pi_{\theta^{t}}$ can learn to separate all $\bm{\mathsf{y}}_{u}^{+}$ and $\bm{\mathsf{y}}_{u}^{-}$ pairs in the training set with a sufficiently large margin,
it also stops imposing a big negative gradient on $\bm{\mathsf{y}}_{u}^{-}$ when the model already has a good separation,
where $\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{-})$ is likely to be very small.
In short, although there are many more RL-free methods and we cannot elaborate on them all in this paper,
we believe the analysis proposed can shed more light on their behaviors.

##### A simple yet effective method.

We also propose a simple practical method to verify our theory further.
Since the “squeezing effect” of the big negative gradient on unlikely predictions can damage the model’s performance,
we can first train the model on *both* $[\bm{\mathsf{x}}_{u};\bm{\mathsf{y}}_{u}^{+}]$ and $[\bm{\mathsf{x}}_{u};\bm{\mathsf{y}}_{u}^{-}]$ during the SFT stage (making the negative response *more* likely), and then run the usual DPO.
Following the analysis of [Section 4](#S4),
we can expect the $\mathcal{Y}_{\text{IF}}$ region,
including $\bm{\mathsf{y}}_{u}^{-}$,
to be pulled up more than with the baseline SFT.
To compare, we consider a baseline of
standard off-policy DPO (i.e., SFT only on $[\bm{\mathsf{x}}_{u};\bm{\mathsf{y}}_{u}^{+}]$, then DPO).
Compared with the baseline,
we find the proposed method
has more stable learning dynamics that match our analysis well,
generates fewer “degenerate” responses,
and generates responses that are preferred to those of the baseline almost 70% of the time.
[Appendix D](#A4) has more details.
In the future, this simple method inspired by our analysis could be further improved by introducing more responses,
e.g., rephrases of $\bm{\mathsf{y}}_{u}^{+}$, etc.,
during both stages,
and also by combining with many existing methods we mentioned before.

## 7 Conclusion

Learning dynamics,
which depicts how the model’s prediction changes when it learns new examples,
provide a powerful tool to analyze the behavior of models trained with gradient descent.
To better use this tool in the context of LLM finetuning,
we first derive the step-wise decomposition of LLM finetuning for various common algorithms.
Then, we propose a unified framework for understanding LLM predictions’ behaviors across different finetuning methods.
The proposed analysis successfully explains various phenomena during LLM’s instruction tuning and preference tuning,
some of them are quite counter-intuitive.
We also shed light on how specific hallucinations are introduced in the SFT stage, as previously observed [],
and where the improvements of some new RL-free algorithms come from compared with the default off-policy DPO.
Finally,
inspired by this analysis,
we propose a simple (but counter-intuitive) method that is effective in improving the alignment of models.

## Acknowledgements

This research was enabled in part by support provided by the Canada CIFAR AI Chairs program, WestGrid, and Compute Canada.
We also acknowledge the inspiring discussion with Hamed Shirzad and Shangmin Guo.

[sorting=nyt]

## Appendix A Proof of Propositions and Residual Term for Different Losses

### A.1 Proof of Equation 4

*

###### Proof.

Suppose we want to observe the model’s prediction on an “observing example” ${\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}}$.
Starting from [Equation 3](#S3.E3),
we first approximate $\log\pi^{t+1}(\bm{\mathsf{y}}\mid{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}})$ using first-order Taylor expansion (we use $\pi^{t}$ to represent $\pi_{\theta^{t}}$ here for notation conciseness):

$$ $\log\pi^{t+1}(\bm{\mathsf{y}}\mid{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}})=\log\pi^{t}(\bm{\mathsf{y}}\mid{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}})+\ \langle\nabla\log\pi^{t}(\bm{\mathsf{y}}\mid{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}}),\ \theta^{t+1}-\theta^{t}\rangle+O(\|\theta^{t+1}-\theta^{t}\|^{2}).$ $$

Then, assuming the model updates its parameters using SGD calculated by an “updating example” $({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}},{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{y}}_{u}})$,
we can rearrange the terms in the above equation to get the following expression:

$$ $\Delta\log\pi^{t}(\bm{\mathsf{y}}\mid{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}})=\underbrace{\log\pi^{t+1}(\bm{\mathsf{y}}\mid{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}})}_{V\times 1}-\underbrace{\log\pi^{t}(\bm{\mathsf{y}}\mid{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}})}_{V\times 1}=\underbrace{\nabla_{\theta}\log\pi^{t}(\bm{\mathsf{y}}\mid{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}})|_{\theta^{t}}}_{V\times d}\underbrace{\big{(}\theta^{t+1}-\theta^{t}\big{)}}_{d\times 1}{}+O\big{(}\lVert\theta^{t+1}-\theta^{t}\rVert^{2}\big{)},$ $$

where $d$ is the number of parameters of the model.
To evaluate the leading term,
we plug in the definition of SGD
and repeatedly use the chain rule:

$$ $\displaystyle\underbrace{\nabla_{\theta}\log\pi^{t}(\bm{\mathsf{y}}\mid{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}})|_{\theta^{t}}}_{V\times d}\underbrace{\big{(}\theta^{t+1}-\theta^{t}\big{)}}_{d\times 1}$ $\displaystyle=\big{(}\underbrace{\nabla_{\bm{\mathsf{z}}}\log\pi^{t}({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}})|_{\bm{\mathsf{z}}^{t}}}_{V\times V}\underbrace{\nabla_{\theta}\bm{\mathsf{z}}^{t}({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}})|_{\theta^{t}}}_{V\times d}\big{)}\big{(}-\eta\underbrace{\nabla_{\theta}\mathcal{L}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})|_{\theta^{t}}}_{1\times d}\big{)}^{\mathsf{T}}$ $\displaystyle=\underbrace{\nabla_{\bm{\mathsf{z}}}\log\pi^{t}({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}})|_{\bm{\mathsf{z}}^{t}}}_{V\times V}\underbrace{\nabla_{\theta}\bm{\mathsf{z}}^{t}({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}})|_{\theta^{t}}}_{V\times d}\big{(}\underbrace{-\eta\nabla_{\bm{\mathsf{z}}}\mathcal{L}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})|_{\bm{\mathsf{z}}^{t}}}_{1\times V}\underbrace{\nabla_{\theta}\bm{\mathsf{z}}^{t}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})|_{\theta^{t}}}_{V\times d}\big{)}^{\mathsf{T}}$ $\displaystyle=-\eta\underbrace{\nabla_{\bm{\mathsf{z}}}\log\pi^{t}({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}})|_{\bm{\mathsf{z}}^{t}}}_{V\times V}\big{[}\underbrace{\nabla_{\theta}\bm{\mathsf{z}}^{t}({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}})|_{\theta^{t}}}_{V\times d}\underbrace{\left(\nabla_{\theta}\bm{\mathsf{z}}^{t}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})|_{\theta^{t}}\right)^{\mathsf{T}}}_{d\times V}\big{]}\underbrace{\big{(}\nabla_{\bm{\mathsf{z}}}\mathcal{L}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})|_{\bm{\mathsf{z}}^{t}}\big{)}^{\mathsf{T}}}_{V\times 1}$ $\displaystyle=-\eta\mathcal{A}^{t}({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}})\mathcal{K}^{t}({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}},{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})\mathcal{G}^{t}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}},{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{y}}_{u}})$ (11) $$

For the higher-order term, using as above that

$$ $\theta^{t+1}-\theta^{t}=-\eta\nabla_{\theta}\bm{\mathsf{z}}^{t}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})|_{\theta^{t}}^{\mathsf{T}}\mathcal{G}^{t}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}},\hat{\bm{\mathsf{y}}})$ $$

and noting that, since the residual term $\mathcal{G}^{t}$ is usually bounded (and the practical algorithms will also use gradient clip to avoid too large gradient),
we have that

$$ $O\big{(}\lVert\theta^{t+1}-\theta^{t}\rVert^{2}\big{)}=O\big{(}\eta^{2}\,\lVert\left(\nabla_{\theta}\bm{\mathsf{z}}^{t}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})|_{\theta^{t}}\right)^{\mathsf{T}}\rVert_{\mathrm{op}}^{2}\,\lVert\mathcal{G}^{t}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}},\hat{\bm{\mathsf{y}}})\rVert^{2}\big{)}=O\big{(}\eta^{2}\lVert\nabla_{\theta}\bm{\mathsf{z}}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})\rVert_{\mathrm{op}}^{2}\big{)}.\qed$ $$

In the decomposition,
using $\{\pi_{1},\dots,\pi_{V}\}$ to represent the model’s prediction on different dimensions,
we can write our $\mathcal{A}^{t}$ as:

$$ $\mathcal{A}^{t}({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}})=I-\bm{\mathsf{1}}(\pi^{t})^{\top}=\left[\begin{matrix}1-\pi_{1}&-\pi_{1}&\cdots&-\pi_{1}\\ -\pi_{2}&1-\pi_{2}&\cdots&-\pi_{2}\\ \dots&\dots&\ddots&\dots\\ -\pi_{V}&-\pi_{V}&\cdots&1-\pi_{V}\end{matrix}\right],$ (12) $$

The second term in this decomposition, $\mathcal{K}^{t}({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}},{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})$, is the product of gradients at ${\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}}$ and ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}}$.
Intuitively, if their gradients have similar directions, this matrix is large, and vice versa.
This matrix is known as the empirical neural tangent kernel,
and it can change through the course of training as the network’s notion of “similarity” evolves.
For appropriately initialized very wide networks trained with very small learning rates, $\mathcal{K}^{t}$ remains almost constant during the course of training,
the kernel it converges to is known as the neural tangent kernel [, ].
Note that the assumption that $\mathcal{K}^{t}({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}},{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})$ is unchanged (usually used in theoretical analysis) might be too strong in the LLM’s finetuning.
However,
as stated in many related works [, ],
assuming $\mathcal{K}^{t}({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}},{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})$ as a slow-change and almost consistent similarity measurement between ${\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}}$ and ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}}$ from the model’s perspective is reasonable.

###### Proof.

### A.2 Residual Term for Different LLM Finetuning Algorithms

As stated in [Section 4](#S4),
one of the conundrums of decomposing the learning dynamics of LLM is its auto-regression nature of the output sequence.
Different from the multi-label classification problem,
where $\bm{\mathsf{y}}_{l}$ for different $l$ is independently generated as long as the shared network is fixed,
the $\bm{\mathsf{y}}_{l}$ for the LLM’s output depends on $\bm{\mathsf{y}}_{1:l-1}$,
which is usually sampled from the model’s prediction iteratively.
However,
in most of the finetuning cases where the supervisory signal ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{y}}_{u}}$ is given,
the model will apply the so-called “teacher forcing” mechanism when calculating the predicting probabilities.
In other words,
when generating the output of each $\bm{\mathsf{y}}_{l}$,
the $\bm{\mathsf{y}}_{1:l-1}$ is given rather than sampled on-policy.
This mechanism makes it possible for us to define $\bm{\mathsf{\chi}}=[\bm{\mathsf{x}};\bm{\mathsf{y}}]$ and hence merge the auto-regressive nature of the sequence prediction into the shared $\mathcal{K}^{t}({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{\chi}}_{o}},{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{\chi}}_{u}})$.
After this step,
the decomposition of LLM’s finetuning learning dynamics then becomes similar to a multi-label classification task.

#### A.2.1 Instruction finetuning using auto-regression loss (SFT)

Here we derive the residual term, i.e., $\mathcal{G}^{t}$ for different algorithms in LLM’s finetuning.
We first rewrite [Equation 6](#S4.E6) here:

$$ $[\underbrace{\Delta\log\pi^{t}(\bm{\mathsf{y}}\mid{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{\chi}}_{o}})}_{V\times M}]_{m}=-\sum_{l=1}^{L}\eta[\underbrace{\mathcal{A}^{t}({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{\chi}}_{o}})}_{V\times V\times M}]_{m}[\underbrace{\mathcal{K}^{t}({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{\chi}}_{o}},{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{\chi}}_{u}})}_{V\times V\times L}]_{l}[\underbrace{\mathcal{G}^{t}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{\chi}}_{u}})}_{V\times L}]_{l}+O(\eta^{2}),m\in\{1,\dots,M\},l\in\{1,\dots,L\},$ $$

where $\mathcal{G}^{t}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{\chi}}_{u}})=\nabla_{\bm{\mathsf{z}}}\mathcal{L}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{\chi}}_{u}})|_{\bm{\mathsf{z}}^{t}}$ is a $V\times L$ matrix.
As the auto-regression nature of the SFT loss is already encoded in the causal mask used in $h_{\theta}$,
the columns in $\mathcal{G}^{t}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{\chi}}_{u}})$ are independent of each other, which can be separately calculated.
Plus, the summation over $l$ can also be achieved by left-multiplying a length-$L$ all-one vector $\bm{\mathsf{1}}$.
Specifically,
the SFT loss for each $l$ is:

$$ $[\mathcal{L}_{\text{SFT}}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{\chi}}_{u}})]_{l}=-\log\pi(\bm{\mathsf{y}}_{l}=\hat{\bm{\mathsf{y}}}_{l}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{\chi}}_{u}})=-\bm{\mathsf{e}}_{\hat{\bm{\mathsf{y}}}_{l}}^{\top}\log\pi(\bm{\mathsf{y}}_{l}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{\chi}}_{u}})=-\bm{\mathsf{e}}_{\bm{\mathsf{y}}_{u}^{+}}^{\top}\log\left(\texttt{Softmax}(\bm{\mathsf{z}}_{l})\right).$ $$

The gradient of $\mathcal{L}$ on $\bm{\mathsf{z}}$ can be then calculated as:

$$ $\displaystyle[\mathcal{G}_{\text{SFT}}^{t}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{\chi}}_{u}})]_{l}$ $\displaystyle=\underbrace{\nabla_{\bm{\mathsf{z}}_{l}}[\mathcal{L}_{\text{SFT}}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{\chi}}_{u}})]_{l}}_{1\times V}=\left(\underbrace{\nabla_{\pi}[\mathcal{L}_{\text{SFT}}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{\chi}}_{u}})]_{l}}_{V\times 1}\right)^{\top}\underbrace{\nabla_{\bm{\mathsf{z}}_{l}}\pi}_{V\times V}$ (13) $\displaystyle=-\left(\bm{\mathsf{e}}_{\bar{\bm{\mathsf{y}}}_{l}}\oslash\pi\right)^{\top}[\mathcal{A}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{\chi}}_{u}})]_{l}=\pi(\bm{\mathsf{y}}_{l}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{\chi}}_{u}})-\bm{\mathsf{e}}_{\bar{\bm{\mathsf{y}}}_{l}},$ $$

where $\oslash$ is element-wise division and $[\mathcal{A}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{\chi}}_{u}})]_{l}$ has a form like [Equation 12](#A1.E12).
By stacking the terms with different $l\in[L]$,
we can get

$$ $\mathcal{G}^{t}_{\text{SFT}}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{\chi}}_{u}})=\nabla_{\bm{\mathsf{z}}}\mathcal{L}_{\text{SFT}}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{\chi}}_{u}})|_{\bm{\mathsf{z}}^{t}}=\pi_{\theta^{t}}(\bm{\mathsf{y}}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{\chi}}_{u}})-\bm{\mathsf{e}}_{\bm{\mathsf{y}}_{u}^{+}},$ (14) $$

as stated in [Equation 7](#S4.E7).

#### A.2.2 Different preference finetuning algorithms

Direct Preference Optimization (DPO []) is usually considered the first RL-free alignment algorithm for preference finetuning.
Different from the standard RLHF (reinforcement learning with human feedback []),
the training of off-policy DPO is more similar to SFT,
where the model keeps learning from a pre-generated preference dataset.
We hence start from DPO to analyze the learning dynamics of different preference finetuning algorithms (actually, the on-policy versions of these algorithms could also be explained using the proposed framework).

Following [],
the training loss of DPO is:

$$ $\mathcal{L}_{\text{DPO}}(\theta)=-\sum_{({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}},\bm{\mathsf{y}}_{u}^{+},\bm{\mathsf{y}}_{u}^{-})}\log\sigma\left(\beta\log\frac{\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{+}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})}{\pi_{\text{ref}}(\bm{\mathsf{y}}_{u}^{+}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})}-\beta\log\frac{\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{-}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})}{\pi_{\text{ref}}(\bm{\mathsf{y}}_{u}^{-}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})}\right).$ (15) $$

Before calculating the residual term $\mathcal{G}^{t}_{\text{DPO}}$,
we need to re-calculate the learning dynamics decomposition,
because the loss term now depends on both $\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{+}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})$ and $\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{-}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})$,
which involves two different $\bm{\mathsf{z}}$ terms.
Specifically,
we define $\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{+}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})=\operatorname{\mathsf{Softmax_column}}(\bm{\mathsf{z}}^{+})$ and $\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{-}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})=\operatorname{\mathsf{Softmax_column}}(\bm{\mathsf{z}}^{-})$,
where $\bm{\mathsf{z}}^{+}=h_{\theta}(\bm{\mathsf{\chi}}_{u}^{+})$ and $\bm{\mathsf{z}}^{-}=h_{\theta}(\bm{\mathsf{\chi}}_{u}^{-})$ respectively ($\bm{\mathsf{\chi}}_{u}^{+}=[{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}};\bm{\mathsf{y}}_{u}^{+}]$ and $\bm{\mathsf{\chi}}_{u}^{-}=[{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}};\bm{\mathsf{y}}_{u}^{-}]$).
Then, considering the $L=1$ case, [Equation 11](#A1.E11) for the DPO loss could be written as:

$$ $\displaystyle\underbrace{\nabla_{\theta}\log\pi^{t}({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{\chi}}_{o}})|_{\theta^{t}}}_{V\times d}\underbrace{\Delta\theta^{t}}_{d\times 1}$ $\displaystyle=\big{(}\underbrace{\nabla_{\bm{\mathsf{z}}}\log\pi^{t}({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{\chi}}_{o}})|_{\bm{\mathsf{z}}^{t}}}_{V\times V}\underbrace{\nabla_{\theta}\bm{\mathsf{z}}^{t}({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{\chi}}_{o}})|_{\theta^{t}}}_{V\times d}\big{)}\big{(}-\eta\underbrace{\nabla_{\theta}\mathcal{L}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}},\bm{\mathsf{y}}_{u}^{+},\bm{\mathsf{y}}_{u}^{-})|_{\theta^{t}}}_{1\times d}\big{)}^{\mathsf{T}}$ $\displaystyle=\underbrace{\nabla_{\bm{\mathsf{z}}}\log\pi^{t}({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{\chi}}_{o}})|_{\bm{\mathsf{z}}^{t}}}_{V\times V}\underbrace{\nabla_{\theta}\bm{\mathsf{z}}^{t}({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{\chi}}_{o}})|_{\theta^{t}}}_{V\times d}\big{(}\underbrace{-\eta\nabla_{[\bm{\mathsf{z}}^{+};\bm{\mathsf{z}}^{-}]}\mathcal{L}|_{\bm{\mathsf{z}}^{t}}}_{1\times 2V}\underbrace{\left[\nabla_{\theta}\bm{\mathsf{z}}^{+}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{\chi}}_{u}^{+}});\nabla_{\theta}\bm{\mathsf{z}}^{-}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{\chi}}_{u}^{-}})\right]|_{\theta^{t}}}_{2V\times d}\big{)}^{\mathsf{T}}$ $\displaystyle=-\eta\underbrace{\nabla_{\bm{\mathsf{z}}}\log\pi^{t}({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}})|_{\bm{\mathsf{z}}^{t}}}_{V\times V}\Big{[}\underbrace{\nabla_{\theta}\bm{\mathsf{z}}^{t}({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{o}})|_{\theta^{t}}}_{V\times d}\underbrace{\left(\left[\nabla_{\theta}\bm{\mathsf{z}}^{+}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{\chi}}_{u}^{+}});\nabla_{\theta}\bm{\mathsf{z}}^{-}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{\chi}}_{u}^{-}})\right]|_{\theta^{t}}\right)^{\mathsf{T}}}_{d\times 2V}\Big{]}\underbrace{\big{(}\nabla_{[\bm{\mathsf{z}}^{+};\bm{\mathsf{z}}^{-}]}\mathcal{L}|_{\bm{\mathsf{z}}^{t}}\big{)}^{\mathsf{T}}}_{2V\times 1}$ $\displaystyle=-\eta\mathcal{A}^{t}({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{\chi}}_{o}})\big{[}\mathcal{K}^{t}({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{\chi}}_{o}},{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{\chi}}_{u}^{+}});\mathcal{K}^{t}({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{\chi}}_{o}},{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{\chi}}_{u}^{-}})\big{]}\big{(}\nabla_{[\bm{\mathsf{z}}^{+};\bm{\mathsf{z}}^{-}]}\mathcal{L}|_{\bm{\mathsf{z}}^{t}}\big{)}^{\mathsf{T}}$ $\displaystyle=-\eta\mathcal{A}^{t}({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{\chi}}_{o}})\mathcal{K}^{t}({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{\chi}}_{o}},{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\tilde{\bm{\mathsf{\chi}}}_{u}})\mathcal{G}^{t}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}},\bm{\mathsf{y}}_{u}^{+},\bm{\mathsf{y}}_{u}^{-}),$ (16) $$

where $\mathcal{K}^{t}({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{\chi}}_{o}},{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\tilde{\bm{\mathsf{\chi}}}_{u}})$ is the equivalent eNTK that average the influence of both $\bm{\mathsf{y}}_{u}^{+}$ and $\bm{\mathsf{y}}_{u}^{-}$.

We then calculate the residual term $\mathcal{G}^{t}$ for DPO. For ease of calculation,
we decompose the loss into:

$$ $\displaystyle\mathcal{L}_{\text{DPO}}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}},\bm{\mathsf{y}}_{u}^{+},\bm{\mathsf{y}}_{u}^{-}\mid\theta)$ $\displaystyle=-\log(a)$ $\displaystyle a$ $\displaystyle\triangleq\sigma(b)$ $\displaystyle b$ $\displaystyle\triangleq\beta\left(\log\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{+})-\log\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{-})\right)-c$ $\displaystyle=\beta\left(\bm{\mathsf{e}}_{\bm{\mathsf{y}}_{u}^{+}}-\bm{\mathsf{e}}_{\bm{\mathsf{y}}_{u}^{-}}\right)^{\top}\log\pi_{\theta^{t}}(\bm{\mathsf{y}}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})-c$ $\displaystyle=\beta\left(\mathcal{L}_{\text{SFT}}([{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}},\bm{\mathsf{y}}_{u}^{-}])-\mathcal{L}_{\text{SFT}}([{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}},\bm{\mathsf{y}}_{u}^{+}])\right)-c$ $\displaystyle c$ $\displaystyle\triangleq\beta\left(\bm{\mathsf{e}}_{\bm{\mathsf{y}}_{u}^{+}}-\bm{\mathsf{e}}_{\bm{\mathsf{y}}_{u}^{-}}\right)^{\top}\log\pi_{\text{ref}}(\bm{\mathsf{y}}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}}),$ (17) $$

where $c$ is not a function of $\theta$.
Using the chain rule, the $l$-th column of the residual term $\mathcal{G}^{t}_{\text{DPO}}$ can be calculated as:

$$ $\displaystyle[\mathcal{G}_{\text{DPO}}^{t}]_{l}$ $\displaystyle=\frac{\partial\mathcal{L}_{\text{DPO}}}{\partial a}\frac{\partial a}{\partial b}\nabla_{\pi}b|_{\pi_{\theta^{t}}}\nabla_{\bm{\mathsf{z}}_{l}}\pi^{t}|_{\bm{\mathsf{z}}_{l}^{t}}$ $\displaystyle=-\frac{1}{a}a(1-a)\langle\nabla_{\pi}b|_{\pi_{\theta^{t}}},[\mathcal{A}^{t}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{\chi}}_{u}})]_{l}\rangle$ $\displaystyle=-(1-a)\langle\beta\left(\mathcal{L}_{\text{SFT}}([{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}},\bm{\mathsf{y}}_{u}^{-}]_{l})-\mathcal{L}_{\text{SFT}}([{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}},\bm{\mathsf{y}}_{u}^{+}]_{l})\right),[\mathcal{A}^{t}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{\chi}}_{u}})]_{l}\rangle$ $\displaystyle=-\beta(1-a)\left(\langle\mathcal{L}_{\text{SFT}}([{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}},\bm{\mathsf{y}}_{u}^{-}]_{l}),[\mathcal{A}^{t}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{\chi}}_{u}})]_{l}\rangle-\langle\mathcal{L}_{\text{SFT}}([{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}},\bm{\mathsf{y}}_{u}^{+}]_{l}),[\mathcal{A}^{t}({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{\chi}}_{u}})]_{l}\rangle\right)$ $\displaystyle=-\beta(1-a)\left((\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{-})-\bm{\mathsf{e}}_{\bm{\mathsf{y}}_{u}^{-}})-(\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{+})-\bm{\mathsf{e}}_{\bm{\mathsf{y}}_{u}^{+}})\right)_{l}$ $\displaystyle\approx\beta(1-a)\left(\bm{\mathsf{e}}_{\bm{\mathsf{y}}_{u}^{-}}-\bm{\mathsf{e}}_{\bm{\mathsf{y}}_{u}^{+}}\right)_{l}.$ (18) $$

The approximation in the last line takes the fact that at the early stage of alignment,
the model might not separate $\bm{\mathsf{y}}_{u}^{+}$ and $\bm{\mathsf{y}}_{u}^{-}$ well.
Hence we can eliminate the influence of $\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{-})-\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{+})$ to have a more concise expression.
By stacking values with different $l$,
we can get the residual term of DPO as

$$ $\mathcal{G}^{t}_{\text{DPO}}\approx\beta(1-a)\left(\bm{\mathsf{e}}_{\bm{\mathsf{y}}_{u}^{-}}-\bm{\mathsf{e}}_{\bm{\mathsf{y}}_{u}^{+}}\right);a=\sigma\left(\beta\log\frac{\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{+}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})}{\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{-}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})}-\beta\log\frac{\pi_{\text{ref}}(\bm{\mathsf{y}}_{u}^{+}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})}{\pi_{\text{ref}}(\bm{\mathsf{y}}_{u}^{-}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})}\right)$ (19) $$

Similarly, we can calculate the residual terms for other off-policy preference optimization methods,
like Identity-preference Optimization (IPO []):

$$ $\mathcal{L}_{\text{IPO}}=-\sum_{({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}},\bm{\mathsf{y}}_{u}^{+},\bm{\mathsf{y}}_{u}^{-})}\left(\left(\log\frac{\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{+}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})}{\pi_{\text{ref}}(\bm{\mathsf{y}}_{u}^{+}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})}-\log\frac{\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{-}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})}{\pi_{\text{ref}}(\bm{\mathsf{y}}_{u}^{-}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})}-\frac{1}{2\beta}\right)\right)^{2}.$ (20) $$

$$ $\mathcal{G}^{t}_{\text{IPO}}\approx 2a\left(\bm{\mathsf{e}}_{\bm{\mathsf{y}}_{u}^{-}}-\bm{\mathsf{e}}_{\bm{\mathsf{y}}_{u}^{+}}\right);a=\log\frac{\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{+}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})}{\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{-}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})}-\log\frac{\pi_{\text{ref}}(\bm{\mathsf{y}}_{u}^{+}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})}{\pi_{\text{ref}}(\bm{\mathsf{y}}_{u}^{-}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})}-\frac{1}{2\beta}$ (21) $$

For the Sequence Likelihood Calibration (SLiC []), we have:

$$ $\displaystyle\mathcal{L}_{\text{SLiC}}$ $\displaystyle=-\sum_{({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}},\bm{\mathsf{y}}_{u}^{+},\bm{\mathsf{y}}_{u}^{-})}\max\left[0,\delta-\log\frac{\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{+}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})}{\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{-}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})}\right]-\beta\cdot\log\pi_{\theta^{t}}(\bm{\mathsf{y}}_{\text{ref}}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})$ (22) $\displaystyle=-\sum_{({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}},\bm{\mathsf{y}}_{u}^{+},\bm{\mathsf{y}}_{u}^{-})}\max\left[0,\delta+\mathcal{L}_{\text{SFT}}([{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}},\bm{\mathsf{y}}_{u}^{+})])-\mathcal{L}_{\text{SFT}}([{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}},\bm{\mathsf{y}}_{u}^{-})])\right]+\beta\mathcal{L}_{\text{SFT}}([{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}},\bm{\mathsf{y}}_{\text{ref}}])$ (23) $$

$$ $\mathcal{G}^{t}_{\text{SLiC}}\approx a\left(\bm{\mathsf{e}}_{\bm{\mathsf{y}}_{u}^{-}}-\bm{\mathsf{e}}_{\bm{\mathsf{y}}_{u}^{+}}\right)+\beta\left(\pi_{\theta^{t}}(\bm{\mathsf{y}}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{\chi}}_{u}})-\bm{\mathsf{e}}_{\bm{\mathsf{y}}_{\text{ref}}}\right);a=\mathds{1}\left(\delta-\log\frac{\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{+})}{\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{-})}>0\right)$ (24) $$

In summary,
these RL-free algorithms all relate to the SFT loss to some extent.
For the DPO and IPO loss,
the directions of the updating signals are identical (i.e., $\bm{\mathsf{e}}_{\bm{\mathsf{y}}_{u}^{-}}-\bm{\mathsf{e}}_{\bm{\mathsf{y}}_{u}^{+}}$, where the $\pi(\bm{\mathsf{y}})$ term is canceled during calculation).
A scalar controls the strength of this update,
which usually correlated with the confidence gap between the model’s current confidence on $\bm{\mathsf{y}}_{u}^{+}$ and $\bm{\mathsf{y}}_{u}^{-}$,
i.e., $Gap(\pi_{\theta^{t}})\triangleq\log\frac{\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{+}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})}{\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{-}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})}$.
Generally, larger this value leads to a bigger $a$,
making the norm of $\mathcal{G}^{t}$ smaller.
In other words,
we see a “regularizing” effect in this term,
where the model should not make $Gap(\pi_{\theta^{t}})$ too large.
The SLiC loss can be considered as a combination of SFT adaptation and preference adaptation.
Similarly,
we can also see a hard version of the regularization effect mentioned above.
If $Gap(\pi_{\theta^{t}})>\delta$,
the indicator function will become zero,
and the model stops pushing $\pi(\bm{\mathsf{y}}_{u}^{+})$ and $\pi(\bm{\mathsf{y}}_{u}^{-})$ away when it already separates $\bm{\mathsf{y}}_{u}^{+}$ and $\bm{\mathsf{y}}_{u}^{-}$ well.

Recently, authors of [] propose another interesting self-play alignment algorithm called SPPO,
which further improves the alignment performance on top of many on-policy DPO methods.
Our framework could also give an interesting explanation of why this method works so well.
Specifically, the loss function of SPPO can be written as:

$$ $\mathcal{L}_{\text{SPPO}}=-\sum_{({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}},\bm{\mathsf{y}}_{u}^{+},\bm{\mathsf{y}}_{u}^{-})}\left(\log\frac{\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{+}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})}{\pi_{\text{ref}}(\bm{\mathsf{y}}_{u}^{+}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})}-\frac{\eta}{2}\right)^{2}+\left(\log\frac{\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{-}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})}{\pi_{\text{ref}}(\bm{\mathsf{y}}_{u}^{-}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})}+\frac{\eta}{2}\right)^{2}.$ (25) $$

$$ $\mathcal{G}^{t}_{\text{SPPO}}=2\left(\log\frac{\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{+}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})}{\pi_{\text{ref}}(\bm{\mathsf{y}}_{u}^{+}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})}-\frac{\eta}{2}\right)(\pi_{\theta^{t}}-\bm{\mathsf{e}}_{\bm{\mathsf{y}}_{u}^{+}})+2\left(\log\frac{\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{-}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})}{\pi_{\text{ref}}(\bm{\mathsf{y}}_{u}^{-}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})}+\frac{\eta}{2}\right)(\pi_{\theta^{t}}-\bm{\mathsf{e}}_{\bm{\mathsf{y}}_{u}^{-}}).$ (26) $$

This loss looks similar to the IPO one,
but the main difference between SPPO and other methods (e.g., DPO, KTO, IPO, SPIN, etc.) is that there is no negative sign in front of $\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{+}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})$ or $\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{-}\mid{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}})$.
From its residual term $\mathcal{G}^{t}_{\text{SPPO}}$,
it is more convenient to understand this algorithm as imposing two positive vectors on both $\bm{\mathsf{y}}_{u}^{+}$ and $\bm{\mathsf{y}}_{u}^{-}$,
but the former has a longer norm,
as illustrated in [Figure 4](#S5.F4).
By doing so,
the big negative gradient no longer exists,
and so does the squeezing effect.
That is partly why this method is more stable and performs better.

## Appendix B More Results of Observing the Learning Dynamics

This section provides more experimental results about the learning dynamics to support our claim.
We will extend our probing dataset by considering more responses and the prompts from the test set.
The full results on both SFT and DPO under different settings are also provided here.

### B.1 Extend the probing dataset by considering the edit distance

To further verify our analysis,
we extend the probing dataset in the following two ways.
First,
based on the 3 groups and 7 types of probing responses provided in [Section 4](#S4),
we further introduce 7 more types.
By considering the format differences (could be understood as edit distance) of these $\bm{\mathsf{y}}$ compared with ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{y}}_{u}^{+}}$,
we extend the 1-D $\mathcal{Y}$-space in [Figure 2](#S4.F2) to a 2-D space in [Figure 6](#A2.F6),
in which we can project all 14 types of responses on this plane.
Second,
we also create another probing dataset (named $\mathcal{DP}_{\text{test}}$) where all $\bm{\mathsf{x}}$ comes from the test set.
Compared with $\mathcal{DP}_{\text{train}}$ that we used in the main context,
all the prompts and responses in $\mathcal{DP}_{\text{test}}$ are never exposed to the model during finetuning.
By comparing the learning curves of these two probing datasets,
we can figure out the difference between the model’s prediction of those directly influenced responses ($\bm{\mathsf{y}}$ appears during training) and the indirectly influenced ones ($\bm{\mathsf{y}}$ that the model never sees during training).
Finally,
we believe the level of the “on-policy” property (which is very important for the preference finetuning []) could also be introduced as the second axis in our 2-D plane.
We left the exploration of this interesting direction in our future work.
We list how the 7 extended responses are generated as follows,
where the prompt templates used to generate them are illustrated in [Figure 7](#A2.F7).
We also provide examples of all 14 types of responses in [Figure 8](#A2.F8).

- 0.
$\bm{\mathsf{y}}_{\pi^{0}}$, the initial response generated by feeding ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}}$ to LLM before finetuning;
- 1.
$\bm{\mathsf{y}}_{u}^{+}$, the chosen (i.e., the preferred) response of ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}}$;
1.1
$\bm{\mathsf{y}}_{\text{selfr}}^{+}$, rephrase $\bm{\mathsf{y}}_{u}^{+}$ using $\bm{\mathsf{y}}_{\pi^{0}}$, algorithm from [];
1.2
$\bm{\mathsf{y}}_{\text{gpts}}^{+}$, rephrase $\bm{\mathsf{y}}_{u}^{+}$ using GPT4, keep the semantics while changing the format;
1.3
$\bm{\mathsf{y}}_{\text{gptf}}^{+}$, rephrase $\bm{\mathsf{y}}_{u}^{+}$ using GPT4, keep the format while changing the semantics;
- 2.
$\bm{\mathsf{y}}_{u}^{-}$, the rejected (i.e., the less preferred) response of ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}}$;
2.1
$\bm{\mathsf{y}}_{\text{selfr}}^{-}$, rephrase $\bm{\mathsf{y}}_{u}^{-}$ using $\bm{\mathsf{y}}_{\pi^{0}}$, algorithm from [];
2.2
$\bm{\mathsf{y}}_{\text{gpts}}^{-}$, rephrase $\bm{\mathsf{y}}_{u}^{-}$ using GPT4, keep the semantics while changing the format;
2.3
$\bm{\mathsf{y}}_{\text{gptf}}^{-}$, rephrase $\bm{\mathsf{y}}_{u}^{-}$ using GPT4, keep the format while changing the semantics;

Figure: Figure 6: The 2-D plane of $\mathcal{Y}$ by considering the distance in both format and semantics.
Refer to caption: /html/2407.10490/assets/x6.png

Figure: Figure 7: The prompts used to generate $\bm{\mathsf{y}}_{\text{selfr}}^{+}$, $\bm{\mathsf{y}}_{\text{gpts}}^{+}$, and $\bm{\mathsf{y}}_{\text{gptf}}^{+}$. The rephrases of rejected samples are generated similarly. The self-rephrase template comes from [].
Refer to caption: /html/2407.10490/assets/x7.png

Figure: Figure 8: Example of all possible responses for one $\bm{\mathsf{x}}$ in our probing dataset. Note that the pure random token is generated by first creating a random sentence, then randomly permuting its tokens.
Refer to caption: /html/2407.10490/assets/x8.png

### B.2 More results on different settings: SFT case

Consistent learning dynamics for different models.
In this subsection,
we provide more results to support our analysis on SFT in [Section 4](#S4).
The first thing to verify is the consistency of the trends of learning dynamics across different settings.
As illustrated in [Figure 10](#A2.F10),
we conduct SFT on five models with different sizes pretrained using different recipes.
Note that Pythia-410M/1B/1.4B/2.8B are pretrained using exactly the same dataset and pipeline [],
while Qwen1.5-0.5B are pretrained differently.
Hence we can observe a slight difference between the curves from Pythia series and Qwen series,
e.g., those in $\bm{\mathsf{y}}_{\text{hum}}$.
However, the trends demonstrated in [Figure 3](#S4.F3) consistently hold for all models.

Compare the rephrases of $\bm{\mathsf{y}}_{u}^{+}$ and $\bm{\mathsf{y}}_{u}^{-}$.
See [Figure 12](#A2.F12),
where we put the rephrases of the same response into the same figure.
We can treat the red curve, i.e., the one of $\bm{\mathsf{y}}$ generated by $\pi^{0}(\bm{\mathsf{x}})$, as a baseline,
whose decaying suggests the policy model is deviating from the initial point.
The first observation is that after several updates,
$\bm{\mathsf{y}}_{u}^{+}$ is the only one that keeps increasing,
which means the “pull up” pressure generated by $[{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}};{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{y}}_{u}^{+}}]$ do not have that strong influence on these rephrases compared to $[{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{x}}_{j\neq u}};{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{y}}_{u}^{+}}]$,
even though these ${\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathsf{y}}}$ are good rephrases of ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{y}}_{u}^{+}}$ (recall the curve $\bm{\mathsf{y}}_{j\neq n}^{+}$ always increase in [Figure 10](#A2.F10)).
Furthermore,
by carefully comparing the decreasing speed of $\bm{\mathsf{y}}_{\pi^{0}}$ and other curves,
we find those rephrases decays slower than $\bm{\mathsf{y}}_{\pi^{0}}$ in the chosen case,
but not the case for the rejected responses.
This phenomenon also supports our analysis well:
because we train the model using $\bm{\mathsf{y}}_{u}^{+}$,
their rephrases are “pulled up” more than the rephrases of $\bm{\mathsf{y}}_{u}^{-}$.
Such a claim is also verified by the experiment in the last column of this figure,
where we train the model using $[{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{x}}_{u}};{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\mathsf{y}}_{u}^{-}}]$ rather than $\bm{\mathsf{y}}_{u}^{+}$.
In these two panels, we see the decaying speed of rephrases of $\bm{\mathsf{y}}_{u}^{+}$ is now identical to that of $\bm{\mathsf{y}}_{\pi^{0}}$ while the decaying speed of rephrases for $\bm{\mathsf{y}}_{u}^{-}$ is slightly slower.
Last,
compare the green and orange curves (i.e., the format-keeping and semantics-keeping GPT rephrases),
we find the predicting probabilities of those format-keeping curves are usually larger than their semantic-keeping counterparts.
This is a sign that the model during SFT might care more about the format rather than the semantics of one sentence.
We will delve into this interesting phenomenon in our future work.

Compare $\mathcal{DP}_{\text{train}}$ and $\mathcal{DP}_{\text{test}}$.
To isolate the influence of the “pull up” pressure introduced by the training updates,
we also create another probing dataset $\mathcal{DP}_{\text{test}}$ using the same pipeline as $\mathcal{DP}_{\text{train}}$.
The only difference between them is that all $\bm{\mathsf{x}}$ in $\mathcal{DP}_{\text{test}}$ comes from the test set,
and hence neither the prompts nor the responses ever occur during training.
See [Figure 14](#A2.F14),
where the solid curves and dotted curves represent the learning dynamics of responses in $\mathcal{DP}_{\text{train}}$ and $\mathcal{DP}_{\text{test}}$ respectively.
The color of the curves represents the model we are finetuning.
By qualitatively comparing the trend difference between curves coming from $\mathcal{DP}_{\text{train}}$ and $\mathcal{DP}_{\text{test}}$,
we roughly observe that trend_diff$(\bm{\mathsf{y}}_{u}^{+})$ > trend_diff($\bm{\mathsf{y}}_{j\neq u}^{+}$) > trend_diff($\bm{\mathsf{y}}_{\text{gpts}}^{+}$) > trend_diff($\bm{\mathsf{y}}_{\text{gptf}}^{+}$),
which aligns well with our hypothesis about how strong the “pull up” pressure influence different responses.

Figure: Figure 9: Comparison of the learning dynamics of responses in different groups in the proposed probing dataset. The Antropic-HH version is [Figure 3](#S4.F3) in the main context. Trends to observe: 1.) $\bm{\mathsf{y}}_{u}^{+}$ increase and $\bm{\mathsf{y}}_{u}^{-}$ first increase then decrease; 2.) both $\bm{\mathsf{y}}_{\text{urnd}}^{+}$ and $\bm{\mathsf{y}}_{\text{rnd}}^{\prime}$ decrease and very small; 3.) $\bm{\mathsf{y}}_{j\neq u}^{+}$ increases with a smaller rate than $\bm{\mathsf{y}}_{u}^{+}$, although the $[\bm{\mathsf{x}}_{u};\bm{\mathsf{y}}_{j\neq u}^{+}]$ never occurs during training; 4.) both $\bm{\mathsf{y}}_{\text{test}}^{+}$ and $\bm{\mathsf{y}}_{\text{hum}}$ has a bell-shape curve; 5.) the inflection of $\bm{\mathsf{y}}_{\text{hum}}$ is earlier. (UltraFeedback, SFT)
Refer to caption: /html/2407.10490/assets/x9.png

Figure: Figure 10: Trend to observe: curves of different models exhibit similar trends. (Antropic-HH, SFT)
Refer to caption: /html/2407.10490/assets/x10.png

Figure: Figure 11: The trends of learning dynamics of different models are similar. (UltraFeedback, SFT)
Refer to caption: /html/2407.10490/assets/x11.png

Figure: Figure 12: Compare different rephrases of $\bm{\mathsf{y}}_{u}^{+}$ and $\bm{\mathsf{y}}_{u}^{-}$ under different models. Key trend to observe: 1.) For the first row, the decaying speed of $\bm{\mathsf{y}}_{\text{gpts}}^{+}$ and $\bm{\mathsf{y}}_{\text{gptf}}^{+}$ are smaller than $\bm{\mathsf{y}}_{\pi^{0}}$, which means the pull-up pressure exists; 2.) For the second row, the decaying speed of $\bm{\mathsf{y}}_{\text{gpts}}^{-}$ and $\bm{\mathsf{y}}_{\text{gptf}}^{-}$ are similar to that of $\bm{\mathsf{y}}_{\pi^{0}}$, because the pull-up pressures on rejected samples are smaller; 3.) For the last column, since we SFT the model using the rejected sample rather than the chosen one, the trend in (1) and (2) reverses. (Antropic-HH, SFT)
Refer to caption: /html/2407.10490/assets/x12.png

Figure: Figure 13: Compare different rephrases of $\bm{\mathsf{y}}_{u}^{+}$ and $\bm{\mathsf{y}}_{u}^{-}$ under different models. (UltraFeedback, SFT)
Refer to caption: /html/2407.10490/assets/x13.png

Figure: Figure 14: Compare the learning dynamics of examples from $\mathcal{DP}_{\text{train}}$ and $\mathcal{DP}_{\text{test}}$. Key trend to observe: for $\mathcal{DP}_{\text{train}}$, since many responses and prompts ever occur during training, the pull-up pressure is generally stronger. Curves of $\bm{\mathsf{y}}_{u}^{+}$, $\bm{\mathsf{y}}_{\text{gpts}}^{+}$, $\bm{\mathsf{y}}_{\text{gptf}}^{+}$ and $\bm{\mathsf{y}}_{j\neq u}^{+}$ shows a clear trend. (Antropic-HH, SFT)
Refer to caption: /html/2407.10490/assets/x14.png

### B.3 More results on different settings: off-policy DPO case

Similar to [Section B.2](#A2.SS2),
we also provide extra experiments for DPO in this part using the same probing dataset.
Note that as the responses of on-policy DPO change generation-by-generation,
it is hard to observe the dynamics of a pre-collected probing dataset.
We left the exploration of how to effectively probe other DPO variants in our future work.

Consistent learning dynamics for different models.
Compare [Figure 5](#S5.F5) in the main context and [Figure 16](#A2.F16),
where we provide the results on many different models (Pythia-410M/1B/2.8B and Qwen1.5-0.5B).
Their trends on different $\pi_{\theta^{t}}(\bm{\mathsf{y}})$ are quite consistent:

- 1.)
in the first column, the margin $\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{+})-\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{-})$ keeps increasing. The $\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{+})$ first increase and then decrease, always with a smaller decay speed than that of $\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{-})$;
- 2.)
in the second column, $\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{+})$ decreases slower than the other rephrases, verifying the “pull up” pressure and the influence on other responses via $\mathcal{K}^{t}$;
- 3.)
in the third column, $\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{-})$ decreases faster than the other rephrases, verifying the “push down” pressure and the influence on other $\bm{\mathsf{y}}$;
- 4.)
in the fourth column, the rephrases of $\bm{\mathsf{y}}_{u}^{+}$ decay slower than those of $\bm{\mathsf{y}}_{u}^{-}$, supporting the claims that the rephrases near the chosen responses are influenced by the “pull up” pressure while the rephrases of the rejected ones are influenced by the “push down” pressure.

Learning dynamics of conducting SFT first, then DPO.
As stated in [],
conducting SFT before DPO is a common pipeline for alignment.
Using $[\bm{\mathsf{x}};\bm{\mathsf{y}}_{u}^{+}]$ as the SFT dataset is also a common practice in many existing works.
Hence in this part,
we plot the curves of different $\pi_{\theta^{t}}(\bm{\mathsf{y}})$ in both two stages to demonstrate their differences.
See [Figure 15](#A2.F15),
where the difference between the experiments in these three rows is how long the model is trained using SFT before DPO.
The learning rate of both SFT and DPO are controlled to be the same (i.e., $5\times 10^{-7}$, the default value in []).
All the curves are aligned by the 10th epoch on the x-axis (i.e., the starting time for the DPO training) for the convenience of comparing the trends across different settings.

We first check the curves of SFT and DPO parts separately and find that all the above relative trends still hold in these experiments.
We then compare the model’s behavior in these two phases respectively.
In the last two rows of [Figure 15](#A2.F15),
where the epoch for SFT is non-zero,
it is clear that the decaying speed of most observing $\pi_{\theta^{t}}(\bm{\mathsf{y}})$ is much larger in DPO than those in SFT.
The main reason for this is the existence of a big negative gradient introduced in DPO.
This gradient, especially conducted on a “valley” region of the model’s prediction,
will “push down” the whole curve significantly, except the one with the highest confidence before updating.
This non-trivial trend is named “squeezing effect”,
which is elaborated on in [Appendix C](#A3).
Furthermore,
a more peaky $\pi_{\theta^{0}}(\bm{\mathsf{y}})$ and a smaller $\pi_{\theta^{0}}(\bm{\mathsf{y}}_{u}^{-})$ will lead to a stronger “squeezing effect”,
which can be verified by comparing the curves of the last two panels:
longer SFT makes the model’s prediction peakier when DPO is conducted,
which leads to a larger decay on all $\pi_{\theta^{t}}(\bm{\mathsf{y}})$ during DPO.

Figure: Figure 15: The learning dynamics of conducting DPO after SFT the model for several epochs. We align the starting point of DPO (i.e., the 10th epoch from the x-axis) to better compare the curves. Key trend to observe: 1.) Confidence of all responses decays way faster when DPO starts, which is caused by the squeezing effect introduced via a big negative gradient; 2.) The more epochs we SFT the model, the more serious the squeezing effect is (confidence decays faster). (Antropic-HH, SFT $\rightarrow$ DPO)
Refer to caption: /html/2407.10490/assets/x15.png

Figure: Figure 16: The learning dynamics of DPO on different models. Key trends to observe: 1.) Confidence of $\bm{\mathsf{y}}_{u}^{+}$ decays slower than that of $\bm{\mathsf{y}}_{u}^{-}$; 2.) Confidence of $\bm{\mathsf{y}}_{u}^{+}$ decays slower than those of $\bm{\mathsf{y}}_{\text{gpts}}^{+}$ and $\bm{\mathsf{y}}_{\text{gptf}}^{+}$, because the pull-up pressure is directly imposed on $\bm{\mathsf{y}}_{u}^{+}$; 3.) Confidence of $\bm{\mathsf{y}}_{u}^{-}$ decays faster than those of $\bm{\mathsf{y}}_{\text{gpts}}^{-}$ and $\bm{\mathsf{y}}_{\text{gptf}}^{-}$, because the push-down pressure is directly imposed on $\bm{\mathsf{y}}_{u}^{-}$; 4.) Confidence of the rephrases of rejected responses decays faster than the rephrases of chosen responses. (Antropic-HH, DPO)
Refer to caption: /html/2407.10490/assets/x16.png

Figure: Figure 17: The learning dynamics of DPO on different models. (UltraFeedback, DPO)
Refer to caption: /html/2407.10490/assets/x17.png

## Appendix C The Squeezing Effect Introduced by Big Negative Gradient

In DPO,
the model gradually learns how to separate the chosen and rejected responses by imposing one positive and one negative adaptation vector on $\bm{\mathsf{y}}_{u}^{+}$ and $\bm{\mathsf{y}}_{u}^{-}$ respectively,
as illustrated in the second panel in [Figure 4](#S5.F4).
These two opposite pressures ensure the margin reward $\pi_{\theta}(\bm{\mathsf{y}}_{u}^{+})-\pi_{\theta}(\bm{\mathsf{y}}_{u}^{-})$ keep increasing,
which makes the model align with human preferences better.
However,
if we go deeper and observe $\pi_{\theta}(\bm{\mathsf{y}}_{u}^{+})$ and $\pi_{\theta}(\bm{\mathsf{y}}_{u}^{-})$ separately,
a very interesting phenomenon occurs.
See the first column of [Figure 16](#A2.F16),
we find although DPO also contains a strong positive adaptation vector,
the curve of $\pi_{\theta}(\bm{\mathsf{y}}_{u}^{+})$ all goes down after several updates,
which is very different from $\pi_{\theta}(\bm{\mathsf{y}}_{u}^{+})$ in the SFT case.
Such an observation is also reported in many related works [, , ],
but a clear-cut explanation of it is still missing.
Furthermore,
although the relative behaviors of various rephrases matches our analysis of learning dynamics well,
merely the two pressures on $\bm{\mathsf{y}}_{u}^{+}$ and $\bm{\mathsf{y}}_{u}^{-}$ cannot explain why all these observed $\pi_{\theta}(\bm{\mathsf{y}})$ keeps decreasing during training.
So, it is natural to ask:

Where has the probability mass gone?

### C.1 What is the Squeezing Effect and Why it Exist

To answer the above question,
we can start from the properties of the basic $\operatorname{\mathsf{Softmax}}$ function by analyzing a simple multi-class logistic regression problem.
Because no matter how complex the LLM is,
its predictions are made by converting the logits into probabilities using $\operatorname{\mathsf{Softmax}}$ heads.

Consider a simple $V$-class logistic regression problem where each high-dimensional input data $\bm{\mathsf{x}}$ is converted to a length-$d$ feature vector via a deep neural network $\phi$.
In other words,
we have $\phi(\bm{\mathsf{x}})\in\mathbb{R}^{d\times 1}$.
The model uses a linear read-out layer $\bm{\mathsf{w}}\in\mathbb{R}^{d\times V}$ to convert the feature vector to logits $\bm{\mathsf{z}}=\bm{\mathsf{w}}^{\top}\phi(\bm{\mathsf{x}})$ and then generate the probability prediction vector $\bm{\mathsf{p}}$ using a $\operatorname{\mathsf{Softmax}}$ head.
We consider a common cross-entropy loss function for each input pair $(\bm{\mathsf{x}},y)$.
In summary, we have:

$$ $\mathcal{L}_{\text{CE}}(\bm{\mathsf{p}}^{t},y)=-\bm{\mathsf{e}}_{y}^{\top}\log\bm{\mathsf{p}}^{t};\quad\bm{\mathsf{p}}^{t}=\operatorname{\mathsf{Softmax}}(\bm{\mathsf{z}}^{t});\quad\bm{\mathsf{z}}^{t}=(\bm{\mathsf{w}}^{t})^{\top}\phi(\bm{\mathsf{x}}),$ (27) $$

where $t$ is the index of the step during training and $\bm{\mathsf{e}}_{y}$ is a length-$V$ one-hot vector determined by the ground truth label $y$.
To simplify our analysis,
we assume a fixed $\phi$ and only update the parameters of the read-out layer $\bm{\mathsf{w}}$ using stochastic gradient descent:

$$ $\bm{\mathsf{w}}^{t+1}=\bm{\mathsf{w}}^{t}-\eta\nabla_{\bm{\mathsf{w}}}\mathcal{L}=\bm{\mathsf{w}}^{t}-\eta\phi(\bm{\mathsf{x}})(\bm{\mathsf{p}}^{t}-\bm{\mathsf{e}}_{y})^{\top},$ (28) $$

where $\eta$ is the learning rate which can be negative if we consider a negative gradient during training.
With [Equation 27](#A3.E27) and ([28](#A3.E28)),
we can write down each dimension of $\bm{\mathsf{p}}^{t}$ and $\bm{\mathsf{p}}^{t+1}$ after some calculations.
To quantitatively analyze how the model’s confidence in each class changes,
we define a ratio $\alpha_{i}\triangleq\frac{p_{i}^{t+1}}{p_{i}^{t}}$ and use the following lemma to describe its behavior:

lemsqueeze
The ratio of confidence change for each $i$ can be represented as:

$$ $\alpha_{i}\triangleq\frac{p_{i}^{t+1}}{p_{i}^{t}}=\frac{\sum_{j=1}^{V}e^{z_{j}^{t}}}{\sum_{j=1}^{V}\beta_{j}e^{z_{j}^{t}}}.$ (29) $$

Note that the values of $\beta_{j}$ also depends on whether $i$ equals $y$,
hence for Case 1 ($i=y$) and Case 2 ($i\neq y$), we have ($\eta^{\prime}\triangleq\eta\|\phi(\bm{\mathsf{x}})\|_{2}^{2}$ is the equivalent learning rate):

$$ $\text{Case 1: }\beta_{j}=\left\{\begin{aligned} &e^{-\eta^{\prime}(1+p_{j}^{t}-p_{i}^{t})}&\text{if }j\neq y\\ &\quad\quad\quad 1&\text{if }j=y\end{aligned}\right.;\quad\text{Case 2: }\beta_{j}=\left\{\begin{aligned} &e^{-\eta^{\prime}(p_{j}^{t}-p_{i}^{t})}&\text{if }j\neq y\\ &e^{-\eta^{\prime}(p_{j}^{t}-p_{i}^{t}-1)}&\text{if }j=y\end{aligned}\right.$ (30) $$

###### Proof.

To derive [Equation 29](#A3.E29),
we need to have the analytical expression of each $p_{i}^{t+1}$ and $p_{i}^{t}$.
As $\bm{\mathsf{p}}=\operatorname{\mathsf{Softmax}}(\bm{\mathsf{z}})$,
we need to link $\bm{\mathsf{z}}^{t+1}$ and $\bm{\mathsf{z}}^{t}$ first.
With [Equation 27](#A3.E27) and ([28](#A3.E28)),
$\bm{\mathsf{z}}^{t+1}$ can be recursively written down as:

$$ $\displaystyle\bm{\mathsf{z}}^{t+1}$ $\displaystyle=(\bm{\mathsf{w}}^{t+1})^{\top}\phi(\bm{\mathsf{x}})$ $\displaystyle=\left(\bm{\mathsf{w}}^{t}-\eta\phi(\bm{\mathsf{x}})(\bm{\mathsf{p}}^{t}-\bm{\mathsf{e}}_{y})^{\top}\right)^{\top}\phi(\bm{\mathsf{x}})$ $\displaystyle=(\bm{\mathsf{w}}^{t})^{\top}\phi(\bm{\mathsf{x}})-\eta\left(\phi(x)(\bm{\mathsf{p}}^{t}-\bm{\mathsf{e}}_{y})^{\top}\right)^{\top}\phi(\bm{\mathsf{x}})$ $\displaystyle=\bm{\mathsf{z}}^{t}-\eta\|\phi(\bm{\mathsf{x}})\|_{2}^{2}(\bm{\mathsf{p}}^{t}-\bm{\mathsf{e}}_{y})$ $\displaystyle=\bm{\mathsf{z}}^{t}-\eta^{\prime}(\bm{\mathsf{p}}^{t}-\bm{\mathsf{e}}_{y})$ (31) $$

where $\eta^{\prime}\triangleq\eta\|\phi(\bm{\mathsf{x}})\|_{2}^{2}$ is the equivalent learning rate that depends on the norm of feature representation.
Note that $\bm{\mathsf{z}}$, $\bm{\mathsf{p}}$ and $\bm{\mathsf{e}}_{y}$ are all length-$V$ vectors and $y$ is an integer ranging from 1 to $V$.
Then we can write down each $z_{i}^{t+1}$ as:

$$ $z_{i}^{t+1}=\left\{\begin{aligned} &z_{i}^{t}-\eta^{\prime}p_{i}^{t}+\eta^{\prime},\quad&\text{if }i=y\\ &z_{i}^{t}-\eta^{\prime}p_{i}^{t},\quad&\text{if }i\neq y\end{aligned}\right.$ (32) $$

Then, we can combine the definition of $\operatorname{\mathsf{Softmax}}$ function and write down different $p_{i}^{t+1}$ case-by-case.
For Case 1 where $i=y$,
we have:

$$ $p_{i=y}^{t+1}=\frac{e^{z_{i}^{t+1}}}{\sum^{V}_{j=1}e^{z_{j}^{t+1}}}=\frac{e^{z_{i}^{t}-\eta^{\prime}p_{i}^{t}+\eta^{\prime}}}{\sum_{j\neq y}e^{z_{j}^{t}-\eta^{\prime}p_{j}^{t}}+e^{z_{y}^{t}-\eta^{\prime}p_{y}^{t}+\eta^{\prime}}}=\frac{e^{z_{i}^{t}}}{\sum_{j\neq y}e^{z_{j}^{t}{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}-\eta^{\prime}(1+p_{j}^{t}-p_{i}^{t})}}+e^{z_{y}^{t}{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}-0}}},$ (33) $$

combining the fact that $p_{i}^{t}=\frac{e^{z_{i}^{t}}}{\sum^{K}_{j=1}e^{z_{j}^{t}}}$,
we can derive $\alpha_{i}$ and $\beta_{j}$ as the left part of [Equation 30](#A3.E30).
Similarly,
when $i\neq y$, we have:

$$ $p_{i\neq y}^{t+1}=\frac{e^{z_{i}^{t+1}}}{\sum^{V}_{j=1}e^{z_{j}^{t+1}}}=\frac{e^{z_{i}^{t}-\eta^{\prime}p_{i}^{t}}}{\sum_{j\neq y}e^{z_{j}^{t}-\eta^{\prime}p_{j}^{t}}+e^{z_{y}^{t}-\eta^{\prime}p_{y}^{t}+\eta^{\prime}}}=\frac{e^{z_{i}^{t}}}{\sum_{j\neq y}e^{z_{j}^{t}{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}-\eta^{\prime}(p_{j}^{t}-p_{i}^{t})}}+e^{z_{y}^{t}{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}-\eta^{\prime}(p_{y}^{t}-p_{i}^{t}-1)}}},$ (34) $$

which leads to the right part of [Equation 30](#A3.E30).

∎

We can now better understand how each $p_{i}$ changes after this update.
Specifically,
if $\alpha_{i}>1$,
the corresponding $p_{i}$ increases, and vice versa.
To determine the value of $\alpha_{i}$,
we can treat any $\beta_{j}>1$ as contributing to the conclusion that $\alpha_{i}<1$ while any $\beta_{j}<1$ against it.
The value of the corresponding $e^{z_{j}^{t}}$ and $|\beta_{j}-1|$ controls how strong the contribution is.
With the preparations above, we derive the following observations on how the confidence evolves when a gradient ascent (i.e., $\eta<0$) is imposed on class $y$.

###### Proof.

##### Claim 1: The value of p y subscript 𝑝 𝑦 p_{y} is guaranteed to decrease, i.e., α y < 1 subscript 𝛼 𝑦 1 \alpha_{y}<1 .

We start from the value of $\beta$ in Case 1 as illustrated in [Equation 30](#A3.E30).
It is clear that for any $j\neq y$,
we have $\beta_{j}>1$, because $1+p_{j}^{t}-p_{i}^{t}>0$.
Combining with $\beta_{y}=1$, it is straightforward to have Claim 1.

##### Claim 2: The value of p i ∗ subscript 𝑝 superscript 𝑖 p_{i^{*}} where i ∗ = argmax i ∈ [ V ] ∖ { y } p i t superscript 𝑖 subscript argmax 𝑖 delimited-[] 𝑉 𝑦 superscript subscript 𝑝 𝑖 𝑡 i^{*}=\operatorname*{argmax}_{i\in[V]\setminus\{y\}}p_{i}^{t} is guaranteed to increase, i.e., α i ∗ > 1 subscript 𝛼 superscript 𝑖 1 \alpha_{i^{*}}>1 .

We now use the value of $\beta$ in Case 2,
since $i^{*}$ cannot equal $y$ by definition.
When $j\neq y$,
we have $p_{j}^{t}-p_{i^{*}}^{t}\leq 0$ for all possible $j$,
because $p_{i^{*}}^{t}$ is the largest among all $p_{i\neq y}^{t}$ of $\bm{\mathsf{p}}^{t}$.
Hence all $\beta_{j\neq y}$ must be smaller than one.
Combining with the fact that $\beta_{y}<1$ (because $p_{y}^{t}-p_{i^{*}}^{t}-1$ must be negative),
we can prove that $\alpha_{i^{*}}>1$.

The two claims above demonstrate that the parameter update can be imagined as taking the probability mass from $p_{y}$ and redistributing that to other dimensions.
From Claim 2, we know some of the mass is guaranteed to be “squeezed” into the dimension with the highest $p_{i^{*}}^{t}$
(if $p_{y}^{t}$ is the highest value, then $p_{i^{*}}^{t}$ is the second highest in $\bm{\mathsf{p}}^{t}$).
But how other $p_{i}$ changes is still not clear yet.
Will the probability mass from $p_{y}$ is also split into other $p_{i}$ (i.e., other $p_{i}$ increases)?
Or will $p_{i^{*}}$ absorb the mass not only from $p_{y}$ but also from other dimensions (i.e., other $p_{i}$ decreases)?
To get a clearer picture,
we need to track the adaptations of each $p_{i}$.
To achieve this,
we now must scrutinize the distribution of $\bm{\mathsf{p}}^{t}$,
because it controls the value of $e^{z_{j}^{t}}$ for different $j$.
We chose three typical scenarios where $\bm{\mathsf{p}}^{t}$ is strictly uniform,
slightly non-uniform, and extremely peaky, and leads to the following claims.

##### Claim 3A: When 𝗽 t superscript 𝗽 𝑡 \bm{\mathsf{p}}^{t} is a uniform distribution, the probability mass decreased from class y 𝑦 y is uniformly distributed to all other i ≠ y 𝑖 𝑦 i\neq y , i.e., all p i ≠ y t + 1 superscript subscript 𝑝 𝑖 𝑦 𝑡 1 p_{i\neq y}^{t+1} increase the same value.

With the uniform $\bm{\mathsf{p}}^{t}$ assumption,
[Equation 29](#A3.E29) can be simplified to $\alpha_{i}=\frac{V}{\sum_{j=1}^{V}\beta_{j}}$.
Note that the first two claims hold for any distribution $\bm{\mathsf{p}}^{t}$,
hence we only check the values of $\alpha_{i\neq y}$ here to verify the “uniformly distributed mass” hypothesis.
Substituting the values of $\beta_{j}$ to this new $\alpha$ leads to $\alpha_{i}=\frac{V}{V-1+e^{\eta^{\prime}}}$ for all $i\neq y$.
Since $\eta^{\prime}<0$ and $e^{\eta^{\prime}}<1$,
we must have $\alpha_{i\neq y}>1$.
Combined with the fact that all $p_{i}^{t}$ are the same,
this claim can be proved.

##### Claim 3B: When 𝗽 t superscript 𝗽 𝑡 \bm{\mathsf{p}}^{t} is slightly non-uniform, p i subscript 𝑝 𝑖 p_{i} with smaller p i t superscript subscript 𝑝 𝑖 𝑡 p_{i}^{t} tend to decrease, and vice versa.

This claim is a general trend and might not have any guarantees.
However, analyzing such a scenario helps us to understand the influence of $\bm{\mathsf{p}}^{t}$ better.
Assume we are observing $\alpha_{i^{\prime}}$ where $i^{\prime}$ is not $y$ nor $i^{*}$.
We consider two subsets of $[V]\setminus\{y\}$,
i.e., $\mathcal{B}$, which contains all $j$ with $p_{i^{\prime}}^{t}\leq p_{j}^{t}$ and $\mathcal{S}$ that contains all $j$ with $p_{i^{\prime}}^{t}>p_{j}^{t}$.
Now consider Case 2 in [Equation 30](#A3.E30),
we have:

$$ $\beta_{j=y}\ll\beta_{j\in\mathcal{S}}<1;\quad\beta_{j\in\mathcal{B}}>1.$ (35) $$

Note that we misuse the $\ll$ notation to
highlight the fact that $\beta_{j=y}$ would be much smaller than $\beta_{j\in\mathcal{S}}$,
because there is a negative one term in the exponential.
With the above expression,
we can imagine that if $p_{i^{\prime}}^{t}$ is relatively small,
the size of $\mathcal{B}$ would be large,
which means there will be more $\beta_{j}>1$ contributing to the conclusion that $\alpha_{i^{\prime}}<1$.
If the influence of $\beta_{j\in\mathcal{B}}$ is strong enough to override the influence of other $\beta$ (especially $\beta_{j=y}$ which is way smaller than other $\beta$),
$\alpha_{i^{\prime}}$ would be smaller than one and hence $p_{i^{\prime}}$ decreases.
On the contrary,
for those $i^{\prime}$ with relatively large $p_{i^{\prime}}^{t}$,
the $\beta<1$ terms becomes dominant and hence lead to $\alpha_{i^{\prime}}>1$,
i.e., $p_{i^{\prime}}$ increases.

In the analysis above,
we assume $\bm{\mathsf{p}}^{t}$ is only slightly non-uniform (i.e., not so peaky),
which means the values of different $e^{z_{j}^{t}}$ are relatively comparable.
However, in practical machine learning systems like LLM’s finetuning,
the distribution $\bm{\mathsf{p}}^{t}$ would be very non-uniform,
which means most of the probability mass is obtained by a few dimensions.
That is because the LLM’s vocabulary size is usually very large and the reasonable choice of the next word is only a small portion of the whole vocabulary.
Thus we have the following claim to describe this practical scenario.

##### Claim 3C: When 𝗽 t superscript 𝗽 𝑡 \bm{\mathsf{p}}^{t} is very peaky, which means most of the probability mass is obtained by i ∗ superscript 𝑖 i^{*} , then all other p i subscript 𝑝 𝑖 p_{i} will decrease. In other words,
the probability mass of all other p i subscript 𝑝 𝑖 p_{i} is squeezed to p i ∗ subscript 𝑝 superscript 𝑖 p_{i^{*}} .

We continue the analysis in Claim 3B but consider a more extreme influence on $e^{z_{j}^{t}}$.
For this peaky $\bm{\mathsf{p}}^{t}$,
we might have an very large $e^{z_{i^{*}}^{t}}$ that dominates $\alpha$.
In other words, $\alpha_{i}\approx\frac{e^{z_{i^{*}}^{t}}}{\beta_{i^{*}}\cdot e^{z_{i^{*}}^{t}}}=\frac{1}{\beta_{i^{*}}}$.
Then for any $i^{\prime}$ we want to observe,
the $\alpha_{i^{\prime}}\approx\frac{1}{\beta_{i^{*}}}<1$.
In other words,
the model’s predictions on all dimensions other than the one with the highest confidence in $\bm{\mathsf{p}}^{t}$ will decrease.

Last, we analyze the influence of $p_{y}$ to explain why “imposing a large negative gradient on the valley region” makes the squeezing effect more serious.

##### Claim 4: Smaller p y t superscript subscript 𝑝 𝑦 𝑡 p_{y}^{t} makes those non-max p i subscript 𝑝 𝑖 p_{i} easier to decay, i.e., a stronger squeezing effect.

This is also a general trend that is observed in the experiments in [Figure 19](#A3.F19).
Intuitively, since the model is already confident that $y$ cannot be the correct label (i.e., $p_{y}$ is very small),
letting the model further decrease the prediction on $p_{y}$ does not make sense.
We can also use the analysis above to understand how it happens.
As illustrated in [Equation 35](#A3.E35),
where the value of $\beta$ is decomposed into three subgroups.
Recall the definition of $\alpha_{i}$,
we know all $\beta_{j}<1$ contribute to the hypothesis that $p_{i}$ increases after this update,
where the strength of this contribution is controlled by $e^{z_{j}^{t}}$.
Since a $p_{y}^{t}$ small means a small $e^{z_{j}^{t}}$,
the influence of $\beta_{j=y}\ll 1$ is significantly weakened under this scenario.
In other words,
$\alpha_{i}<1$ is more likely to occur for all possible $i$,
which means the squeezing effect (all $p_{j\neq y}$ decreases) becomes more serious.

##### Claim 5: The learning rate with a larger absolute value | η | 𝜂 |\eta| and a larger feature norm ‖ ϕ ​ ( 𝘅 ) ‖ 2 2 superscript subscript norm italic-ϕ 𝘅 2 2 \|\phi(\bm{\mathsf{x}})\|_{2}^{2} will amplify all the trends, maybe more serious than our expectation.

Throughout our analysis,
the equivalent learning rate $\eta^{\prime}<0$ is a shared scalar in all $\beta_{j}$.
Hence larger $|\eta^{\prime}|$ can amplify all the trends aforementioned.
Furthermore,
recall the shape of an exponential function $e^{x}$,
where a small change of $x$ (especially when $x>1$) will make $e^{x}$ changes a lot.
Then the terms $\beta_{j\neq y}=e^{-\eta^{\prime}(1+p_{j}^{t}-p_{i}^{t})}$ in Case 1 and $\beta_{j=y}=e^{-\eta^{\prime}(p_{j}^{t}-p_{i}^{t}-1)}$ in Case 2 will play a stronger role if we use a larger learning rate $|\eta|$ or the norm of features is larger.

### C.2 Verify the Squeezing Effect using a Simple Experiment

Figure: Figure 18: Illustration of how big positive and negative gradients influence the model’s prediction.
Refer to caption: /html/2407.10490/assets/x18.png

Figure: Figure 19: Experimental verification of the “squeezing effect” illustrated in [Figure 18](#A3.F18) using a simple multi-class logistic regression task.
Refer to caption: /html/2407.10490/assets/x19.png

Let us analyze a simple example to get an intuition.
We set $V=50$, $d=5$, $|\eta|=0.5$, and a randomly generated $\phi(\bm{\mathsf{x}})$.
In the first row of [Figure 19](#A3.F19),
we consider the model updates its parameters using standard SGD assuming the label of this $\bm{\mathsf{x}}$ is 21.
Specifically,
we randomly generate $\bm{\mathsf{w}}^{0}$ by sampling each parameter from a standard Gaussian distribution and calculate $\bm{\mathsf{w}}^{1}$ using [Equation 28](#A3.E28).
The two curves in each panel demonstrate the model’s predicted distribution before and after this update.
As we expected,
the positive vector on the 21st class “pull up” $\bm{\mathsf{p}}^{0}(y=21)$ and “push down” all other $\bm{\mathsf{p}}^{1}(y)$ at the same time.
This trend is quite consistent under different settings (i.e., different choices of $V,d,\bm{\mathsf{x}},\eta,\bm{\mathsf{w}}^{0}$, etc.),
which can be depicted by the first panel in [Figure 18](#A3.F18).

We then set $\eta=-0.5$ to simulate the negative gradient in DPO and consider three different settings.
First,
we assume the model’s prediction on $\bm{\mathsf{x}}$ is relatively flat,
as demonstrated in the second row of [Figure 19](#A3.F19),
where the predicting probability of every class is around 0.02.
The negative gradient is imposed on $y=42$,
a randomly selected number.
We see the negative adaptation vector “push down” $\bm{\mathsf{p}}^{1}(y=42)$ heavily and re-assign those decreased probability mass evenly to all other classes,
as illustrated in the second panel in [Figure 18](#A3.F18).

Although the behavior described above follows our intuitions well,
a flat $\bm{\mathsf{p}}^{0}$ is not common in LLM’s finetuning.
Because finetuning usually starts from a pre-trained $\bm{\mathsf{w}}$,
where the model’s prediction would likely be non-uniform.
So in the third row of [Figure 19](#A3.F19),
we consider a more practical $\bm{\mathsf{w}}^{0}$ that leads to a multi-mode $\bm{\mathsf{p}}^{0}$.
In this example,
the model has relatively high confidence in classes 5 to 11 and low confidence in all other dimensions.
We set the target label as 11 (i.e., the one in the model has the highest confidence) and use $\eta=-0.5$ to “push down” the model’s prediction on this class.
As demonstrated by the blue curve,
$\bm{\mathsf{p}}^{1}(y=11)$ decreases a lot as we expected.
However,
different from the flat $\bm{\mathsf{p}}^{0}$ case,
where the model evenly assigns the reduced probability mass to all other $\bm{\mathsf{y}}$,
the model in this example “squeezes” the mass to those confident predictions, i.e., classes 6, 9, and 10,
leaving the confidence of other classes almost unchanged.
Such a trend is consistent when the negative gradient is imposed on the “peaky” region of a non-uniform distribution,
as illustrated in the third panel in [Figure 18](#A3.F18).

The previous setting simulates the on-policy DPO well,
where the rejected examples $\bm{\mathsf{y}}_{u}^{-}$ are sampled from the high confidence region of the model’s predictions.
Then, what will happen if we conduct off-policy DPO and impose a big negative gradient on those classes that already have very low confidence?
See the fourth row of [Figure 19](#A3.F19),
where we use the same $\bm{\mathsf{w}}^{0}$ and $\eta$ as in the previous case.
The only difference is that we change the label of $\bm{\mathsf{x}}$ to 42,
where $\bm{\mathsf{p}}^{0}(y=42)$ is very small (roughly $10^{-5}$) before training.
The behavior in this setting is quite interesting:
we first observe a big increase on $\bm{\mathsf{p}}^{1}(y=11)$,
which means the model “squeezes” the probability mass to the most confident one in $\bm{\mathsf{p}}^{0}$,
similar to the previous setting.
More interesting,
the predictions on all other $\bm{\mathsf{y}}$ are heavily “pushed down”,
even including classes 6, 9, and 10, whose confidence is relatively high before training.
In the last two panels of [Figure 19](#A3.F19),
we set $V=1000$ and find this trend is more obvious (that might be because the absolute value of the efficient learning rate, which depends on $\|\phi(\bm{\mathsf{x}})\|$, becomes larger).
Since the vocabulary size of a common LLM is usually more than 50k, the squeezing effect in real systems would be non-negligible even if the learning rate is small.
Such a trend is also quite consistent as long as we impose a big negative gradient on the “valley” region of the model’s prediction,
as illustrated in the last panel in [Figure 18](#A3.F18).
Now we can answer the question of why all observing $\pi_{\theta^{t}}(\bm{\mathsf{y}})$ decreases and where the probability mass has gone:

For each token, the probability mass is squeezed to the one with the highest confidence.

Note that the tokens with the highest confidence do not necessarily form a preferred response:
it just reinforces the prior knowledge contained in $\theta^{0}$,
which could be a drawback for off-policy DPO.

The hypothesis above is not only supported by this simple logistic regression problem but also by many consistent trends in LLM’s finetuning experiments.
First,
by comparing the average decaying speed of the $\pi_{\theta^{t}}(\bm{\mathsf{y}})$ when the model SFT different epochs before DPO (in [Figure 15](#A2.F15)),
we notice that longer SFT leads to a more peaky $\pi_{\theta^{0}}(\bm{\mathsf{y}})$ and hence leads to a faster decaying speed of all non-argmax responses.
That is because the longer SFT stage will eventually push down $\pi_{\theta^{0}}(\bm{\mathsf{y}}_{u}^{-})$ more.
Hence in the DPO stage,
the big negative gradient is imposed on a deeper valley region,
which makes the squeezing effect stronger.
Second,
to directly verify this hypothesis,
we track the sum of the log-likelihood of the tokens with the largest confidence and call it “argmax confidence”,
i.e., $\sum_{l}\pi_{\theta^{t}}(\text{argmax}_{\bm{\mathsf{y}}_{l}\in\mathcal{Y}_{l}}\bm{\mathsf{y}}_{l}\mid\bm{\mathsf{x}},\bm{\mathsf{y}}_{1:l-1})$.
As illustrated in the last panel in [Figure 5](#S5.F5),
the argmax confidence keeps increasing while all other $\pi_{\theta^{t}}(\bm{\mathsf{y}})$ decreases:
the missing probability mass is found!
Last,
in the dataset-extension method we proposed in [Section 6](#S6) and [Appendix D](#A4),
we train the model using both $[\bm{\mathsf{x}},\bm{\mathsf{y}}_{u}^{+}]$ and $[\bm{\mathsf{x}},\bm{\mathsf{y}}_{u}^{-}]$ during SFT to also “pull up” the $\bm{\mathsf{y}}_{u}^{-}$ region before conducting DPO.
Then, we observe compared with the standard training flow,
i.e., SFT using $[\bm{\mathsf{x}};\bm{\mathsf{y}}_{u}^{+}]$ first and then DPO,
the proposed flow has a lower “argmax confidence” during DPO.
That is because we pulled up $\pi_{\theta^{0}}(\bm{\mathsf{y}}_{u}^{-})$ during the modified SFT stage,
the big negative gradient is then imposed on the peaky region rather than the valley region of the model’s prediction.
Such a change in turn weakens the squeezing effect,
as illustrated in [Figure 21](#A4.F21).

## Appendix D A Simple Method to Improve Alignment

### D.1 Pinpointing the drawback of off-policy DPO

Based on our observations and analysis above,
we speculate that “imposing big negative gradients on the valley region” is one of the bottlenecks of off-policy RL-free methods.
Starting from this hypothesis,
we believe introducing on-policy sampling has the potential to mitigate this problem,
as demonstrated in SPIN [] and other online algorithms [].
However,
we also speculate that these methods improve the model’s performance not only by mitigating the squeezing effect.
Hence to figure out to what extent the squeezing effect can harm the model’s performance,
we propose a simple yet effective method to isolate its influence.
As this method can directly mitigate this effect,
it can also be considered as an ablation study of this interesting phenomenon.

Figure: Figure 20: Illustration of the proposed method and baseline. “E” is short for the “dataset extension”.
Refer to caption: /html/2407.10490/assets/x20.png

### D.2 A simple method inspired by learning dynamics

As illustrated in [Figure 20](#A4.F20),
where the baseline method is a standard SFT-then-DPO pipeline.
The proposed method is very simple.
We only need to augment the dataset used in SFT by adding $(\bm{\mathsf{x}},\bm{\mathsf{y}}_{u}^{-})$ pairs for each sample into it.
All other settings are unchanged.
The motivation for this method is also quite simple:
as SFT can pull up the region of supervised $\hat{\bm{\mathsf{y}}}$ and we don’t want the model to impose big negative gradients on a valley region,
we can just pull up those $\bm{\mathsf{y}}_{u}^{-}$ before DPO.
Furthermore,
as demonstrated in the third panel in [Figure 18](#A3.F18) and [Equation 19](#A1.E19),
the negative gradient in DPO would be strong enough to push down $\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{-})$,
because the gradient will be large if the model cannot separate $\bm{\mathsf{y}}_{u}^{+}$ and $\bm{\mathsf{y}}_{u}^{-}$ well.
In other words,
under DPO’s loss,
there is no need to worry about the model overfitting those $\bm{\mathsf{y}}_{u}^{-}$ during SFT.

Figure: Figure 21: Learning dynamics of the baseline and the proposed method with training data extension. Key trends to observe: 1.) Baseline and the extend method have similar behavior on $pi_{\bm{\mathsf{y}}^{+}_{u}}$ during SFT; 2.) The extend method considerably increase $pi_{\bm{\mathsf{y}}^{-}_{u}}$ during SFT; 3.) The squeezing effect of the extend method is weaker (all other responses decay slower and the confidence on argmax response increases slower). (Antropic-HH)
Refer to caption: /html/2407.10490/assets/x21.png

Figure: Figure 22: Learning dynamics of the baseline and the proposed method with training data extension. (UltraFeedback)
Refer to caption: /html/2407.10490/assets/x22.png

### D.3 Experimental verification

To verify our analysis,
we conduct experiments by finetuning a pretrained Qwen1.5-1.8B [] model using Antropic-HH dataset [] (we use a subset containing 5000 random examples from the training split).
The pipelines of different methods are demonstrated in [Figure 20](#A4.F20).
In this experiment,
we call the pretrained model $B_{0}$ (and $E_{0}$, which is identical to $B_{0}$),
which is an official checkpoint pretrained by [].
Model $B_{1}$ and $E_{1}$ are the ones after SFT, which are different for these two methods.
Model $B_{2-2/4/6}$ and $E_{2-2/4/6}$ are the models finetuned using DPO for 2/4/6 epochs.
All the settings (except the starting model) of the DPO stage are the same for these two methods.

We first observe the learning dynamics of these two methods in [Figure 21](#A4.F21),
where all the trends support our analysis quite well.
See the first two panels that compare $\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{+})$ and $\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{-})$ respectively.
It is clear that these two methods have an almost identical curve on $\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{+})$ in the SFT stage but behave quite differently on $\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{-})$:
because we directly train the model using $(\bm{\mathsf{x}},\bm{\mathsf{y}}_{u}^{-})$ in the proposed method.
Then, after the SFT stage,
we conduct DPO using identical settings for these two methods.
From the first three panels,
we can observe the decay speed of all curves of the proposed method is smaller than its counterpart in the baseline.
That is the benefit introduced by “pulling up” the $\pi_{\theta^{0}}(\bm{\mathsf{y}}_{u}^{-})$ region before conducting DPO.
With this specific design,
the big negative gradients in DPO are imposed on the peaky region (the behavior is like the third panel in [Figure 18](#A3.F18)) rather than the valley region (see the fourth panel),
hence the squeezing effect is successfully restrained.
The results in the last panel of [Figure 21](#A4.F21) are also a strong verification of the whole picture.
During the SFT stage,
the observed “argmax-probability” of the proposed method is higher than the baseline,
because we impose twice “pull up” pressure, i.e., those for $(\bm{\mathsf{x}},\bm{\mathsf{y}}_{u}^{-})$,
compared with the baseline.
However,
at the beginning of DPO,
we observe a clear drop in the orange curve.
That is because the negative gradients are exactly imposed on those $\bm{\mathsf{y}}_{u}^{-}$ (in the second panel of [Figure 21](#A4.F21), $\pi_{\theta^{0}}(\bm{\mathsf{y}}_{u}^{-})$ is already very high).
Furthermore, at the end of DPO,
we see the “argmax-probability” of the proposed method is significantly lower than the baseline setting,
which implies that the squeezing effect is restrained in our setting.

In order to figure out whether the model trained using the proposed flow,
which successfully restrains the squeezing effect,
indeed does alignment better,
we conduct pair-wise comparisons of these models’ responses and report their win rate as in [].
Specifically,
we first randomly select 1000 test questions from the test split of Antropic-HH and generate 1000 responses by feeding the prompts to each of these models (we use the default sampling setting provided in []).
Then,
with the prompt template provided in [Figure 23](#A4.F23),
we evaluate the win rate of the responses pairs using GPT3.5-turbo and Claude3-Haiku.
Here we report the average win rate of different comparisons (the degenerated responses are not compared, so the number of compared examples is slightly smaller than 1000).
Note that a win rate greater than 0.5 means the method that comes first is preferred by the evaluator.

- 1.
Compare models after SFT: $E_{1}$ v.s. $B_{1}$, win rate is 0.4729 and 0.4679;
- 2.
Demonstrate benefits of DPO:
a.
$B_{2-4}$ v.s. $B_{1}$, win rate is 0.6727 and 0.6411;
b.
$E_{2-4}$ v.s. $E_{1}$, win rate is 0.6898 and 0.7321;
- 3.
Compare the proposed method and baseline after DPO for different epochs:
a.
$E_{2-2}$ v.s. $B_{2-2}$, win rate is 0.6518 and 0.5151;
b.
$E_{2-4}$ v.s. $B_{2-4}$, win rate is 0.6928 and 0.6045;
c.
$E_{2-6}$ v.s. $B_{2-6}$, win rate is 0.6667 and 0.5432;
- 4.
Compare the best $E_{2-4}$ with other 2 checkpoints:
a.
$E_{2-4}$ v.s. $E_{2-2}$, win rate is 0.6853 and 0.5517;
b.
$E_{2-4}$ v.s. $E_{2-6}$, win rate is 0.6324 and 0.5316;

In the first comparison,
we find the model trained using both $(\bm{\mathsf{x}},\bm{\mathsf{y}}_{u}^{+})$ and $(\bm{\mathsf{x}},\bm{\mathsf{y}}_{u}^{-})$ loses more (win rate is smaller than 0.5),
which makes sense because $E_{1}$ assigns higher probabilities on those less preferred responses.
In the second comparison,
the model fine-tuned using DPO indeed aligns with human value better.
The win rate of the proposed method is slightly higher,
which might also be explained as $E_{1}$ leaving more space for improvement.
Hence we then directly compare the models after DPO in these two methods in the third group.
In this group,
all models in the proposed method win the baseline counterparts by a large margin,
which demonstrates the effectiveness of our proposed method.
Furthermore,
we find the evaluation made by Claude is more reserved compared with GPT (the numbers are smaller).
However,
the trends among the comparisons in this group are consistent:
$E_{2-4}$ brings the largest improvement,
which is potentially the best model.
This fact is verified in the fourth group comparison,
where we evaluate $E_{2-4}$ against $E_{2-2}$ and $E_{2-6}$.
The results demonstrate that both a too-long or too-short finetuning stage using DPO is not the best choice.

Figure: Figure 23: Prompt used for evaluating model’s response (from []), an example feedback from GPT3.5-turbo, and two examples of the “degenerate” effect described in []. Although both $B_{2}$ and $E_{2}$ inevitably generate such degenerate responses, we find this phenomenon is less common in the proposed method.
Refer to caption: /html/2407.10490/assets/x23.png

## Peer Reviews

confidence: 4
contribution: 4
initial_rating: 8
presentation: 4
questions: 1. Can you discuss the analytic or empirical argument for "Peakier πθt suffer a more serious squeezing effect." in a bit more detail? I see arguments titles Claim 3A and 3B on pg 27, but they seem to be more intuitive that formal. It could be useful to separate the claims in the Sec 3.3 enumeration into those with direct proof/analytic evidence and those that are more informal or empirical -- this would not weaken the arguments, but rather making these subtle distinctions clear would increase the soundness of the presentation.

2. The color coding of the variables in section 2 and 3 is non-standard but fine if the authors like this (and no other reviewers mind it). However, it seems inconsistent to not continue this into Sec 4 and 5? Additionally, if there is a way to continue the scheme into the figures, especially Fig 2, that would make more full use of the fact that  the x and y's are colored in the related written sections.
soundness: 4
strengths: 1. They motivate and ground the study of training dynamics for LLMs well and highlight key difficulties in applying standard influence function style analyses to LLM finetuning and preference optimization, a highlight of the preliminary material.
2. The formalism and analysis in Section 3 is generally clear and well written. Figures 2 and the bullet point enumeration in 3.3 are concise presentations of the core claims of the analysis.
3. The experimental design of the empirical verification section is well done and cleanly visualized, particularly the use of completions $y^{+}$ gpts, gptf, test, hum.
4. Proposal for a mitigation to the "squeezing effect" (while not the main contribution of the paper) is well motivated by the analysis and simple to implement.
summary: This work studies the learning dynamics of LLM finetuning by analyzing the way in which likelihoods assigned by the model to different completions of training prompts evolve throughout the finetuning process. By explicitly writing out the single step update rules in terms of the change in model parameters and change in the functional output of the model, and then leveraging the empirical NTK for the LM head/logit layer of the model, they show that one can analytically describe the sample wise influence dynamics of both the SFT and DPO objectives for LLMs. Corroborating prior observations of model distributions degenerating during (off-policy) DPO, their analysis describes a "squeezing effect" due to the performing of gradient _ascent_ during standard preference tuning, and they validate this and other qualitative predictions via the finetuning of 1B-3B parameter LLMs on carefully augmented preference tuning data.
weaknesses: 1. The clarity of Sec 3.1 when discussing the causal masking and teacher-forced production of the full next-token logits set could be improved, though this is admittedly tricky.  It is possible that for some readers, a diagram of the matrix structure here could be helpful since most papers don't tackle the more complex formulation of the influence problem and so readers may not be clear on it. (The reviewer imagines a teacher-forced causally masked model forward on input and label sequences X,Y as a forward on an augmented view of X,Y according to causal mask M, such that if the length of X+Y=N, the input has X' now has dim NxN (lower triangular consisting of all prefixes of the sequence X+Y) and we consider the output to be labels Y_i for each row of X' corresponding to the next token following the tokens in row X'_i.)

2. Additionally, while the analysis relies on the kernel values between the feature maps of $\mathcal{X_0}$ and $\mathcal{\tilde{X_u}}$ to implement the pressure transfer between various samples during learning, these intermediate values are not actually experimentally estimated. To concretely tie the analysis to the empirical observations, it would be useful to compute examples of the decomposition values for the terms in Eq 5 and Eq 7. If this intermediary relationship can be shown as clearly as the loss trends described in Figures 3 and 4, a stronger assertion of causality between the theoretical model described by Eq 5 and Eq 7 and the empirical learning dynamics can be made, strengthening the impact of the work.

---

confidence: 3
contribution: 4
initial_rating: 8
presentation: 4
questions: How does the squeezing effect differ from overfitting with DPO?
soundness: 4
strengths: 1) The paper has strong theoretical and experimental backing for their analysis.
2) First paper to propose a framework extending learning dynamics to LLMs.
summary: The paper extends the existing learning dynamics for deep learning systems to LLMs and  argues that doing so is a non-trivial problem due to high dimensionality and sequential nature of both inputs and outputs. The paper talks about a “squeezing effect” where running DPO for too many epochs causes outputs to deviate from ideal behavior.
weaknesses: 1) Did not find any specific weaknesses.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 4
questions: Line 31: "Many interesting phenomena" -- such as? (not saying there aren't, but this is an odd statement to have without providing at least some refs)

Line 61: I feel that this is conflating different things, i.e some notion of "interpretability" (some "compact" understanding of how model predictions/behavior are determined by the parameters), and notions of learning dynamics (how model parameters evolve during training). One can in principle discuss the one without the other, and it would be better to spell out more clearly / explicitly the current motivation.

Figure 1: The red arrows on panel a, top do not actually represent $\Delta\theta$ but rather $\Delta \pi$. It's also not clear why a continuous curve is plotted, given the fact that $\pi$ is discrete -- why not show all 10 digits as bars (as is later done in panels (c) and (d))? Perhaps most importantly, it's not clear whether or not this figure should be understood only qualitatively, or is this the result of an actual computation.
The same comments apply for all panels in Fig.1 (b). If this is an actual result then many details are missing, both in presentation (bars; scale of the y-axis) and perhaps more crucially, at what stage of learning this is done. The entire argument relies on some implicit assumption that "4" images are similar to "9" images, but this crucially depends on the current learned representation of the model (since this is a "feature-learning" rather than "lazy" regime). If this is just an illustration, then it is highly misleading presenting it along with panels (c) and (d) which seems to be actual numeric results.
I'm also unable to understand how to interpret panel (c). The caption says "accumulated change" but the the charts seems to indicate absolute probabilities. It's also unclear what is being tracked, over how many epochs, and what are the examples being presented. The main text (lines 134--138) is unhelpful and reads as another explanation of what's actually illustrated in panels (a)-(b) (i.e., the single-step influence)
Overall, the entire discussion of the motivating example in Section 2 is hard to follow. I think the overall intention of illustrating the idea in a simple example is a good one, but the example itself should be made much more clear.


Line 233: "Any no assumptions" -> "no assumptions" 


Line 253: Calling this "unexpected behavior" might be too strong, see previous literature
Line 314: "Fine grind" -> "Fine grained"
soundness: 4
strengths: The paper is overall well written, clearly explains the motivation and the approach. The use of simple "pedagogical" examples also helps in communicating the main message of the paper.

The main contribution seems to be in the application of the theory to the different loss functions used in LLM finetuning, making explicit use of the "decomposition" of the effective learning dynamics into 3 different terms, and identify which one directly depends on the loss function being used. This is demonstrated to be productive as it can provide (at least a preliminary) explanation for different observed behaviors.

Overall, this is an interesting paper and the results are likely to be of interest to the community.
summary: The learning dynamics of a linear-softmax classifier on top of fixed features is analyzed. The method is applied to the settings of LLM finetuning, by considering particular loss/objective functions specific to this settings and analyzing their effect on learning dynamics. The results helps elucidating previously observed phenomenon in LLM finetuning, and constructing new methods for overcoming potential "bottlenecks" in the process.
weaknesses: There are two major limitations here that, while are (somewhat) acknowledged by the authors, can benefit from a more careful discussion and/or analysis.

1. The first is the obvious limitation that the entire analysis is being done under the assumption that the "feature map" is held fixed, and only the classification layer is changing during learning, but the extent to which this assumption holds in practice is not being evaluated at all. Without such an evaluation (for example, either by quantifying/tracking the change in $\mathcal{K}^t$, or by repeating some experiments when the weights are *actually* frozen) it is hard to say whether the theoretical explanation about the squeezing effect is in fact an important part of the observed phenomena (or whether it is largely / to some extent driven by more complicated dynamics that involves changes in the feature map as well). I think this should, at the very least, be acknowledged more explicitly in the text.
   
2. The second limitation is more nuanced, and has to do with the relying of some intuitive understanding of "similarity" (between prompts/examples), that may or may not be valid. Again no evaluation is offered -- despite the fact that the authors explicitly specify how this similarity should be understood (i.e., it is measured by the eNTK, $\mathcal{K}$ ). It is far from being clear a-priori that the learned features agree with some intuitive judgement of similarity. Moreover, what "dissimilar" means, in the extremely vast space of strings, is also non-trivial. For example (and as pointed out by the authors), it might be that all "well-formed"/grammatical sentences are somewhat similar, and the model is able to push probability mass to entirely non-grammatical/nonsensical strings. This leads to some arguments being made only on the basis of some intuition. For example, "the model’s confidence on y+u keeps increasing and the update energy... gradually decreases"  (Line 350) -- this seems to be inconsistent with the fact that we see no slow-down of the updates for $\mathbf{y}^+_{u}$ itself.

Finally I would also like to point out that peakiness (/"squeezing") effect in text generating models followed by RL tuning is a phenomena that has, in fact, been observed and to some reason explained before, including the observation that the on/off policy choice might play an important role in dealing with the "exposure bias". The paper will benefit from referring to some of these previous works, for example:
- Caccia et al., Language GANs Falling Short, (arxiv 2018 / ICLR 2020)
- Choshen et al., On the Weaknesses of Reinforcement Learning for Neural Machine Translation (arxiv 2019 / ICLR 2020)
- Kiegeland and Kreutzer, Revisiting the Weaknesses of Reinforcement Learning for Neural Machine Translation (naacl 2021)

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. In your page 5, "make any no assumptions" -> "make no assumptions"
2. In your appendix A, add more derivations to show that -(e_y_u^+ / pi)^T [A(x_u)]_l = pi(y_l | x_u) - e_y_u^+. All parts in Appendix A until this point is easy to follow.
3. In your page 16, lines 712-713, are there some nablas that are left behind?
soundness: 3
strengths: 1. Analyzed the theoretical process behind DPO gradient updates. Explained deeply how the effect of gradient updates with respect to the training examples affect the model's performance on unrelated examples. Provided a theoretical framework for (potentially) understanding the origin of hallucinations.
2. Explained a common problem in DPO that was not illustrated well in prior work. For example, Rafael et.al. 2024 only vaguely claimed that "DPO decreases the likelihood of all data in favor of extrapolated responses", but this paper clearly indicated that the decreased likelihood all (or mostly) added to the greedy decoding sample (y^*).
3. The paper is well written and easy to follow.
summary: This paper analyzed the learning dynamics of LLM fine-tuning, in particular, the change of the loss function for other examples delta f(x_o) during training time parameter updates (from gradient descent with respect to input examples x_i), for both SFT and DPO. For SFT, the research used MNIST classification as an illustrative example to indicate that if x_o is similar to x_u but classified differently, the predicted probability of the x_u's class for x_o also got pulled up. For DPO, the research found that log likelihoods of both the positive examples y_u^+ and negative examples y_u^- all decreased, and the only example with log likelihood increase is the greedy decoding result y_u^*. This phenomenon could potentially explain the increased repetitiveness of DPO generated results towards the end of fine-tuning, and the increase of halluncinations during SFT. The researcher proposed a method to mitigate this effect by first SFT-ing on both positive examples and negative examples, and then perform DPO.
weaknesses: Disclaimer: Only checked the main text and the Appendix A. Did not check Appendices B,C,D.

1. Need a "related work" section. The paper in its current form does not explain the relations between "learning dynamics" with related literature, for example, preventing negative transfers during transfer learning, neural tangent kernels, or loss landscape that enabled adversarial attacks. Offering a "related work" section whereas reducing the theoretical focus a little bit would help ICLR readers better position your work.

2. Need to indicate assumptions in your prop 1 and section 3.2 clearer. As far as I could see, there are at least 2 assumptions: 
- Higher order terms has little to no impact to learning dynamics
- K changes slowly during the initial training stages and thus has a limited impact to training dynamics

Please list out these assumptions (potentially what I did not listed out) so that users will follow and understand the limitations of your analysis.

3. Should design more experiments that 

- (a) During off-policy DPO, show that no other sentence's log probability significantly increased other than y^*. Although your theoretical analysis seems to have proved this, we still need to show this point in experiments due to your analysis assumptions. The sampling should be much more extensive to show the non-existence of such examples (and statistically significant)
- (b) For SFT, quantitatively show the causal effect of increased hallucinations due to the "pull up" effect of similar but negative examples (scaling up more than MNIST)
- (c) For both experiments, show that neuron tangent kernels (your K-term) did not change much, and thus the learning dynamics can be fully attributed to your G-term
- (d) Qualitatively demonstrate the repetitiveness of off-policy DPO towards the end of fine-tuning
- (e) Quantitatively measures how your proposed approach reduced hallucinations / answer repetitiveness via more rigorous benchmarks, other than just a measurement of the win rate

Meta-review: ### Summary of the paper
The paper analyzes the learning dynamics of LLM finetuning by examining how model-assigned likelihoods to different prompt responses evolve during training. Key findings and contributions include:
- Development of an analytical framework to describe sample-wise influence dynamics for both SFT and DPO objectives using empirical NTK 
- Identification of a "squeezing effect" during DPO training where gradient ascent causes model distributions to degenerate
- Studying cause of hallucination by demonstrating that when training on one response, similar responses get "pulled" toward similar prompts, even if incorrectly
- Finding out where the probability mass has gone in DPO: finding that during DPO, log likelihoods decrease for both positive and negative examples, with only the greedy decoding result showing increased likelihood

### Strengths
- Strong theoretical foundation with clear formalism and analysis
- Well-designed experimental verification
- Provides both theoretical analysis and experimental verifications for several unsolved but popular phenomenon in LLM finetuning, like "where has the probability mass gone in DPO", what is the cause of repeated patterns after finetuning, and the cause of hallucination (might have been better understood in previous works compared to the other two, but conclusions are consistent)
- Practical contribution through proposed mitigation strategy for the "squeezing effect"

### Weaknesses
- The theoretical analysis assumes fixed features with only classification layer changes. This assumption may not hold well in practice. However, the authors added the “relatively stable eNTK” assumption together with new experiments in the rebuttal to show the eNTK does change noticeably.
- Lack of quantitative demonstration of hallucination increases in SFT. But this has been shown in previous works.

Decision: Accept (Oral)