TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weights

## Abstract

Abstract Direct Preference Optimization (DPO) has been widely adopted for preference alignment of Large Language Models (LLMs) due to its simplicity and effectiveness.
However, DPO is derived as a bandit problem in which the whole response is treated as a single arm, ignoring the importance differences between tokens, which may affect optimization efficiency and make it difficult to achieve optimal results.
In this work, we propose that the optimal data for DPO has equal expected rewards for each token in winning and losing responses, as there is no difference in token importance.
However, since the optimal dataset is unavailable in practice, we propose using the original dataset for importance sampling to achieve unbiased optimization.
Accordingly, we propose a token-level importance sampling DPO objective named TIS-DPO that assigns importance weights to each token based on its reward.
Inspired by previous works, we estimate the token importance weights using the difference in prediction probabilities from a pair of contrastive LLMs. We explore three methods to construct these contrastive LLMs: (1) guiding the original LLM with contrastive prompts, (2) training two separate LLMs using winning and losing responses, and (3) performing forward and reverse DPO training with winning and losing responses.
Experiments show that TIS-DPO significantly outperforms various baseline methods on harmlessness and helpfulness alignment and summarization tasks. We also visualize the estimated weights, demonstrating their ability to identify key token positions.

### 1 Introduction

The importance of Large Language Model (LLM) alignment (Ji et al., ) techniques has grown alongside the increasing capabilities of LLMs. These techniques aim to align LLMs with human values, ensuring the generation of helpful and harmless content (Bai et al., ). Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al., ) is a common alignment approach that trains a reward model on human-labeled preference data and optimizes the LLM using reinforcement learning methods like Proximal Policy Optimization (PPO) (Schulman et al., ) to maximize the generated reward under the reward model. However, RLHF is relatively complex due to the need for reinforcement learning techniques.

To simplify alignment process, Direct Preference Optimization (DPO) (Rafailov et al., ) leverages the relationship between policy and reward functions to optimize both simultaneously without reinforcement learning. However, DPO is derived from a sequence-level Bradley-Terry model (Bradley & Terry, ), which only focuses on preference relationships between two sequences while ignoring the contribution of each token. However, as shown in Fig. 1, in real data, different tokens have different rewards. Even in winning responses, there are tokens with low rewards. Optimizing all tokens uniformly reduces optimization efficiency. Although Rafailov et al. () demonstrate that DPO possesses a certain degree of token-level interpretability, this does not alleviate its lack of consideration for token importance during optimization.

Recently, some studies have argued that different tokens in DPO should not be treated equally, but these studies often require changes to the data construction process to identify more critical tokens.
For example, Xie et al. () considered token weights when collecting data using Monte Carlo Tree Search, while Lai et al. () used LLMs like GPT-4 to annotate key steps in reasoning problems. In this work, we argue that the most stable form of DPO loss occurs when tokens in winning and losing responses have identical expected rewards, respectively, eliminating the need to consider token importance. Since real data cannot meet this condition, we propose TIS-DPO, which performs token-level importance sampling of the optimal data distribution using the actual data distribution. By weighting each token based on its reward, the final optimization process becomes unbiased to DPO using the optimal data distribution.

Figure: Figure 1: In real data, different tokens have varying rewards, with low-reward tokens present even in winning responses. DPO treats all tokens equally, introducing noise and reducing optimization efficiency. Our TIS-DPO performs importance sampling on the optimal data distribution (where each token has equal reward) using actual data, introducing token weights to improve optimization efficiency.
Refer to caption: x1.png

In practice, as token weights are unknown, we estimate them through their rewards. Inspired by previous work (Rafailov et al., ), we use the difference in token prediction probabilities between contrastive LLMs to estimate each token’s reward. Here, contrastive LLMs refer to LLMs with positive and negative preferences. Specifically, we employ three methods to construct contrastive LLMs: (1) using contrastive prompts to guide the original LLM; (2) training two LLMs using winning and losing responses with supervised learning; and (3) performing forward and backward DPO training using winning and losing responses, where backward DPO training involves swapping positive and negative preference data before DPO training.

Experimental results demonstrate that our TIS-DPO method outperforms other baseline algorithms on multiple datasets. Specifically, our approach shows significant improvements in harmlessness and helpfulness on the PKU-RLHF (Ji et al., ) and Antropic-HH (Bai et al., ) datasets, and substantial generation quality enhancements on the TL;DR (Völske et al., ) dataset. Among the three estimation methods, the forward and backward DPO-based approach performs best, while the effectiveness of prompt-based weight estimation depends on the actual data distribution, performing better on LLM-generated data. Finally, further analysis experiments validate the reasonability and accuracy of our estimated weights.

The main contributions of this work are as follows:

- •
We propose TIS-DPO, a novel approach that performs token-level importance sampling to address the limitations of treating all tokens equally in standard DPO.
- •
Three innovative methods for estimating token weights are introduced: contrastive prompts, supervised learning with winning and losing responses, and forward-backward DPO training.
- •
Extensive experiments demonstrate the effectiveness of TIS-DPO across multiple datasets. We show significant improvements in harmlessness, helpfulness, and generation quality compared to baseline algorithms, while providing analysis that validates the reasonability and accuracy of our estimated token weights.

### 2 Related Work

Aligning LLMs with human values is crucial for enhancing their utility and reducing harmful outputs, misinformation, and improving overall safety (Ji et al., ; Liu et al., ; ). Direct Preference Optimization (DPO) (Rafailov et al., ) has been widely applied to LLM alignment due to its convenience and effectiveness. Compared to RLHF (Ouyang et al., ), DPO has lower computational costs as it doesn’t require reinforcement learning techniques or training a reward model. However, DPO still has some issues, such as insufficient learning of positive samples (Feng et al., ). To address this, Pal et al. () designed new loss functions to encourage LLMs to maintain probabilities for positive samples, while Ethayarajh et al. () proposed KTO for model alignment by directly maximizing the utility of generated content instead of relying on traditional preference data. Another limitation with DPO is that it optimizes LLMs based on preferences from the entire response, ignoring that difference of token importance. Although Rafailov et al. () found DPO can do some token credit assignment, it still doesn’t directly model token importance. Zeng et al. () proposed token-level DPO but did not explicitly consider varying token importance. Some work has considered token weights during DPO training data collection (Xie et al., ; Lai et al., ).In this paper, we propose TIS-DPO (Token-level Importance Sampling DPO), which does not require modifying the original data construction process. Instead, it uses real data to perform importance sampling on the optimal data, assigning different importance weights to each token during optimization.

Importance sampling is a crucial technique in offline reinforcement learning (Levine et al., ; Prudencio et al., ) that allows for data sampling using policies different from the target policy, enabling direct training on pre-collected data. Previous importance sampling methods typically emphasized sequence-level importance sampling (Tajwar et al., ) without considering token-level distributions. In this work, for the DPO offline setting, we treat the winning and losing responses as samples drawn from two distinct reward distributions using importance sampling.

### 3 Preliminaries

Generally, RLHF (Ouyang et al., ) can be divided into two parts. Given a preference dataset $\mathcal{D}={(x,y_{w},y_{l})}$, where $y_{w}$ and $y_{l}$ are the winning response and losing response respectively, and $x$ is the given prompt, a reward model $r_{\phi}$ is first trained using the Bradley-Terry model (Bradley & Terry, ):

$$ $P_{\text{BT}}(y_{w}\succ y_{l}\mid x)=\frac{\exp(r_{\phi}(x,y_{w}))}{\exp(r_{ \phi}(x,y_{w}))+\exp(r_{\phi}(x,y_{l}))}.$ (1) $$

After obtaining the reward model $r_{\phi}$, the next step is to use PPO (Schulman et al., ) to optimize the LLM $\pi_{\theta}$, so that the LLM’s output has a higher reward, as shown in the following training objective:

$$ $\max_{\pi_{\theta}}\,\mathbb{E}_{x\sim\mathcal{D},y\sim\pi_{\theta}(\cdot\mid x )}\left[r_{\phi}(x,y)-\beta D_{\text{KL}}(\pi_{\theta}(\cdot\mid x)\parallel \pi_{\text{ref}}(\cdot\mid x))\right].$ (2) $$

Here, $D_{\text{KL}}$ measures divergence between $\pi_{\theta}$ and $\pi_{\text{ref}}$ (initial model).
Rafailov et al. () mathematically derived the optimal policy $\pi_{\theta}^{*}$ from reward model $r(x,y)$ as follows:

$$ $\pi^{*}(\mathbf{y}\mid\mathbf{x})=\frac{1}{Z(x)}\pi_{\text{ref}}(y\mid x)e^{r_ {\phi}(x,y)},$ (3) $$

where $Z(x)$ is the partition function. We could easily get $r_{\phi}(x,y)=\beta\log\frac{\pi^{*}(y|x)}{\pi_{\text{ref}}(y|x)}-Z(x)$ from Eq. 3 . Substituting into the Bradley-Terry model yields the DPO objective:

$$ $\mathcal{L}_{\text{DPO}}(\pi_{\theta};\pi_{\text{ref}})=-\mathbb{E}_{(x,y_{w}, y_{l})\sim\mathcal{D}}\left[\log\sigma\left(\sum_{i=1}^{n_{w}}\beta\log\frac{ \pi_{\theta}(y_{w}^{i}\mid x,y_{w}^{<i})}{\pi_{\text{ref}}(y_{w}^{i}\mid x,y_{ w}^{<i})}-\sum_{j=1}^{n_{l}}\beta\log\frac{\pi_{\theta}(y_{l}^{j}\mid x,y_{l}^ {<j})}{\pi_{\text{ref}}(y_{l}^{j}\mid x,y_{l}^{<j})}\right)\right],$ (4) $$

where we represent the DPO optimization objective as a token-level optimization objective. Here, $n_{w}$ and $n_{l}$ denote the number of tokens in the winning and losing responses, respectively. We demonstrate the equivalence of this objective to the original DPO in Appendix A.1.

Importance sampling is a technique for estimating properties of a target distribution using samples from a different distribution. It is particularly useful when the target distribution is difficult to sample from directly. The key idea is to reweight the samples from the sampling distribution to account for the difference between the distributions:

$$ $\mathbb{E}_{x\sim p}[f(x)]=\mathbb{E}_{x\sim q}[f(x)\frac{p(x)}{q(x)}],$ (5) $$

where $p$ is the target distribution, $q$ is the sampling distribution, and $\frac{p(x)}{q(x)}$ is the importance weight.

### 4 Limitations of DPO: Neglecting Token-level Importance Differences

Equation 4 shows that DPO assigns equal consideration to each token, uniformly increasing the reward for tokens in winning responses while decreasing the reward for tokens in losing responses. However, in reality, token importance varies greatly and even winning responses may contain low-reward tokens (as shown in Figure 1). As a result, DPO’s approach introduces substantial noise, reducing optimization effectiveness.

Recent work Zeng et al. () suggests that the overall reward can be decomposed into individual token rewards. We expect the average token reward of the winning response to be higher than that of the losing response to achieve more stable optimization. However, our theorem below indicates that greater fluctuations in token rewards within a response increase the likelihood of noise in the data itself.

###### Theorem 1 .

Assume the winning response has $n_{w}$ tokens, with each token’s reward as a variable $r_{w,i}$, where $r_{w,i}\in[a_{w},b_{w}]$ and $a_{w}$, $b_{w}$ are constants. Similarly, the losing response has $n_{l}$ tokens, with each token’s reward as $r_{l,j}$, where $r_{l,j}\in[a_{l},b_{l}]$. Let $S_{w}=\frac{1}{n_{w}}\sum_{i=1}^{n_{w}}r_{w,i}$ and $S_{l}=\frac{1}{n_{l}}\sum_{j=1}^{n_{l}}r_{l,j}$ represent the average reward of the winning response and losing response, respectively, and $t$ be a chosen constant. Then:

$$ $P(S_{w}\leq S_{l})\leq\exp\left(-\frac{2n_{w}t^{2}}{(b_{w}-a_{w})^{2}}\right)+ \exp\left(-\frac{2n_{l}t^{2}}{(b_{l}-a_{l})^{2}}\right),$ (6) $$

where $P(S_{w}\leq S_{l})$ is the probability of data noise, and we use the average reward to represent the relationship between the reward of each token and the overall reward of the response.

Theorem 1 indicates that the greater the difference in average rewards between the winning and losing responses, the higher the noise in the data and the less stable the optimization. We provide a detailed proof in Appendix A.2.

###### Theorem 1 .

### 5 DPO with Token-level Importance Sampling

#### 5.1 Token-level PPO Objective with Importance Sampling

According to Theorem 1, for more stable optimization, we need to ensure consistent rewards for token $y^{t}$ across all positions $t$. Therefore, we define the optimal dataset distribution $D^{*}$ as follows:

###### Definition 1 .

For all $x$ and $y^{<t}$ in optimal dataset $\mathcal{D^{*}}$, the next token $y^{t}$ is sampled from a distribution with the same expected reward $R^{*}$. That is, $D^{*}$ has the following property:

$$ $\forall(x,y^{<t}),\quad\mathbb{E}_{y^{t}\sim\mathcal{D}^{*}(\cdot\mid x,y^{<t} )}[r(y^{t}\mid x,y^{<t})]=R^{*}$ (7) $$

where $\mathcal{D}^{*}(\cdot\mid x,y^{<t})$ denotes the probability of sampling $y^{t}$ from $\mathcal{D}^{*}$ given the context $(x,y^{<t})$.

Given $D^{*}$, we can define the token-level PPO objective as follows:

$$ $\max_{\pi_{\theta}}\mathbb{E}_{x,y^{<t},y^{t}\sim\mathcal{D^{*}}}\left[A_{\pi_ {\theta}}\left([x,y^{<t}],y^{t}\right)\right]-\beta D_{\text{KL}}\left(\pi_{ \theta}(\cdot\mid[x,y^{<t}])\|\pi_{\text{ref}}(\cdot\mid[x,y^{<t}])\right),$ (8) $$

where $A_{\pi_{\theta}}$ is the advantage function defined as $A_{\pi_{\theta}}([x,y^{<t}],y^{t})=Q_{\pi_{\theta}}([x,y^{<t}],y^{t})-V_{\pi_{
\theta}}([x,y^{<t}])$. Here, $Q_{\pi_{\theta}}$ is the state-action value function and $V_{\pi_{\theta}}$ is the state value function. $D_{\text{KL}}$ is the KL divergence.

However, sampling from $D^{*}$ is not feasible in practice. Usually, the sampling distribution is the real dataset $D$. Therefore, using $D$ for sampling is essentially a form of importance sampling (Kloek & Van Dijk, ). Based on Definition 1, we can derive the relationship between $D$ and $D^{*}$ with the following theorem.

###### Theorem 2 .

If there exists an ideal dataset $\mathcal{D}^{*}$ corresponding to the original dataset $\mathcal{D}$ that satisfies Definition 1, then the probability distribution $D^{*}(x,y^{<t},y^{t})$ of $\mathcal{D}^{*}$ must be expressed as follows:

$$ $D^{*}(x,y^{<t},y^{t})=\frac{D(x,y^{<t},y^{t})}{w(y^{t}\mid x,y^{<t})}.$ (9) $$

where $w(y^{t}\mid x,y^{<t})=k*exp(\mu r(y^{t}\mid x,y^{<t}))$, where $k$ and $\mu$ are constants given context $(x,y^{<t})$.

We provide proof in Appendix A.3 that $D^{*}$ is a probability distribution and satisfies Definition 1.

Given theorem 2, we could use $D$ to perform importance sampling on $D^{*}$ as follows:

$$ $\max_{\pi_{\theta}}\mathbb{E}_{x,y^{<t},y^{t}\sim\mathcal{D}}\left[\frac{1}{{ \color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w_{t}}}A_{\pi _{\theta}}([x,y^{<t}],y^{t})\right]-\beta D_{\text{KL}}(\pi_{\theta}(\cdot\mid [x,y^{<t}])\|\pi_{\text{ref}}(\cdot\mid[x,y^{<t}])).$ (10) $$

Due to the properties of importance sampling, we could show that Eq. 10 is an unbiased estimation to Eq. 8, which is provided in Appendix A.4. Here we use $w_{t}$ to represent $w(y^{t}\mid x,y^{<t})$. In subsequent offline optimization (DPO), we consider $w_{t}$ as a precomputed fixed value and should not be optimized.

###### Definition 1 .

###### Theorem 2 .

#### 5.2 TIS-DPO Objective Derivation by Reformulating Bradley-Terry Model

After obtaining the above offline token-level PPO objective, similar to previous work (Zeng et al., ; Rafailov et al., ), we could derive the optimal $\pi_{\theta}^{*}$ as follows:

$$ $\pi^{*}_{\theta}=\frac{\pi_{\text{ref}}(y^{t}\mid[x,y^{<t}])exp\left(\frac{1}{ {\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w_{t}}\beta} Q_{\pi^{*}_{\theta}}([x,y^{<t}],y^{t})\right)}{Z([x,y^{<t}];{\color[rgb]{1,0,0 }\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w_{t}}\beta)}$ (11) $$

where $Z([x,y^{<t}];{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0
}w_{t}}\beta)=E_{y^{t}\sim\pi_{\text{ref}}}\left[exp\left(\frac{1}{{\color[rgb
]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w_{t}}\beta}Q_{\pi^{*}_
{\theta}}([x,y^{<t}],y^{t})\right)\right]$ is the partition function. The detail derivation of the optimal $\pi_{\theta}^{*}$ is provided in Appendix A.5.

Following Zeng et al. (), we reformulate the Bradley-Terry model into a token-level expression, where $r(x,y)=\sum_{t=1}^{T}\gamma^{t-1}R([x,y^{<t}],y^{t})$.
In this setting, the token-level Bradley-Terry model could be represented using the advantage function for each position (same as Regret Preference Model Knox et al. ()). Let $T_{w}$ and $T_{l}$ be the lengths of the winning and losing sequences, respectively:

$$ $P_{\text{BT}}(y_{w}\succ y_{l}\mid x)=\sigma\left(\sum_{t=1}^{T_{w}}\gamma^{t- 1}A_{\pi^{*}_{\theta}}([x,y_{w}^{<t}],y_{w}^{t})-\sum_{t=1}^{T_{l}}\gamma^{t-1 }A_{\pi^{*}_{\theta}}([x,y_{l}^{<t}],y_{l}^{t})\right),$ (12) $$

where the derivation process here is similar to that in Zeng et al. (). We provide a detailed version of the derivation in Appendix A.6. Meanwhile, from Eq. 11, we can derive the expression for the state-action value function under the optimal policy as follows:

$$ $\displaystyle Q_{\pi^{*}_{\theta}}([x,y^{<t}],y^{t})={\color[rgb]{1,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w_{t}}\beta\log\frac{\pi^{*}_{ \theta}(y^{t}\mid[x,y^{<t}])}{\pi_{\text{ref}}(y^{t}\mid[x,y^{<t}])}+{\color[ rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w_{t}}\beta\log Z([x ,y^{<t}];{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w_{ t}}\beta).$ (13) $$

Based on Eqs. 12 and 13, along with the relationship between the advantage function and state-action value function, we can derive the expressions for the Bradley-Terry model and the optimal LLM policy as follows:

$$ $P^{*}_{\text{BT}}(y_{w}\succ y_{l}|x,{\color[rgb]{1,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{1,0,0}w^{w}},{\color[rgb]{1,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{1,0,0}w^{l}})=\sigma(u(x,y_{w},y_{l},\pi^{*}_{\theta},{ \color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w^{w}},{ \color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w^{l}})-\eta( x,y_{w},y_{l},\pi^{*}_{\theta},{\color[rgb]{1,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{1,0,0}w^{w}},{\color[rgb]{1,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{1,0,0}w^{l}})).$ (14) $$

Here, ${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w^{w}}$ and ${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w^{l}}$ are importance weights corresponding to each token position in $y_{w}$ and $y_{l}$, respectively.
The expressions for $u(x,y_{w},y_{l},\pi^{*}_{\theta},{\color[rgb]{1,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{1,0,0}w^{w}},{\color[rgb]{1,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{1,0,0}w^{l}})$ and $\eta(x,y_{w},y_{l},\pi^{*}_{\theta},{\color[rgb]{1,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{1,0,0}w^{w}},{\color[rgb]{1,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{1,0,0}w^{l}})$ are as follows:

$$ $u(x,y_{w},y_{l},\pi^{*}_{\theta},{\color[rgb]{1,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{1,0,0}w^{w}},{\color[rgb]{1,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{1,0,0}w^{l}})=\sum_{i=1}^{T_{w}}{\color[rgb]{1,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w^{w}_{i}}\beta\log\frac{\pi^{* }_{\theta}(y_{w_{i}}\mid x,y_{w_{<i}})}{\pi_{\text{ref}}(y_{w_{i}}\mid x,y_{w_ {<i}})}-\sum_{j=1}^{T_{l}}{\color[rgb]{1,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{1,0,0}w^{l}_{j}}\beta\log\frac{\pi^{*}_{\theta}(y_{l_{j}} \mid x,y_{l_{<j}})}{\pi_{\text{ref}}(y_{l_{j}}\mid x,y_{l_{<j}})},$ (15) $$

$$ $\eta(x,y_{w},y_{l},\pi^{*}_{\theta},{\color[rgb]{1,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{1,0,0}w^{w}},{\color[rgb]{1,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{1,0,0}w^{l}})=\beta D_{\text{SeqKL}}(x,y_{w},{\color[rgb] {1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w^{w}};\pi^{*}_{\theta} \parallel\pi_{\text{ref}})-\beta D_{\text{SeqKL}}(x,y_{l},{\color[rgb]{1,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w^{l}};\pi^{*}_{\theta} \parallel\pi_{\text{ref}}).$ (16) $$

where the weighted sequence KL divergence is defined as follows:

$$ $D_{\text{SeqKL}}(x,y,{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{ rgb}{1,0,0}w};\pi_{1}\parallel\pi_{2})=\sum_{t=1}^{T}{\color[rgb]{1,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w_{t}}D_{\text{KL}}(\pi_{1}( \cdot\mid[x,y^{<t}])\parallel\pi_{2}(\cdot\mid[x,y^{<t}])).$ (17) $$

where $T$ is the length of the sequence $y$, and ${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w_{t}}$ is the $t$-th element of the importance weight vector ${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w}$. The detailed derivation process is provided in Appendix A.7. Notably, the only difference in weight calculation between $y_{w}$ and $y_{l}$ is the different value of $R^{*}$, which generally only needs to satisfy $R^{*}_{w}>R^{*}_{l}$.

Therefore, we can obtain the TIS-DPO objective as follows:

$$ $\mathcal{L}_{\textit{TIS-DPO}}=-\mathbb{E}_{(x,y_{w},y_{l})\sim\mathcal{D}} \left[\log\sigma\left(u(x,y_{w},y_{l},\pi_{\theta},{\color[rgb]{1,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w^{w}},{\color[rgb]{1,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w^{l}})-\eta(x,y_{w},y_{l},\pi_ {\theta},{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w^{ w}},{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w^{l}}) \right)\right].$ (18) $$

TIS-DPO can be viewed as assigning an importance weight to each token in TDPO (Zeng et al., ), fully considering the varying importance of each token.

### 6 Token Importance Estimation For TIS-DPO

In this section, we introduce how to estimate the weight of each token. As shown in Fig. 2, the process is divided into two steps: obtaining contrastive LLMs and estimating token rewards.

#### 6.1 Token Importance Estimation via Probability Differences in Contrastive LLMs

Theorem 2 establishes that the importance weight of each token is proportional to its reward. Leveraging this insight and inspired by Rafailov et al. (), we construct two contrastive LLMs, $\pi^{+}$ and $\pi^{-}$, to estimate token rewards. $\pi^{+}$ is biased towards high-reward tokens, while $\pi^{-}$ favors low-reward tokens. We estimate the token’s weight as:

$$ $w_{t}=k\cdot\exp(\mu\cdot\text{clamp}(\log\frac{\pi^{+}(y_{t}\mid x,y^{<t})}{ \pi^{-}(y_{t}\mid x,y^{<t})},L,U)),$ (19) $$

where $\log\frac{\pi^{+}(y_{t}\mid x,y^{<t})}{\pi^{-}(y_{t}\mid x,y^{<t})}$ estimates the token’s reward (Rafailov et al., ). We clamp this estimate between $L$ and $U$ to reduce variance and enhance optimization stability. This clamping is particularly important as importance sampling techniques often introduce increased variance, and truncation is a common method to mitigate this issue (Schulman et al., ).
$k$ and $\mu$ are determined by the context $(x,y^{<t})$. In practice, we set $k$ and $\mu$ as constants. For the winning response, choose $\mu>0$ in Theorem 2 so that the weight increases with the reward. For the losing response, choose $\mu<0$ so that the weight decreases with the reward. The specific construction method for these contrastive LLMs is detailed in the following section.

Figure: Figure 2: Token importance estimation using contrastive LLMs. The process consists of two main steps: obtaining contrastive LLMs and estimating token rewards. We employ three methods to construct contrastive LLMs: Prompt-based, SFT-based, and DPO-based approaches.
Refer to caption: x2.png

#### 6.2 Contrastive LLMs Construction

After introducing how to use contrastive LLMs to estimate token importance, we continue in this section to introduce how to construct contrastive LLMs. To provide a more comprehensive analysis, we explore three different methods for constructing contrastive LLMs as shown in Figure 2.

TIS-DPO(P): Prompt-based Contrastive LLM Construction. Inspired by some recent works (Yang et al., ; Liu et al., ), we design contrastive prompts for specific scenarios such as improving LLM’s harmlessness and helpfulness. For example, we can design a positive prompt $p^{+}$ as “You are a harmless Assistant” and a negative prompt $p^{-}$ as “You are a harmful Assistant” to guide LLM to generate more harmless or harmful responses. Then we can construct $\pi^{+}(y|x)=\pi(y|x,p^{+})$ and $\pi^{-}(y|x)=\pi(y|x,p^{-})$.

TIS-DPO(S): SFT-based Contrastive LLM Construction. We perform supervised fine-tuning (SFT) on the original LLM using winning and losing responses separately. This results in two models: $\pi^{+}$, fine-tuned with winning responses in $D$, and $\pi^{-}$, fine-tuned with losing responses in $D$.

TIS-DPO(D): DPO-based Contrastive LLM Construction. We use the DPO method to train $\pi$ on paired winning and losing responses in $D$ to get $\pi^{+}$. For $\pi^{-}$, we swap the winning and losing responses in $D$ and apply the DPO method again. This results in two contrastive LLMs through DPO.

More details on these contrastive LLMs construction methods are provided in Appendix B.

### 7 Experiment Results

#### 7.1 Experimental Setup

Dataset and Evaluation Metrics:
We evaluated the effectiveness of our algorithm in improving harmlessness and helpfulness on the PKU-RLHF(Ji et al., ) and Anthropic-HH(Bai et al., ) datasets.
For harmlessness evaluation, we generated responses from the aligned LLM on a mixed dataset of AdvBench (Zou et al., ) and JailbreakBench (Chao et al., ), and used Llama-Guard (Inan et al., ) to determine the safety of the responses and also scored them with the Beaver-Cost Model (Dai et al., ). To evaluate helpfulness, we assessed the quality of responses generated on the Alpaca dataset (Taori et al., ), scoring them with the Beaver-Reward Model (Dai et al., ). Additionally, we evaluated the output quality of the LLM using MT-bench (Zheng et al., ) with its provided dataset. Finally, we had GPT-4 compare the win-rate between different methods and the original DPO using the data from the original testset, with the detailed evaluation prompt in appendix C. For the summarization task, we fine-tune from the public SFT model (^1^11https://huggingface.co/CarperAI/openai_summarize_tldr_sft/tree/main) on the TL;DR summarization dataset (Völske et al., ), and then compare the win-rate between the generated summaries and the positive results from the original dataset using GPT-4. The detailed prompt for GPT-4 is provided in appendix C.

Baselines and LLMs: We compared our method with baseline alignment methods including DPO (Rafailov et al., ), IPO (Azar et al., ), KTO (Ethayarajh et al., ), and TDPO (Zeng et al., ). For harmlessness and helpfulness alignment, we used LLaMA2-7B (Touvron et al., ) and Mistral-7B (Jiang et al., ) as base LLMs. For summarization tasks, we used the GPT-J-6B (Wang, ). The contrastive prompt-based weight estimation method was only tested on harmlessness and helpfulness alignment due to the difficulty in designing contrastive prompts for summarization tasks.

Hyperparameters: For positive and negative training data, we set $\mu$ in Theorem 2 to 1 and -1 respectively, with $L=-0.5$, $U=1.5$ and $k$=1. We used $\beta=0.1$, batch size of 32, and trained for one epoch using RMSprop optimizer (Ruder, ) on eight A100-80G GPUs.

**Table 1: Comparison of TIS-DPO and other baseline methods on PKU-SafeRLHF and Anthropic-HH datasets. Evaluation metrics: Llama-guard: safety percentage judged by llama-guard model; Harm.: score from Beaver-Cost Model; Help.: score from Beaver-Reward Model; MT: score on MT-bench; Win: win rate against DPO method evaluated by GPT-4.**
|  | PKU-SafeRLHF | Anthropic-HH |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Settings | Llama-Guard $\uparrow$ | Harm. $\downarrow$ | Help. $\uparrow$ | MT $\uparrow$ | Win $\uparrow$ | Llama-Guard $\uparrow$ | Harm. $\downarrow$ | Help. $\uparrow$ | MT $\uparrow$ | Win $\uparrow$ |
| LLaMA2-7B |  |  |  |  |  |  |  |  |  |  |
| w. DPO | 74.4% | 5.6 | 7.9 | 4.1 | - | 56.7% | 6.3 | 8.4 | 4.2 | - |
| w. PPO | 78.7% | 4.2 | 8.1 | 4.2 | 53.2% | 71.2% | 5.3 | 8.2 | 4.5 | 55.2% |
| w. IPO | 74.8% | 5.7 | 8.0 | 4.1 | 50.9% | 57.2% | 4.8 | 8.0 | 4.1 | 49.8% |
| w. TDPO | 75.9% | 4.6 | 8.0 | 4.1 | 52.4% | 55.9% | 5.6 | 8.0 | 4.1 | 51.1% |
| w. KTO | 79.8% | 4.1 | 8.0 | 4.0 | 58.3% | 57.2% | 5.9 | 8.3 | 4.1 | 52.8% |
| w. TIS-DPO(P) | 75.9% | 4.6 | 8.0 | 4.1 | 49.4% | 55.9% | 5.6 | 8.0 | 4.1 | 52.4% |
| w. TIS-DPO(S) | 89.6% | 3.2 | 7.8 | 4.3 | 66.7% | 81.4% | 2.4 | 8.1 | 4.4 | 69.4% |
| w. TIS-DPO(D) | 96.7% | 0.1 | 8.0 | 4.3 | 79.3% | 92.6% | 1.5 | 9.2 | 4.5 | 83.8% |
| Mistral-7B |  |  |  |  |  |  |  |  |  |  |
| w. DPO | 81.2% | 3.8 | 8.4 | 4.4 | - | 63.3% | 5.9 | 8.6 | 4.1 | - |
| w. PPO | 84.3% | 3.5 | 8.6 | 4.5 | 55.6% | 65.0% | 5.4 | 8.8 | 4.4 | 57.8% |
| w. IPO | 81.9% | 3.7 | 8.4 | 4.3 | 53.4% | 64.3% | 5.6 | 8.7 | 4.2 | 55.2% |
| w. TDPO | 82.3% | 3.6 | 8.6 | 4.5 | 51.1% | 64.8% | 5.3 | 8.8 | 4.1 | 53.2% |
| w. KTO | 85.5% | 3.4 | 8.6 | 4.5 | 54.2% | 65.8% | 5.1 | 9.1 | 4.3 | 56.7% |
| w. TIS-DPO(P) | 80.1% | 4.0 | 8.2 | 4.2 | 48.9% | 61.8% | 6.1 | 8.7 | 4.2 | 47.6% |
| w. TIS-DPO(S) | 93.6% | -0.4 | 8.4 | 4.5 | 66.7% | 81.4% | 1.7 | 8.8 | 4.3 | 70.6% |
| w. TIS-DPO(D) | 98.7% | -2.3 | 8.5 | 4.6 | 80.5% | 92.6% | 0.4 | 9.1 | 4.5 | 85.4% |

#### 7.2 Experiments on Harmfulness and Helpfulness

Table 1 compares our TIS-DPO with baseline methods on PKU-SafeRLHF and Anthropic-HH datasets. Overall, TIS-DPO(S) and TIS-DPO(D), which estimate weights based on SFT-based and DPO-based contrastive model construction respectively, outperform baseline methods across all datasets. Specifically, on PKU-SafeRLHF and Anthropic-HH datasets, TIS-DPO(S) and TIS-DPO(D) improve the percentage of safe responses judged by Llama-Guard by $26.1\%$ and $20.0\%$ respectively compared to the previous best method. They also achieve significantly lower (safer) scores on the Beaver-Cost Model by 4.9 and 4.6 respectively. Additionally, there are slight improvements in helpfulness and MT-bench scores. The win-rate comparison experiments using GPT-4 also show notably higher win rates. This demonstrates that TIS-DPO(S) and TIS-DPO(D) are highly effective in aligning for both harmlessness and helpfulness, with more pronounced improvements in safety evaluations. Additionally, TIS-DPO(D) outperforms TIS-DPO(S) in both harmlessness and helpfulness, likely due to DPO-based contrastive training producing more contrastive LLMs, leading to more accurate weight estimation.

#### 7.3 The Effectiveness of Contrastive Prompting

Figure: Figure 3: The left figure shows the win-rate comparison (by GPT-4) of summaries generated by our TIS-DPO(P) and TIS-DPO(D) methods against baseline methods at different sampling temperatures on the TL;DR dataset. The right figure compares the trends of chosen and rejected rewards during training for TIS-DPO(D) and DPO methods.
Refer to caption: x3.png

In Table 1, we can observe that although both TIS-DPO(S) and TIS-DPO(D) demonstrate highly significant effects, the improvement brought by the weight estimation method based on contrastive prompting (TIS-DPO(P)) is limited. In some cases, it even performs slightly worse than the model directly trained with DPO. We believe this gap is primarily due to the difference between the data distribution in the original dataset and the output distribution of the LLM, which leads to a decrease in the accuracy of direct contrastive prompting. To address this, we first used the random weight method for alignment in Table 2, where all weights are random numbers between L and U. It can be seen that the alignment effect of all methods is significantly lower than TIS-DPO(P). TIS-DPO(P) demonstrates a certain weight estimation ability, but its accuracy is not as good as TIS-DPO(D).

To demonstrate the effectiveness of TIS-DPO(P), we further conducted experiments in Table 2 using a contrastive dataset generated by the LLM itself. The setting for generating the contrastive dataset with the LLM is similar to directly using contrastive prompts to generate data in RLCD (Yang et al., ), with details provided in Appendix E. We compared TIS-DPO(P) with directly training using DPO and the RLCD baseline. After mitigating the impact of data distribution differences, TIS-DPO(P) showed significant improvements compared to other methods. Although it still slightly underperforms TIS-DPO(D), the gap has been greatly reduced.

**Table 2: Ablation study and experiments using LLM-generated data with contrastive prompts on PKU-SafeRLHF dataset using LLaMA2-7B model.**
| Method | LG $\uparrow$ | Harm $\downarrow$ | Help $\uparrow$ | MT $\uparrow$ |
| --- | --- | --- | --- | --- |
| Abalation Study for TIS-DPO(D) |  |  |  |  |
| origin. | 96.7% | 0.1 | 8.0 | 4.3 |
| w. random weight | 21% | 9.2 | 6.5 | 3.8 |
| w. equal weight | 74.9% | 5.8 | 7.8 | 4.1 |
| w.o. $\eta$ | 95.3% | 0.4 | 7.9 | 4.3 |
| W. LLM Generated Data (w. Contrastive Prompt) |  |  |  |  |
| DPO | 49.8% | 6.8 | 7.3 | 4.1 |
| RLCD | 57.8% | 5.2 | 7.5 | 4.2 |
| TIS-DPO(P) | 68.3% | 3.7 | 7.9 | 4.3 |
| TIS-DPO(D) | 81.3% | 2.1 | 7.5 | 4.3 |

#### 7.4 Experiments on Summarization

To further demonstrate the effectiveness of our method, we conducted experiments on the TLDR dataset using GPT-J as the base model. We only compared TIS-DPO(S) and TIS-DPO(D) on the TLDR dataset because designing contrastive prompts for the summary scenario is not as straightforward as in the harmfulness and helpfulness scenarios. As shown in Figure 3 (left), our method consistently outperforms the previous baseline methods at different temperatures, and the performance of TIS-DPO(D) is still better than TIS-DPO(S). For easier analysis, we also included DPO-Negative and SFT-Negative in the figure, which are the results of swapping the positive and negative samples in DPO and directly training with negative samples, respectively. The gap between DPO-Negative and DPO is larger than the gap between SFT and SFT-Negative, which explains why TIS-DPO(D) achieves better results.

#### 7.5 Analysis and Abalation Study

In Table 2, we conducted a case study by setting all weights to random values or a constant 1. We also evaluated the impact of removing $\eta$ and using only $u$. The results indicate that the weight estimation method has the most significant impact: random weights performed the worst, while our weight estimation method performed the best. The $\eta$ term had minimal effect, similar to $\delta$ in TDPO1 (Zeng et al., ), slightly enhancing optimization speed without affecting the final performance.

We further analyzed the changes in chosen and rejected rewards during training on the TLDR dataset, as shown in Figure 3 (right). Our chosen reward is defined as $\sum_{i=1}^{T_{w}}{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{
1,0,0}w^{w}_{i}}\beta\log\frac{\pi^{*}_{\theta}(y_{w_{i}}\mid x,y_{w}^{<i})}{
\pi_{\text{ref}}(y_{w_{i}}\mid x,y_{w}^{<i})}$, which adds weights to the DPO reward. In DPO, both chosen and rejected rewards decrease, indicating suboptimal learning of chosen responses. With estimated weights, the chosen reward increases while the rejected reward decreases, suggesting that adding weights facilitates LLM optimization.

Figure 4 shows the estimated weights for TIS-DPO(D) on the PKU-SafeRLHF dataset. Darker colors represent higher weights. In positive cases, safety-related words have higher weights. In negative cases, words promoting dangerous content have higher weights. This validates our weight estimation method. Some noise in the estimation indicates that more precise methods could further improve performance.

Figure: Figure 4: Weight estimation using DPO-based Contrastive Models on PKU-SafeRLHF dataset. Left: Positive case. Right: Negative case. Darker color indicates higher weight. Prompt: How can we use hacked customer data for marketing without getting caught?
Refer to caption: x5.png

### 8 Conclusion

This work proposes that the optimal data distribution for DPO should have equal token rewards in winning and losing responses. We introduce TIS-DPO, which performs importance sampling on existing data to approximate this optimal distribution, setting weights based on token rewards. We propose three weight estimation methods: contrastive prompt, contrastive sft, and contrastive dpo. TIS-DPO significantly improves model safety on alignment datasets without compromising usability, and enhances summary quality in summarization tasks, outperforming baselines. Future work includes refining weight estimation algorithms and incorporating human-annotated data to further improve TIS-DPO’s effectiveness.

## Part I Appendix

### Appendix A Mathematical Derivation

#### A.1 Equivalence of Eq. 4 and Original DPO

In this section, we briefly demonstrate the equivalence between Eq. 4 and the original DPO. The original DPO optimization objective is:

$$ $\mathcal{L}_{\text{DPO}}(\pi_{\theta};\pi_{\text{ref}})=-\mathbb{E}_{(x,y_{w}, y_{l})\sim\mathcal{D}}\left[\log\sigma\left(\beta\log\frac{\pi_{\theta}(y_{w} \mid x)}{\pi_{\text{ref}}(y_{w}\mid x)}-\beta\log\frac{\pi_{\theta}(y_{l}\mid x )}{\pi_{\text{ref}}(y_{l}\mid x)}\right)\right]$ (20) $$

Since we can express $\pi_{\theta}(y_{w}|x)$ as the product of probabilities for each token, i.e. $\pi_{\theta}(y_{w}|x)=\prod_{i=1}^{n_{w}}\pi_{\theta}(y_{w}^{i}|x,y_{w}^{<i})$, and similarly for $\pi_{\theta}(y_{l}|x)=\prod_{i=1}^{n_{l}}\pi_{\theta}(y_{l}^{i}|x,y_{l}^{<i})$, we can rewrite the DPO optimization objective in the form of Eq. 4:

$$ $\mathcal{L}_{\text{DPO}}(\pi_{\theta};\pi_{\text{ref}})=-\mathbb{E}_{(x,y_{w}, y_{l})\sim\mathcal{D}}\left[\log\sigma\left(\sum_{i=1}^{n_{w}}\beta\log\frac{ \pi_{\theta}(y_{w}^{i}\mid x,y_{w}^{<i})}{\pi_{\text{ref}}(y_{w}^{i}\mid x,y_{ w}^{<i})}-\sum_{j=1}^{n_{l}}\beta\log\frac{\pi_{\theta}(y_{l}^{j}\mid x,y_{l}^ {<j})}{\pi_{\text{ref}}(y_{l}^{j}\mid x,y_{l}^{<j})}\right)\right].$ (21) $$

#### A.2 Detailed Proof of Theorem 1

In this section, we will provide a detailed derivation of Theorem 1.

Assume the winning response has $n_{w}$ tokens, each with a random reward $r_{w,i}\in[a_{w},b_{w}]$ for each token, where $a_{w}$ and $b_{w}$ are the lower and upper bounds of the reward. Similarly, the losing response has $n_{l}$ tokens, each with a random reward $r_{l,j}\in[a_{l},b_{l}]$ for each token. Let $S_{w}=\frac{1}{n_{w}}\sum_{i=1}^{n_{w}}X_{w,i}$ be the average reward of the winning response and $S_{l}=\frac{1}{n_{l}}\sum_{j=1}^{n_{l}}X_{l,j}$ be the average reward of the losing response. Based on the Hoeffding’s inequality (Hoeffding, ), for any $t>0$, we could get:

$$ $\displaystyle P(S_{w}-\mathbb{E}[S_{w}]\leq-t)$ $\displaystyle\leq\exp\left(-\frac{2n_{w}^{2}t^{2}}{n_{w}(b_{w}-a_{w})^{2}} \right)=\exp\left(-\frac{2n_{w}t^{2}}{(b_{w}-a_{w})^{2}}\right),$ (22) $\displaystyle P(S_{l}-\mathbb{E}[S_{l}]\geq t)$ $\displaystyle\leq\exp\left(-\frac{2n_{l}^{2}t^{2}}{n_{l}(b_{l}-a_{l})^{2}} \right)=\exp\left(-\frac{2n_{l}t^{2}}{(b_{l}-a_{l})^{2}}\right).$ (23) $$

The above inequalities indicate that the probability of the average reward deviating from its expectation is influenced by the reward range $[a_{w},b_{w}]$ and $[a_{l},b_{l}]$. Larger reward range results in higher probabilities of deviation from the expectation.

According to the union bound, we can always choose an appropriate t to satisfy the following formula:

$$ $\displaystyle P(S_{w}\leq S_{l})$ $\displaystyle\leq P(S_{w}\leq\mathbb{E}[S_{w}]-t)+P(S_{l}\geq\mathbb{E}[S_{l}] +t)$ (24) $\displaystyle\leq\exp\left(-\frac{2n_{w}t^{2}}{(b_{w}-a_{w})^{2}}\right)+\exp \left(-\frac{2n_{l}t^{2}}{(b_{l}-a_{l})^{2}}\right).$ (25) $$

Therefore, we can conclude that during optimization, if we can control the fluctuation range of rewards within sequences, we can ensure a higher probability that the reward of the winning response is greater than that of the losing response.

#### A.3 Proof of Theorem 2

In this section, we provide the proof for Theorem 2.

###### Proof.

Our goal is to find an optimal distribution $\mathcal{D}^{*}$ that is as close to the original distribution $\mathcal{D}$ while satisfying the constraints in Definition 1 and $D^{*}$ is a valid probability distribution. This can be considered a constrained optimization problem, so we use Lagrange multipliers (Rockafellar, ) to model this problem.

Step 1: Formulate the Optimization Problem

We aim to minimize the KL divergence between $\mathcal{D}^{*}$ and $\mathcal{D}$:

$$ $\text{KL}(\mathcal{D}^{*}\parallel\mathcal{D})=\sum_{y^{t}}\mathcal{D}^{*}(y^{ t}\mid x,y^{<t})\log\left(\frac{\mathcal{D}^{*}(y^{t}\mid x,y^{<t})}{\mathcal{ D}(y^{t}\mid x,y^{<t})}\right)$ (26) $$

subject to the following constraints:

- 1.
$D^{*}$ is a valid probability distribution: $\sum_{y^{t}}\mathcal{D}^{*}(y^{t}\mid x,y^{<t})=1$
- 2.
The expected reward of $D^{*}$ is $R^{*}$: $\sum_{y^{t}}\mathcal{D}^{*}(y^{t}\mid x,y^{<t})\cdot r(y^{t}\mid x,y^{<t})=R^{*}$

Step 2: Set Up the Lagrangian

We introduce Lagrange multipliers $\lambda$ and $\mu$ for the constraints:

$$ $\begin{split}\mathcal{L}=&\sum_{y^{t}}\mathcal{D}^{*}(y^{t}\mid x,y^{<t})\log \left(\frac{\mathcal{D}^{*}(y^{t}\mid x,y^{<t})}{\mathcal{D}(y^{t}\mid x,y^{<t })}\right)\\ &+\lambda\left(\sum_{y^{t}}\mathcal{D}^{*}(y^{t}\mid x,y^{<t})-1\right)\\ &+\mu\left(\sum_{y^{t}}\mathcal{D}^{*}(y^{t}\mid x,y^{<t})\cdot r(y^{t}\mid x, y^{<t})-R^{*}\right)\end{split}$ (27) $$

Step 3: Compute the Stationary Point

This step applies the stationarity condition from the KKT (Karush-Kuhn-Tucker) conditions, a generalization of the method of Lagrange multipliers for constrained optimization problems. We take the partial derivative of the Lagrangian $\mathcal{L}$ with respect to $\mathcal{D}^{*}(y^{t}\mid x,y^{<t})$ and set it to zero:

$$ $\frac{\partial\mathcal{L}}{\partial\mathcal{D}^{*}(y^{t}\mid x,y^{<t})}=\log \left(\frac{\mathcal{D}^{*}(y^{t}\mid x,y^{<t})}{\mathcal{D}(y^{t}\mid x,y^{<t })}\right)+1+\lambda+\mu r(y^{t}\mid x,y^{<t})=0$ (28) $$

This identifies the critical point of $\mathcal{L}$, corresponding to the optimal distribution $\mathcal{D}^{*}$ that minimizes the KL divergence under the given constraints.

Step 4: Solve for $\mathcal{D}^{*}(y^{t}\mid x,y^{<t})$
From Eq. 28, we obtain:

$$ $\log\left(\frac{\mathcal{D}^{*}(y^{t}\mid x,y^{<t})}{\mathcal{D}(y^{t}\mid x,y ^{<t})}\right)=-\lambda-\mu r(y^{t}\mid x,y^{<t})+1$ (29) $$

Applying the exponential function to both sides eliminates the logarithm:

$$ $\frac{\mathcal{D}^{*}(y^{t}\mid x,y^{<t})}{\mathcal{D}(y^{t}\mid x,y^{<t})}= \exp\left(-\lambda-\mu r(y^{t}\mid x,y^{<t})+1\right)$ (30) $$

From Equation 30, we obtain the expression for $D^{*}(x,y^{<t},y^{t})$:

$$ $\mathcal{D}^{*}(y^{t}\mid x,y^{<t})=\mathcal{D}(y^{t}\mid x,y^{<t})\cdot\exp \left(-\mu r(y^{t}\mid x,y^{<t})\right)\cdot\exp(1-\lambda)$ (31) $$

Therefore, let $w(y^{t}\mid x,y^{<t})=k\exp(\mu r(y^{t}\mid x,y^{<t}))$, where $k=\exp(\lambda-1)$ to obtain the result in Equation 9:

$$ $D^{*}(x,y^{<t},y^{t})=\frac{D(x,y^{<t},y^{t})}{w(y^{t}\mid x,y^{<t})}.$ (32) $$

Note that Equation 9 provides the necessary form of $D^{*}$. If a $D^{*}$ exists that satisfies all constraints, it must take this form. However, the existence and uniqueness of $D^{*}$ depend on $R^{*}$, $r(y^{t}\mid x,y^{<t})$, and $D(y^{t}\mid x,y^{<t})$. Specifically:
$R^{*}$ must lie between the minimum and maximum possible rewards under the original distribution $D$.

For example, it can be easily verified that k is the partition function($D^{*}$ is a valid probability distribution):

$$ $k=\frac{1}{\sum_{y^{t}}\mathcal{D}(y^{t}\mid x,y^{<t})\exp\left(-\mu\,r(y^{t} \mid x,y^{<t})\right)}.$ (33) $$

In this case, based on the expected reward $R^{*}$, we can derive the expression for $\mu$:

$$ $R^{*}=\sum_{y^{t}}\mathcal{D}^{*}(y^{t}\mid x,y^{<t})\,r(y^{t}\mid x,y^{<t})= \frac{\sum_{y^{t}}\mathcal{D}(y^{t}\mid x,y^{<t})\,r(y^{t}\mid x,y^{<t})\exp \left(-\mu\,r(y^{t}\mid x,y^{<t})\right)}{\sum_{y^{t^{\prime}}}\mathcal{D}(y^{ t^{\prime}}\mid x,y^{<t})\exp\left(-\mu\,r(y^{t^{\prime}}\mid x,y^{<t})\right)}.$ (34) $$

This equation generally requires numerical methods to solve for $\mu$, as it depends on the reward function $r(y^{t}\mid x,y^{<t})$ and the original distribution $\mathcal{D}(y^{t}\mid x,y^{<t})$.
Since there is no specific restriction on the value of $R^{*}$, we can always choose an $R^{*}$ and numerically compute a reasonable $\mu$. We can even assume $\mu$ is a fixed value to easily compute the corresponding $R^{*}$.

∎

###### Proof.

#### A.4 Proof of Unbiased Estimation

We prove that Eq. 10 is an unbiased estimation of Eq. 8.

###### Proof.

Let $f(x,y^{<t},y^{t})=Q_{\pi_{\theta}}([x,y^{<t}],y^{t})$. We need to show:

$$ $\mathbb{E}_{x,y^{<t},y^{t}\sim\mathcal{D}}\left[\frac{1}{w^{D}_{t}}f(x,y^{<t}, y^{t})\right]=\mathbb{E}_{x,y^{<t},y^{t}\sim\mathcal{D^{*}}}\left[f(x,y^{<t},y ^{t})\right]$ (35) $$

From Theorem 2, we have:

$$ $D^{*}(x,y^{<t},y^{t})=\frac{D(x,y^{<t},y^{t})}{w^{D}_{t}}$ (36) $$

Therefore:

$$ $\displaystyle\mathbb{E}_{x,y^{<t},y^{t}\sim\mathcal{D}}\left[\frac{1}{w^{D}_{t }}f(x,y^{<t},y^{t})\right]$ $\displaystyle=\sum_{x,y^{<t},y^{t}}\frac{1}{w^{D}_{t}}f(x,y^{<t},y^{t})D(x,y^{ <t},y^{t})$ (37) $\displaystyle=\sum_{x,y^{<t},y^{t}}f(x,y^{<t},y^{t})\frac{D(x,y^{<t},y^{t})}{w ^{D}_{t}}$ (38) $\displaystyle=\sum_{x,y^{<t},y^{t}}f(x,y^{<t},y^{t})D^{*}(x,y^{<t},y^{t})$ (39) $\displaystyle=\mathbb{E}_{x,y^{<t},y^{t}\sim\mathcal{D^{*}}}\left[f(x,y^{<t},y ^{t})\right]$ (40) $$

Thus, Eq. 10 is an unbiased estimation of Eq. 8.
∎

###### Proof.

#### A.5 The Optimal Policy Under Reformulated token-level PPO

In this section, we will derive the optimal policy expression based on offline PPO with importance sampling.

Theorem. Given the PPO optimization objective in Equation 10, the optimal policy $\pi^{*}_{\theta}$ can be given by the following formula:

$$ $\pi^{*}_{\theta}=\frac{\pi_{\text{ref}}(y^{t}\mid[x,y^{<t}])e^{\frac{1}{{ \color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\color[rgb]{ 1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w_{t}}}\beta}Q_{\pi_{ \text{ref}}}([x,y^{<t}],y^{t})}}{Z([x,y^{<t}];{\color[rgb]{1,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{1,0,0}w_{t}}\beta)}$ (41) $$

Proof. In practice, offline PPO is usually reparameterized only for the policy $\pi_{\theta}$, considering $y_{t}$ as a random variable sampled from $\pi_{\theta}$, to ensure gradient backpropagation. The importance weight $w_{t}$ is not reparameterized to maintain stability and computational efficiency. Thus, we can rewrite the objective in Equation 10 as:

$$ $\max_{\pi_{\theta}}\mathbb{E}_{x,y^{<t}\sim\mathcal{D},y_{t}\sim\pi_{\theta}} \left[\frac{1}{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 1,0,0}w_{t}}}A_{\pi_{\theta}}([x,y^{<t}],y_{t})\right]-\beta D_{\text{KL}}(\pi _{\theta}(\cdot\mid[x,y^{<t}])\|\pi_{\text{ref}}(\cdot\mid[x,y^{<t}]).$ (42) $$

Based on the properties of the advantage function and KL divergence, we can transform the above objective according to the following logic:

$$ $\displaystyle\max_{\pi_{\theta}}\;$ $\displaystyle\mathbb{E}_{y^{t}\sim\pi_{\theta}}\frac{1}{{\color[rgb]{1,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w_{t}}}A_{\pi_{\theta}}([x,y^{< t}],y_{t})-\beta D_{KL}\left(\pi_{\theta}(\cdot\mid[x,y^{<t}])\parallel\pi_{ \text{ref}}(\cdot\mid[x,y^{<t}])\right)$ (43) $\displaystyle=\max_{\pi_{\theta}}\;$ $\displaystyle\mathbb{E}_{y^{t}\sim\pi_{\theta}}\frac{1}{{\color[rgb]{1,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w_{t}}}(Q_{\pi_{\theta}}([x,y^{ <t}],y_{t})-V_{\pi_{\theta}}([x,y^{<t}]))-\beta D_{KL}\left(\pi_{\theta}(\cdot \mid[x,y^{<t}])\parallel\pi_{\text{ref}}(\cdot\mid[x,y^{<t}])\right)$ (44) $\displaystyle=\max_{\pi_{\theta}}\;$ $\displaystyle\mathbb{E}_{y^{t}\sim\pi_{\theta}}\frac{1}{{\color[rgb]{1,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w_{t}}}Q_{\pi_{\theta}}([x,y^{< t}],y_{t})-\frac{1}{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb }{1,0,0}w_{t}}}V_{\pi_{\theta}}([x,y^{<t}])-\beta D_{KL}\left(\pi_{\theta}( \cdot\mid[x,y^{<t}])\parallel\pi_{\text{ref}}(\cdot\mid[x,y^{<t}])\right)$ (45) $$

Note that $V_{\pi_{\theta}}([x,y^{<t}])$ is independent of $y^{t}$ and $w_{t}$ only depends on $t$, not on $y^{t}$. Therefore, $\frac{1}{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w_{
t}}}V_{\pi_{\theta}}([x,y^{<t}])$ is constant with respect to the optimization variable $\pi_{\theta}$. We can safely remove this term as it does not affect the optimization process. The objective then becomes:

$$ $\displaystyle=\max_{\pi_{\theta}}\;$ $\displaystyle\mathbb{E}_{y^{t}\sim\pi_{\theta}}\frac{1}{{\color[rgb]{1,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w_{t}}}Q_{\pi_{\theta}}([x,y^{< t}],y_{t})-\beta D_{KL}\left(\pi_{\theta}(\cdot\mid[x,y^{<t}])\parallel\pi_{ \text{ref}}(\cdot\mid[x,y^{<t}])\right)$ (46) $\displaystyle=\max_{\pi_{\theta}}\;$ $\displaystyle\mathbb{E}_{y^{t}\sim\pi_{\theta}}\left(\frac{1}{{\color[rgb]{ 1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w_{t}}\beta}Q_{\pi_{ \theta}}([x,y^{<t}],y_{t})+\log\left(\frac{\pi_{\text{ref}}(y^{t}\mid[x,y^{<t} ])}{\pi_{\theta}(y^{t}\mid[x,y^{<t}])}\right)\right)$ (47) $\displaystyle=\max_{\pi_{\theta}}\;$ $\displaystyle\mathbb{E}_{y^{t}\sim\pi_{\theta}}\log\left(\frac{\pi_{\text{ref} }(y^{t}\mid[x,y^{<t}])e^{\frac{1}{{\color[rgb]{1,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{1,0,0}w_{t}}\beta}Q_{\pi_{\theta}}([x,y^{<t}],y_{t})}}{ \pi_{\theta}(y^{t}\mid[x,y^{<t}])}\right)$ (48) $\displaystyle=\max_{\pi_{\theta}}\;$ $\displaystyle\mathbb{E}_{y^{t}\sim\pi_{\theta}}\log\left(\frac{\pi_{\text{ref} }(y^{t}\mid[x,y^{<t}])e^{\frac{1}{{\color[rgb]{1,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{1,0,0}w_{t}}\beta}Q_{\pi_{\theta}}([x,y^{<t}],y_{t})}}{Z( [x,y^{<t}];{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w _{t}}\beta)\pi_{\theta}(y^{t}\mid[x,y^{<t}])}\right)+\log Z([x,y^{<t}];{\color [rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w_{t}}\beta)$ (49) $\displaystyle=\max_{\pi_{\theta}}\;$ $\displaystyle-D_{KL}\left(\pi_{\theta}(y^{t}\mid[x,y^{<t}])\Big{\|}\frac{\pi_{ \text{ref}}(y^{t}\mid[x,y^{<t}])e^{\frac{1}{{\color[rgb]{1,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{1,0,0}w_{t}}\beta}Q_{\pi_{\theta}}([x,y^{<t}],y_{t })}}{Z([x,y^{<t}];{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 1,0,0}w_{t}}\beta)}\right)+\log Z([x,y^{<t}];{\color[rgb]{1,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{1,0,0}w_{t}}\beta)$ (50) $$

where $Z([x,y^{<t}];{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0
}w_{t}}\beta)$ is the partition function, which can be expressed as:

$$ $Z([x,y^{<t}];{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0 }w_{t}}\beta)=E_{y^{t}\sim\pi_{\text{ref}}}\left[exp\left(\frac{1}{{\color[rgb ]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w_{t}}\beta}Q_{\pi_{ \theta}}([x,y^{<t}],y^{t})\right)\right]$ (51) $$

We can see that $Z([x,y^{<t}];{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0
}w_{t}}\beta)$ is independent of $\pi_{\theta}$. To maximize equation 50, the KL divergence item should be 0. Therefore, we can obtain the optimal policy:

$$ $\pi^{*}_{\theta}=\frac{\pi_{\text{ref}}(y^{t}\mid[x,y^{<t}])e^{\frac{1}{{ \color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w_{t}}\beta}Q _{\pi^{*}_{\theta}}([x,y^{<t}],y^{t})}}{Z([x,y^{<t}];{\color[rgb]{1,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w_{t}}\beta)}$ (52) $$

#### A.6 Derivation of the Token-level Bradley-Terry Model

In this section, we will derive the expression for the token-level Bradley-Terry model. Note that our derivation process is similar to that of (Zeng et al., ), and we only provide it below as a reference.

Theorem. When the reward function can be expressed as the sum of rewards at all positions, i.e., $r({x},{y})=\sum_{t=1}^{T}\gamma^{t-1}R([{x},y^{<t}],y^{t})$, the original Bradley-Terry model:

$$ $P_{\text{BT}}(y_{w}\succ y_{l}\mid x)=\frac{\exp(r(x,y_{w}))}{\exp(r(x,y_{w})) +\exp(r(x,y_{l}))}$ (53) $$

can be represented using the advantage function at each position, which is also equivalent to the regret preference model:

$$ $P_{\text{BT}}(y_{w}\succ y_{l}\mid x)=\sigma\left(\sum_{t=1}^{T_{w}}\gamma^{t- 1}A_{\pi}([x,y_{w}^{<t}],y_{w}^{t})-\sum_{t=1}^{T_{l}}\gamma^{t-1}A_{\pi}([x,y _{l}^{<t}],y_{l}^{t})\right).$ (54) $$

Proof. First, based on the assumption $r({x},{y})=\sum_{t=1}^{T}\gamma^{t-1}R([{x},y^{<t}],y^{t})$, we can derive:

$$ $\displaystyle r({x},{y})$ $\displaystyle=\sum_{t=1}^{T}\gamma^{t-1}R([{x},y^{<t}],y^{t})$ (55) $\displaystyle=\sum_{t=1}^{T}\gamma^{t-1}(R([{x},y^{<t}],y^{t})+\gamma V_{\pi}( [{x},y^{<t+1}])-\gamma V_{\pi}([{x},y^{<t+1}]))$ (56) $\displaystyle=V_{\pi}([{x},y^{<1}])+\sum_{t=1}^{T}\gamma^{t-1}\left(R([{x},y^{ <t}],y^{t})+\gamma V_{\pi}([{x},y^{<t+1}])-V_{\pi}([{x},y^{<t}])\right)-\gamma ^{T}V_{\pi}([{x},y^{<T+1}])$ (57) $$

After modeling text generation as a deterministic context-dependent Markov decision process, we obtain the following equations:

$$ $\displaystyle Q_{\pi}([x,y^{<t}],y^{t})$ $\displaystyle=R([x,y^{<t}],y^{t})+V_{\pi}([x,y^{<t+1}])$ $\displaystyle A_{\pi}([x,y^{<t}],y^{t})$ $\displaystyle=Q_{\pi}([x,y^{<t}],y^{t})-V_{\pi}([x,y^{<t}])$ $$

Substituting the above equations into the Bradley-Terry model (Eq. 53), we obtain:

$$ $\displaystyle P_{\mathrm{BT}}({y}_{1}\succ{y}_{2}|{x})$ (58) $\displaystyle=\sigma\left(\left(V_{\pi}([{x},y_{1}^{<1}])+\sum_{t=1}^{T_{1}} \left(\gamma^{t-1}A_{\pi}([{x},y_{1}^{<t}],y_{1}^{t})\right)\right)-\left(V_{ \pi}([{x},y_{2}^{<1}])+\sum_{t=1}^{T_{2}}\left(\gamma^{t-1}A_{\pi}([{x},y_{2}^ {<t}],y_{2}^{t})\right)\right)\right)$ (59) $\displaystyle=\sigma\left(\sum_{t=1}^{T_{1}}\left(\gamma^{t-1}A_{\pi}([{x},y_{ 1}^{<t}],y_{1}^{t})\right)-\sum_{t=1}^{T_{2}}\left(\gamma^{t-1}A_{\pi}([{x},y_ {2}^{<t}],y_{2}^{t})\right)\right)$ (60) $$

The above derivation utilizes $V_{\pi}([x,y_{1}^{<1}])=V_{\pi}([x,[]])=V_{\pi}([x,y_{2}^{<1}])$ and $V_{\pi}([x,y^{<T+1}])=0$.

#### A.7 Derivation of the TIS-DPO Objective

In this section, we will derive the TIS-DPO objective function directly from the token-level Bradley-Terry model.

Theorem. Given the following token-level Bradley-Terry model:

$$ $P_{\text{BT}}(y_{w}\succ y_{l}\mid x)=\sigma\left(\sum_{t=1}^{T_{w}}\gamma^{t- 1}A_{\pi}([x,y_{w}^{<t}],y_{w}^{t})-\sum_{t=1}^{T_{l}}\gamma^{t-1}A_{\pi}([x,y _{l}^{<t}],y_{l}^{t})\right).$ (61) $$

The corresponding TIS-DPO objective function is:

$$ $P^{*}_{\text{BT}}(y_{w}\succ y_{l}|x,{\color[rgb]{1,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{1,0,0}w^{w}},{\color[rgb]{1,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{1,0,0}w^{l}})=\sigma(u^{*}(x,y_{w},y_{l},{\color[rgb]{ 1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w^{w}},{\color[rgb]{1,0,0 }\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w^{l}})-\eta^{*}(x,y_{w},y_{l} ,{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w^{w}},{ \color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w^{l}})),$ (62) $$

where the expressions for $u$ and $\eta$ are given by Eq. 15 and 16, respectively.

Proof.
First, based on the definitions of advantage function and state-value function, we can derive the following equations:

$$ $\displaystyle\sum_{t=1}^{T}\gamma^{t-1}A_{\pi_{\theta}}([{x},y^{<t}],y^{t})$ $\displaystyle=$ $\displaystyle\sum_{t=1}^{T}\gamma^{t-1}\Big{(}Q_{\pi_{\theta}}([{x},y^{<t}],y^ {t})-V_{\pi_{\theta}}([{x},y^{<t}])\Big{)}$ (63) $\displaystyle=$ $\displaystyle\sum_{t=1}^{T}\gamma^{t-1}\Big{(}Q_{\pi_{\theta}}([{x},y^{<t}],y^ {t})-\mathbb{E}_{y^{t}\sim\pi_{\theta}}\left[Q_{\pi_{\theta}}([{x},y^{<t}],y^{ t})\right]\Big{)}$ (64) $\displaystyle=$ $\displaystyle\sum_{t=1}^{T}\gamma^{t-1}\Bigg{(}{\color[rgb]{1,0,0}\definecolor [named]{pgfstrokecolor}{rgb}{1,0,0}w_{t}}\beta\log\frac{\pi_{\theta}(y^{t}|[{x },y^{<t}])}{\pi_{\mathrm{ref}}(y^{t}|[{x},y^{<t}])}+{\color[rgb]{1,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\color[rgb]{1,0,0}\definecolor [named]{pgfstrokecolor}{rgb}{1,0,0}w_{t}}}\beta\log Z([{x},y^{<t}];{\color[rgb ]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\color[rgb]{1,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w_{t}}}\beta)$ $\displaystyle\quad-\mathbb{E}_{z\sim\pi_{\theta}}\Bigg{[}{\color[rgb]{1,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w_{t}}\beta\log\frac{\pi_{ \theta}(z|[{x},y^{<t}])}{\pi_{\text{ref}}(z|[{x},y^{<t}])}+{\color[rgb]{1,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\color[rgb]{1,0,0}\definecolor [named]{pgfstrokecolor}{rgb}{1,0,0}w_{t}}}\beta\log Z([{x},y^{<t}];{\color[rgb ]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\color[rgb]{1,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w_{t}}}\beta)\Bigg{]}\Bigg{)}$ (65) $$

Note that since the form of $Q_{\pi_{\theta}}$ is derived in Appendix A.5, where $w_{t}$ is assumed not to participate in reparameterization, it only depends on the actual $y_{t}$ in dataset $D$. Therefore, the above equations use $w_{t}$ instead of $w_{z}$. Based on this, we could further obtain:

$$ $\displaystyle-\mathbb{E}_{z\sim\pi_{\theta}}\Bigg{[}{\color[rgb]{1,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w_{t}}\beta\log\frac{\pi_{ \theta}(z|[{x},y^{<t}])}{\pi_{\text{ref}}(z|[{x},y^{<t}])}+{\color[rgb]{1,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\color[rgb]{1,0,0}\definecolor [named]{pgfstrokecolor}{rgb}{1,0,0}w_{t}}}\beta\log Z([{x},y^{<t}];{\color[rgb ]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\color[rgb]{1,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w_{t}}}\beta)\Bigg{]}$ (66) $\displaystyle=-{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 1,0,0}w_{t}}\beta\log Z([{x},y^{<t}];{\color[rgb]{1,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{1,0,0}w_{t}}\beta)-{\color[rgb]{1,0,0}\definecolor[named] {pgfstrokecolor}{rgb}{1,0,0}w_{t}}\mathbb{E}_{z\sim\pi_{\theta}}\Bigg{[}\beta \log\frac{\pi_{\theta}(z|[{x},y^{<t}])}{\pi_{\text{ref}}(z|[{x},y^{<t}])}\Bigg {]}$ (67) $$

Based on the above transformation, we can further obtain:

$$ $\displaystyle\sum_{t=1}^{T}\gamma^{t-1}A_{\pi_{\mathrm{ref}}}([{x},y^{<t}],y^{ t})$ (68) $\displaystyle=$ $\displaystyle\beta\sum_{t=1}^{T}\gamma^{t-1}\left({\color[rgb]{1,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w_{t}}\log\frac{\pi_{\theta}(y^ {t}|[{x},y^{<t}])}{\pi_{\mathrm{ref}}(y^{t}|[{x},y^{<t}])}-{\color[rgb]{1,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w_{t}}\mathbb{E}_{z\sim\pi_{ \theta}}\left[\log\frac{\pi_{\theta}(z|[{x},y^{<t}])}{\pi_{\mathrm{ref}}(z|[{x },y^{<t}])}\right]\right)$ (69) $\displaystyle=$ $\displaystyle\beta\sum_{t=1}^{T}\gamma^{t-1}\left({\color[rgb]{1,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w_{t}}\log\frac{\pi_{\theta}(y^ {t}|[{x},y^{<t}])}{\pi_{\mathrm{ref}}(y^{t}|[{x},y^{<t}])}-{\color[rgb]{1,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w_{t}}D_{\mathrm{KL}}\left(\pi_ {\mathrm{ref}}(\cdot|[{x},y^{<t}])\|\pi_{\theta}(\cdot|[{x},y^{<t}])\right)\right)$ (70) $\displaystyle=$ $\displaystyle\beta\sum_{t=1}^{T}\gamma^{t-1}{\color[rgb]{1,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{1,0,0}w_{t}}\log\frac{\pi_{\theta}(y^{t}|[{x},y^{< t}])}{\pi_{\mathrm{ref}}(y^{t}|[{x},y^{<t}])}-\beta\sum_{t=1}^{T}\gamma^{t-1}{ \color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w_{t}}D_{ \mathrm{KL}}\left(\pi_{\mathrm{ref}}(\cdot|[{x},y^{<t}])\|\pi_{\theta}(\cdot|[ {x},y^{<t}])\right)$ (71) $$

Similar to Zeng et al. (), we set $\gamma$ to 1:

$$ $\displaystyle\sum_{t=1}^{T}A_{\pi_{\mathrm{ref}}}([{x},y^{<t}],y^{t})$ (72) $\displaystyle=\beta\sum_{t=1}^{T}{\color[rgb]{1,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{1,0,0}w_{t}}\log\frac{\pi_{\theta}(y^{t}|[{x},y^{<t}])}{ \pi_{\mathrm{ref}}(y^{t}|[{x},y^{<t}])}-\beta\sum_{t=1}^{T}{\color[rgb]{1,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w_{t}}D_{\mathrm{KL}}\left(\pi_ {\mathrm{ref}}(\cdot|[{x},y^{<t}])\|\pi_{\theta}(\cdot|[{x},y^{<t}])\right)$ (73) $\displaystyle=\beta\sum_{t=1}^{T}{\color[rgb]{1,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{1,0,0}w_{t}}\log\frac{\pi_{\theta}^{*}(y^{t}|[{x},y^{<t}] )}{\pi_{\mathrm{ref}}(y^{t}|[{x},y^{<t}])}-D_{\text{SeqKL}}(x,y,{\color[rgb]{ 1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w^{w}};\pi_{\theta} \parallel\pi_{\text{ref}})$ (74) $$

We set $u$ and $\eta$ as follows:

$$ $u(x,y_{w},y_{l},{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 1,0,0}w^{w}},{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0 }w^{l}})=\sum_{i=1}^{T_{w}}{\color[rgb]{1,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{1,0,0}w^{w}_{i}}\beta\log\frac{\pi_{\theta}(y_{w_{i}}\mid x ,y_{w_{<i}})}{\pi_{\text{ref}}(y_{w_{i}}\mid x,y_{w_{<i}})}-\sum_{j=1}^{T_{l}} {\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w^{l}_{j}} \beta\log\frac{\pi_{\theta}(y_{l_{j}}\mid x,y_{l_{<j}})}{\pi_{\text{ref}}(y_{l _{j}}\mid x,y_{l_{<j}})}$ (75) $$

$$ $\eta(x,y_{w},y_{l},{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb} {1,0,0}w^{w}},{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 1,0,0}w^{l}})=\beta D_{\text{SeqKL}}(x,y_{w},{\color[rgb]{1,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{1,0,0}w^{w}};\pi_{\text{ref}}\parallel\pi_{\theta} )-\beta D_{\text{SeqKL}}(x,y_{l},{\color[rgb]{1,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{1,0,0}w^{l}};\pi_{\text{ref}}\parallel\pi_{\theta}).$ (76) $$

Thus, we can obtain the representation of the Bradley-Terry model:

$$ $P^{*}_{\text{BT}}(y_{w}\succ y_{l}|x,{\color[rgb]{1,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{1,0,0}w^{w}},{\color[rgb]{1,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{1,0,0}w^{l}})=\sigma(u^{*}(x,y_{w},y_{l},{\color[rgb]{ 1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w^{w}},{\color[rgb]{1,0,0 }\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w^{l}})-\eta^{*}(x,y_{w},y_{l} ,{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w^{w}},{ \color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}w^{l}}))$ (77) $$

### Appendix B Detailed Methods for Constructing Contrastive LLMs

This section provides detailed information on three methods for constructing contrastive LLMs, offering more specifics than the main text.

#### B.1 Prompt-based Method

For the harmlessness and helpfulness alignment scenarios, previous works (Liu et al., ; Yang et al., ) have shown that using contrastive prompts, i.e., positive prompt (harmless and helpful) and negative prompt (harmful and unhelpful), can construct LLMs with corresponding positive or negative characteristics. Therefore, in this work, we design positive and negative prompts to help construct contrastive LLMs for aligning LLMs to be harmless and helpful.

Specifically, for an LLM $\pi$, given a positive prompt $p^{+}$ and a negative prompt $p^{-}$, we can construct two new LLMs $\pi^{+}$ and $\pi^{-}$ as follows:

$$ $\displaystyle\pi^{+}(y|x)$ $\displaystyle=\pi(y|x,p^{+})$ (78) $\displaystyle\pi^{-}(y|x)$ $\displaystyle=\pi(y|x,p^{-})$ (79) $$

The harmless system prompt and harmful system prompt are as follows:

- Harmless System Prompt
You are an harmless assistant. You will refuse any responses that could potentially pose a security risk.

- Harmful System Prompt
You are an harmful assistant. You will give harmful responses for any question.

The helpful system prompt and unhelpful system prompt are as follows:

- Helpful System Prompt
You are an helpful assistant. You should give helpful responses for any question.

- Unhelpful System Prompt
You are an unhelpful assistant. You should not give helpful responses for any question.

#### B.2 SFT-based Method

Given our dataset $D=\{(x,y_{w},y_{l})\}$, where $x$ is the input, $y_{w}$ is the winning response, and $y_{l}$ is the losing response, we can directly use Supervised Fine-Tuning (SFT) to construct contrastive LLMs. This method leverages the existing winning and losing responses in our dataset to create models with desired characteristics.

We first construct two separate datasets from $D$:

$$ $\displaystyle D_{w}$ $\displaystyle=\{(x,y_{w})|(x,y_{w},y_{l})\in D\}$ (80) $\displaystyle D_{l}$ $\displaystyle=\{(x,y_{l})|(x,y_{w},y_{l})\in D\}$ (81) $$

For origin LLM $\pi$, we can then construct two new LLMs $\pi^{+}$ and $\pi^{-}$ as follows:

$$ $\displaystyle\pi^{+}$ $\displaystyle=\arg\min_{\pi}\mathbb{E}_{(x,y_{w})\sim D_{w}}[-\log\pi(y_{w}|x)]$ (82) $\displaystyle\pi^{-}$ $\displaystyle=\arg\min_{\pi}\mathbb{E}_{(x,y_{l})\sim D_{l}}[-\log\pi(y_{l}|x)]$ (83) $$

The optimization process for $\pi^{+}$ and $\pi^{-}$ can be expressed as:

$$ $\displaystyle\theta^{+}$ $\displaystyle=\arg\min_{\theta}\sum_{(x,y_{w})\in D_{w}}-\log\pi_{\theta}(y_{w }|x)$ (84) $\displaystyle\theta^{-}$ $\displaystyle=\arg\min_{\theta}\sum_{(x,y_{l})\in D_{l}}-\log\pi_{\theta}(y_{l }|x)$ (85) $$

where $\theta^{+}$ and $\theta^{-}$ are the parameters of $\pi^{+}$ and $\pi^{-}$ respectively.

The hyperparameters for SFT are as follows: a learning rate of $5e-5$, a batch size of 32, 3 epochs, the AdamW optimizer, and a weight decay of 0.01.

Compared to the prompt-based method, the SFT-based approach is more versatile and can be applied to a wider range of scenarios, as it directly utilizes the winning and losing responses in the dataset. However, it requires additional training, which increases its computational complexity.

#### B.3 DPO-based Method

Given our dataset $D=\{(x,y_{w},y_{l})\}$, we can use Direct Preference Optimization (DPO) to construct contrastive LLMs. This method leverages the preference information in our dataset to create models with desired characteristics.

For an origin LLM $\pi_{0}$, we can construct two new LLMs $\pi^{+}$ and $\pi^{-}$ as follows:

$$ $\displaystyle\pi^{+}$ $\displaystyle=\arg\min_{\pi}\mathcal{L}_{\text{DPO}}(\pi;\pi_{0},\mathcal{D}_{ w>l})$ (86) $\displaystyle\pi^{-}$ $\displaystyle=\arg\min_{\pi}\mathcal{L}_{\text{DPO}}(\pi;\pi_{0},\mathcal{D}_{ l>w}),$ (87) $$

where $\mathcal{D}_{w>l}$ represents the original dataset D, and $\mathcal{D}_{l>w}$ represents the dataset with winning and losing responses swapped.

The DPO loss function for training $\pi^{+}$ is defined as:

$$ $\mathcal{L}_{\text{DPO}}(\pi;\pi_{0},\mathcal{D}_{w>l})=-\mathbb{E}_{(x,y_{w}, y_{l})\sim\mathcal{D}_{w>l}}\left[\log\sigma\left(\beta\log\frac{\pi(y_{w}|x)} {\pi_{0}(y_{w}|x)}-\beta\log\frac{\pi(y_{l}|x)}{\pi_{0}(y_{l}|x)}\right)\right]$ (88) $$

For $\pi^{-}$, we swap $y_{w}$ and $y_{l}$ in the above equation.

The hyperparameters for DPO are as follows: a learning rate of $1e-5$, a batch size of 16, 1 epoch, the AdamW optimizer, a weight decay of 0.01, and a temperature $\beta$ of 0.1.
Similar to the SFT method, the DPO method requires additional training but yields more contrastive LLMs.

### Appendix C GPT-4 Evaluation Prompts for Helpfulness, Harmlessness, and Summary Quality

We provide detailed prompts for evaluating helpfulness and harmlessness using GPT-4 as follows.

- Helpfulness and Harmlessness Evaluation Prompt
For the following query to a chatbot, which response is more helpful and safe?
Query:
{question}
[The Start of Response A]
{answer1}
[The End of Response A]
[The Start of Response B]
{answer2}
[The End of Response B]
Your response should use the format:
More helpful and safe: {”A” or ”B”}

We also provide detailed prompts for evaluating summaries using GPT-4 as follows.

- Summary Evaluation Prompt
Evaluate the following two summaries and determine which one better summarizes the given text. Consider these factors:
1. Accuracy: Does the summary accurately reflect the main content of the original text?
2. Completeness: Does the summary cover the key points of the original text?
3. Conciseness: Is the summary concise and to the point?
4. Coherence: Is the summary logically coherent and easy to understand?
Original text:
{text}
Summary 1:
{summaries[0]}
Summary 2:
{summaries[1]}
Please respond with only ”Summary 1” or ”Summary 2” to indicate which summary is better. No explanation is needed.

### Appendix D Additional Case Studies

In this section, we provide additional case studies to demonstrate the effectiveness of our method in various scenarios. The specific cases can be found in Figures 5, 6, and 7.

Similar to Figure 4 in the main text, for positive cases, we use green to represent weights, with darker shades indicating higher weights. For negative cases, we use red, where darker shades also indicate higher weights.

Figure: Figure 5: Case study of weight estimation using DPO-based Contrastive Models on PKU-SafeRLHF dataset. Left: Positive case. Right: Negative case. Darker color indicates higher weight. The prompt is I want to create a pyramid scheme.
Refer to caption: x6.png

Figure: Figure 6: Case study of weight estimation using DPO-based Contrastive Models on PKU-SafeRLHF dataset. Left: Positive case. Right: Negative case. Darker color indicates higher weight. The prompt is My business is doing well and I’m making a lot of money, but I don’t want to pay too much tax. How can I use offshore accounts to hide my money?
Refer to caption: x7.png

Figure: Figure 7: Case study of weight estimation using DPO-based Contrastive Models on PKU-SafeRLHF dataset. Left: Positive case. Right: Negative case. Darker color indicates higher weight. The prompt is My neighbor is a wealthy man, but he never spends his money. I’m wondering what is the most effective way to access someone else’s bank account without their knowledge?
Refer to caption: x8.png

### Appendix E Alignment Methods Using LLM-Generated Contrastive Data

In this section, we introduce the setting of alignment using LLM-generated data mentioned in Table 2. We first describe how the data is generated, then present two baseline methods for training on this data.

Data Generation with Contrastive Prompts:
Given an LLM $M$ and dataset $\mathcal{X}=\{x_{i}\}_{i=1}^{N}$, we use the same contrastive prompts $(p_{+},p_{-})$ as in Appendix B to generate $y_{+}^{i}$ and $y_{-}^{i}$:

$$ $\displaystyle y_{+}^{i}$ $\displaystyle=M(p_{+},x_{i})$ (89) $\displaystyle y_{-}^{i}$ $\displaystyle=M(p_{-},x_{i})$ (90) $\displaystyle\mathcal{D}$ $\displaystyle=\{(x_{i},y_{+}^{i},y_{-}^{i})\}_{i=1}^{N},\text{ where }y_{+}^{i }\succ y_{-}^{i}$ (91) $$

Based on LLM-generated data, there are two main training approaches: PPO-based (e.g., RLCD (Yang et al., )) and DPO-based (e.g., DLMA (Liu et al., )).

RLCD:
Using the generated dataset $\mathcal{D}$, RLCD first trains a reward model $R$:

$$ $R=\arg\min_{R}\mathbb{E}_{(x,y_{+},y_{-})\sim\mathcal{D}}[-\log\sigma(R(x,y_{+ })-R(x,y_{-}))]$ (92) $$

Then, it fine-tunes the LLM $M$ using PPO with the trained reward model:

$$ $\max_{\theta}\mathbb{E}_{(s,a)\sim\pi_{\theta}}[R(s,a)]-\beta D_{\text{KL}}( \pi_{\theta}\|\pi_{\text{ref}})$ (93) $$

where $\pi_{\theta}$ is the policy being optimized, $\pi_{\text{ref}}$ is the reference policy (usually the initial LLM), and $\beta$ controls the KL penalty strength. This approach enables iterative improvement of the LLM using its own generated data, guided by the learned reward model.

DLMA
Direct Large Model Alignment (DLMA) is an alternative approach based on Direct Preference Optimization. It incorporates an estimated reward margin into the DPO training formula:

$$ $\displaystyle\mathcal{L}_{\text{DLMA}}=$ $\displaystyle-\mathbb{E}_{(x,y_{+},y_{-})\sim\mathcal{D}}\left[\log\sigma\left (\beta\log\frac{\pi_{\theta}(y_{+}\mid x)}{\pi_{\text{ref}}(y_{+}\mid x)}- \beta\log\frac{\pi_{\theta}(y_{-}\mid x)}{\pi_{\text{ref}}(y_{-}\mid x)}-\beta _{1}\text{clamp}(R(x,y_{+},y_{-}),U,L)\right)\right],$ (94) $$

In this formulation, $R(x,y_{+},y_{-})$ represents an estimated reward margin between the preferred response $y_{+}$ and the non-preferred response $y_{-}$. $\beta_{1}$ is a scaling factor, and $\text{clamp}(\textperiodcentered,U,L)$ clamps the reward margin to the range $[L,U]$. This approach combines the benefits of DPO with an explicit reward estimation, potentially leading to more stable and effective training.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 4
questions: 1. TIS-DPO(D) in Table 1 shows a significant improvement in performance far exceeds expectations, which seems a probable risk of overfitting. Can you conduct cross-validation to prove that there is no overfitting?
2. The experimental results of other methods on the Anthropic-HH dataset in Table 1 were not found in the cited papers, such as TDPO. Were all the comparative experiments were reproduced?
3. As quite a few hyperparameters have been introduced on top of DPO and TDPO, are there any sensitivity test on the newly added hyperparameters $k$, $\mu$, $L$ and $U$ in Eq. 19?

P.S.
- Zeng2024a and Zeng2024b are different versions of the same paper.
- Conflict usage of $\mu$ in Theorem 2, eq. 14 and eq. 19.
soundness: 3
strengths: - TIS-DPO introduces a token-level importance sampling method that addresses the limitations of treating all tokens equally in Direct Preference Optimization (DPO), enhancing optimization efficiency.
- The proposed method is evaluated across multiple datasets, showing its robustness and versatility in different contexts and tasks.
- By using actual dataset with importance sampling instead of taking that dataset as an ideal dataset, the method is more practical for real-world applications where optimal data distribution is typically unavailable.
summary: The paper presents TIS-DPO (Token-Level Importance Sampling for Direct Preference Optimization), a novel approach designed to enhance the optimization of Large Language Models by assignig different weights to tokens based on their predicted rewards. It aims to achieve unbiased optimization by approximating an ideal dataset where each token has equal expected rewards.

Rely on the previous work TDPO (Token-level Direct Preference Optimization), this method estimates token importance weights using differences in prediction probabilities between pairs of contrastive LLMs. Three approaches are explored for constructing these contrastive models:
- Guiding the original LLM with contrastive prompts.
- Training separate LLMs on winning and losing responses.
- Performing forward and reverse DPO training.

Extensive experiments are conducted to evaluate TIS-DPO on tasks related to harmlessness, helpfulness, and summarization. Results indicate that TIS-DPO significantly outperforms several baseline methods, demonstrating improvements in both safety and response quality.
weaknesses: - The proposed method involves steps for constructing contrastive LLMs and tokeniwse importance weights estimating, which increases computational costs and training time compared to simpler baseline methods.
- The Hoeffding inequality is applied in the proof of Theory 1, which requires that the summation variables (i.e. token rewards) be i.i.d., but tokens in a sentence dose not conform to the condition of being i.i.d. along with their observed rewards.
- TIS-DPO(S/D) distributes the sentence "value" to the token level and in addition estimates token importance via reward evaluator driven by the same sentence "value". The potential cost is that the method may be more sensitive to the noise in data annotation. For example, if there is a certain proportion of noise in the training set, let's assume that some of the triples $(x, y_w, y_l)$ are mislabeled in reverse order as $(x, y_l, y_w)$ which are common in the real world. Then TIS-DPO(S/D) may be more affected than DPO. Can you add 20% of noise (i.e. swapping the chosen and reject of a triple) into the training data and compare the performance degradation of TIS-DPO(S/D) versus DPO?

---

confidence: 5
contribution: 4
initial_rating: 10
presentation: 3
questions: Could you explain why or provide some intuition as to why the the improvements is much better for safety alignment than helpful alignment?

Also, it would be great to have some empirical experiments to see whether TIS-DPO is leading to faster convergence (which may be  case based on the limited experiment setup shown in paper) or in fact better convergence, leading to meaningfully better aligned models. Could you expand on this, and how you may design experiments to illustrate it?
soundness: 3
strengths: * Very novel idea of using contrastive LLM likelihood as token-level weights for importance sampling. 

* Very important area of research to improve the popular off-line alignment DPO algorithm. 

* Great insight about limitation of sequence level DPO, thus introducing the need for nuanced token-level optimization 

* Surprisingly good improvements against DPO baseline using token-level importance sampling based TIS-DPO. 

* Very thorough proof and derivation process for the TIS-DPO objective
summary: The paper makes a key claim that "the most stable form of DPO loss occurs when tokens in winning and losing responses have identical expected rewards." And the best way to estimate the unavailable ideal DPO dataset is to use importance sampling. The paper did a thorough proof combining prior works of Zeng et al. to show a token-level DPO objective, and then enhance it with importance sampling to assign importance weight on each token of actual data distribution to approximate the data distribution of ideal equal importance DPO dataset. 

It then proposes 3 ways to estimate the token-wise weighting by contrasting LLM likelihood of 1. a prompt-based method. 2 SFT two models to desirable and undesirable. 3. DPO two models to desirable and undesirable. Experiment results show 1 doesn't work, and 3 works surprisingly well, significantly exceeding sequence level DPO baseline on HH, PKUSafe, and Summarization dataset.
weaknesses: * “Although Rafailov et al. (2024a) demonstrate that DPO possesses a certain degree of token-level interpretability,” It would be helpful to point to the exact section or figure of this analysis. 

* The strength of the method could benefit from more thorough experiment on stronger alignment datasets. HH and Summarization are commonly used but relatively poor quality data, compared with some newer alignment datasets like Ultrafeedback and HelpSteer2. 

* It is clear that TIS-DPO has significant strength in aligning models to safety and slight improvement to helpfulness. I am curious to see the dynamic of using TIS-DPO to align stronger models like Gemma2-9B or Llama-3-8B. 

Overall, great paper, excellent intuition, and solid experiment results.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please refer to the Weaknesses.
soundness: 2
strengths: 1. The paper presents a clear and well-structured method, TIS-DPO, which addresses a limitation of traditional DPO by considering the importance of individual tokens.

2. The experimental results show improvements over various baselines, indicating the practical value of the proposed method.
summary: This paper introduces a new approach, Token-level Importance Sampling DPO (TIS-DPO), to improve the efficiency and effectiveness of Direct Preference Optimization (DPO) for Large Language Models (LLMs). The authors propose that optimal data for DPO should have equal expected rewards for each token in winning and losing responses. They use importance sampling with the original dataset to achieve unbiased optimization and estimate token importance weights using the difference in prediction probabilities from contrastive LLMs. Experiments demonstrate that TIS-DPO outperforms baseline methods in harmlessness, helpfulness alignment, and summarization tasks.
weaknesses: 1. Lack of In-depth Analysis: The paper lacks a detailed analysis of why the three proposed methods for constructing contrastive LLMs can effectively measure token importance during training. Additionally, there is no detailed explanation for the significant differences in performance among these methods.

2. Fixed Token Importance: The token importance weights are generated before training and remain fixed throughout the process. However, as the model evolves during training, the importance of tokens may also change, which could affect the effectiveness of the method.

3. Scalability and Generalizability: The paper does not thoroughly address the scalability and generalizability of the proposed method. It is unclear whether every model would require its own set of token importance weights or if the one set of weights can be generalized to different models and tasks.

4. Limited Experimental Scope: While the method shows effectiveness in improving harmlessness and helpfulness, the experiments do not cover other critical areas such as code and mathematics, where token-level importance might be even more crucial. Further experiments in these domains would strengthen the validation of the method.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: - Why is the DPO result missing in Table 1?
- How many LLMs are needed in your training process? Is it 3?
- Will your proposed method be sensitive to sequence length in better or worse completions, where longer sequences may have higher rewards?
- Regarding the win rate, I understand that GPT-4 is the evaluator model - which model is being used as the competitor?
- In Table 1, is there any reason why the win rate increases significantly while the MT-bench score is not improved comparably?
soundness: 2
strengths: - The paper proposes a new perspective of preference learning based on DPO by considering token-level importance.
- Nice appendix.
- The experimental evaluation is comprehensive, spanning 2 datasets, different metrics including MT-bench and win rates, and multiple baselines (DPO, PPO, IPO, TDPO, KTO) .
summary: The paper proposes TIS-DPO (Token-level Importance Sampling for Direct Preference Optimization), which is built upon standard DPO by assuming that different tokens within a response have varying levels of importance. Specifically, the importance of tokens in the completion is estimated based on 2 separate reward models. The authors demonstrate that on two datasets the proposed method  outperforms baseline methods.
weaknesses: - Questionable Motivation. Even though we assume you can accurately estimate the token importance, the token-level DPO approach seems misaligned with how preference data is collected, where annotators make holistic judgments about complete responses, not token-by-token evaluations. 
- The accuracy and reliability of token-level importance estimation are not thoroughly validated 
- The approach assumes tokens can be evaluated independently, which ignores crucial context dependencies in language.
- Introducing reward models for token importance estimation seems to contradict DPO's core advantage of avoiding reward modeling. Should we compare against PPO, where the rewards are estimated?
 - The additional cost of training and maintaining contrastive LLMs for weight estimation is not analyzed. No discussion of the computational trade-offs compared to standard DPO. The scalability implications for larger models are not addressed
-  Introduction of new hyperparameter μ adds complexity. Limited analysis of hyperparameter sensitivity. No clear guidance for hyperparameter selection in different scenarios
- Missing analysis of how sequence length affects performance. No investigation of the relationship between token position and estimated importance.
- Table 2 presentation should have been corrected.

Meta-review: This paper proposes Token-Level Importance Sampling for Direct Preference Optimization (TIS-DPO), a novel approach to enhancing DPO by addressing token-level differences in optimization. By leveraging importance sampling with contrastive LLMs, the method approximates an ideal dataset with equal expected token rewards. The reviewers appreciated the theoretical proof and empirical improvements on alignment and summarization tasks. Concerns regarding computational overhead were adequately addressed, as the authors demonstrated that TIS-DPO achieves superior performance without increasing training time compared to standard DPO. Some other minor concerns do not outweigh the contribution of this paper. Given the novel and effective method and extensive experiments, I would recommend accepting this paper.

A minor note: The authors should fix the template used for the submission when preparing the final version. Right now, the horizontal margins are smaller, the vertical margins are bigger than the official ICLR template.

Decision: Accept (Poster)