TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weights
Authors: Aiwei Liu
Haoping Bai
Zhiyun Lu
Yanchao Sun
Xiang Kong
Xiaoming Simon Wang
Jiulong Shan
Albin Madappally Jose
Xiaojiang Liu
Lijie Wen
Philip S. Yu
Meng Cao
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Models
Importance Sampling
Preference Learning
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We propose TIS-DPO, a token-level importance sampling method for DPO that improves LLM alignment by assigning different weights to tokens based on their estimated importance.

Abstract: Direct Preference Optimization (DPO) has been widely adopted for preference alignment of Large Language Models (LLMs) due to its simplicity and effectiveness. 
However, DPO is derived as a bandit problem in which the whole response is treated as a single arm, ignoring the importance differences between tokens, which may affect optimization efficiency and make it difficult to achieve optimal results.
In this work, we propose that the optimal data for DPO has equal expected rewards for each token in winning and losing responses, as there is no difference in token importance. 
However, since the optimal dataset is unavailable in practice, we propose using the original dataset for importance sampling to achieve unbiased optimization. 
Accordingly, we propose a token-level importance sampling DPO objective named TIS-DPO that assigns importance weights to each token based on its reward.
Inspired by previous works, we estimate the token importance weights using the difference in prediction probabilities from a pair of contrastive LLMs. We explore three methods to construct these contrastive LLMs: (1) guiding the original LLM with contrastive prompts, (2) training two separate LLMs using winning and losing responses, and (3) performing forward and reverse DPO training with winning and losing responses.
Experiments show that TIS-DPO significantly outperforms various baseline methods on harmlessness and helpfulness alignment and summarization tasks. We also visualize the estimated weights, demonstrating their ability to identify key token positions.

Published as a conference paper at ICLR 2025
TIS-DPO: TOKEN-LEVEL IMPORTANCE SAMPLING FOR DIRECT PREFERENCE OPTIMIZATION WITH ESTI- MATED WEIGHTS
Aiwei Liu Haoping Bai²,
Zhiyun Lu?, Yanchao Sun?, Xiang Kong",
Simon Wang", Jiulong Shan
Albin Madappally Jose", Xiaojiang Liu",
Lijie Wen' : Philip S. Yu²,
Meng Cao² I
'Tsinghua University AAplle University of Illinois at Chicago
1 liuaw20emails. tsinghua. edu wenlj@tsinghua. edu.cn, en, mengcad@apple.com
ABSTRACT
Direct Preference Optimization (DPO) has been widely adopted for preference alignment of Large Language Models (LLMs) due to its simplicity and effective- ness. However, DPO is derived as a bandit problem in which the whole response is treated a single arm, ignoring the importance differences between tokens, which may affect optimization efficiency and make it difficult to achieve optimal results. In this work, we propose that the optimal data for DPO has equal ex- pected rewards for each token in winning and losing responses, as there is no difference in token importance. However, since the optimal dataset unavail- able in practice, we propose using the original dataset for importance sampling achieve unbiased optimization. Accordingly, we propose token-level impor- tance sampling DPO objective named TIS-DPO that assigns importance weights to each token based on its reward. Inspired by previous works, we estimate the token importance weights using the difference in prediction probabilities from a pair of contrastive LLMss. We explore three methods to construct these contrastive LLMs: (1) guiding the original LLM with contrastive prompts, (2) training two separate LLMs using winning and losing responses, and (3) performing forward and reverse DPO training with winning and losing responses. Experiments show that TIS-DPO significantly outperforms various baseline methods on harmlessness and helpfulness alignment and summarization tasks. We also visualize the esti- mated weights, demonstrating their ability to identify key token positions. Code is available https:: //gi thub, com/ ex aw/TIS-DPO.
INTRODUCTION
The importance of Large Language Model (LLM) alignment (Ji et al., 2023) techniques has grown alongside the increasing capabilities of LLMs. These techniques aim to align LLMS with human values, ensuring the generation of helpful and harmless content (Bai et al., 2022). Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al., 2022) is a common alignment approach that trains reward model on human-labeled preference data and optimizes the LLM using rein- forcement learning methods like Proximal Policy Optimization (PPO) (Schulman et al., 2017) to maximize the generated reward under the reward model. However, RLHF is relatively complex due to the need for reinforcement learning techniques.
To simplify alignment process, Direct Preference Optimization (DPO) (Rafailov al., 2024b) lever- ages the relationship between policy and reward functions to optimize both simultaneously with- out reinforcement learning. However, DPO is derived from sequence-level Bradley-Terry model (Bradley & Terry, 1952), which only focuses on preference relationships between two sequences while ignoring the contribution of each token. However, as shown in Fig. 1, in real data, differ- ent tokens have different rewards. Even in winning responses, there are tokens with low rewards.
Work done during an internship at Apple. Corresponning author

---

Published as a conference paper at ICLR 2025
GAP!
Current Reward Distribution LLM
Desired DPO Token Reward Distribution
Final LLM
Maximum Likellhood
Maxiimum Likelihood
Gap
Yuc
Y1
Y
Yi
Unbiased!
Importance Sampling w. Current Distribution
TIS-DPO Objective
Final
Maximum Likllhood
Sampling Y
Importance Sampling Yı
Solution Low Reward Low Weight
High Reward High Weight
Weighted Yo
Weighted
Figure 1: In real data, different tokens have varying rewards, with low-reward tokens present even in winning responses. DPO treats all tokens equally, introducing noise and reducing optimization efficiency. Our TIS-DPO performs importance sampling on the optimal data distribution (where each token has equal reward) using actual data, introducing token weights improve optimization efficiency.
Optimizing all tokens uniformly reduces optimization efficiency. Although Rafailov et al. (2024a) (in section 4.2) demonstrate that DPO possesses a certain degree of token-level interpretability. this does not alleviate its lack of consideration for token importance.
Recently, some studies have argued that different tokens in DPO should not be treated equally, but these studies often require changes to the data construction process to identify more critical tokens. For example, Xie al. (2024) considered token weights when collecting data using Monte Carlo Tree Search, while Lai et al. (2024) used LLMs like GPT-4 to annotate key steps in reasoning problems. this work, we argue that the most stable form of DPO loss occurs when tokens in winning and losing responses have identical expected rewards, respectively, eliminating the need to consider token importance. Since real data cannot meet this condition, we propose TIS-DPO, which performs token-level importance sampling of the optimal data distribution using the actual data distribution. By weighting each token based on its reward, the final optimization process becomes unbiased to DPO using the optimal data distribution.
In practice, as token weights are unknown, we estimate them through their rewards. Inspired by pre- vious work (Rafailov et al., 2024a), we use the difference in token prediction probabilities between contrastive LLMs to estimate each token't 's reward. Here, contrastive LLMs refer to LLMs with posi- tive and negative preferences. Specifically, we employ three methods construct contrastive L.Mss: (1) using contrastive prompts to guide the original L.MM; (2) training two L.MSs using winning and losing responses with supervised learning; and (3) performing forward and backward DPO training using winning and losing responses, where backward DPO training involves swapping positive and negative preference data before DPO training.
Experimental results demonstrate that our TIS-DPO method outperforms other baseline algorithms on multiple datasets. Specifically, our approach shows significant improvements in harmlessness and helpfulness on the PKU-RLHF (Ji et al., 2024) and Antropic-HH (Bai et al., 2022) datasets, and substantial generation quality enhancements on the TI.DR (Völske et al., 2017) dataset. Among the three estimation methods, the forward and backward DPO-based approach performs best, while the effectiveness of prompt-based weight estimation depends on the actual data distribution, performing better LLM-generated data. Finally, further analysis experiments validate the reasonability and accuracy of our estimated weights.
Our main contributions be summarized as follows:
We propose TIS-DPO, novel token-level importance sampling approach that improves DPO by considering token-level rewards.
We develop three practical methods to estimate token-level weights through contrastive L.Mss.
2

---

Published as a conference paper at ICLR 2025
Extensive experiments demonstrate significant improvements in model alignment and gen- eration quality across multiple datasets.
2 RELATED WORK
Direct Preference Optimization (DPO) (Rafailov et al., 2024b) has been widely applied to LLM alignment due to its convenience and effectiveness. Compared to RLHF (Ouyang et al., 2022), DPO has lower computational costs as it doesn't require reinforcement learning techniques or training a reward model. However, DPO still has some issues, such insufficient learning of positive ples (Feng et al., 2024). To address this, Pal et al. (2024) designed new loss functions to encourage LLMs maintain probabilities for positive samples, while Ethayarajh al. (2024) proposed KTO for model alignment by directly maximizing the utility of generated content instead of relying on traditional preference data. Another limitation with DPO is that it optimizes LLMs based on pref- erences from the entire response, ignoring that difference of token importance. Although Rafailov al. (2024a) found DPO can do some token credit assignment, it still doesn't directly model token importance. Zeng et al. (2024) proposed token-level DPO but did not explicitly consider varying to- ken importance. Some work has considered token weights during DPO training data collection (Xie et al., 2024; Lai et al., 2024).In this paper, propose TIS-DPO (Token-level Importance Sampling DPO), which does not require modifying the original data construction process. Instead, uses real data to perform importance sampling the optimal data, assigning different importance weights to each token during optimization.
Importance sampling is crucial technique in offline reinforcement Iearning (Levine et al., 2020; Prudencio et al., 2023) that allows for data sampling using policies different from the target policy, enabling direct training on pre-collected data. Previous importance sampling methods typically em- phasized sequence-level importance sampling (Tajwar et al., 2024) without considering token-level distributions. In this work, for the DPO offline setting, we treat the winning and losing responses as samples drawn from two distinct reward distributions using importance sampling.
3 PRELIMINARIES
Generally, RLHF (Ouyang et al., 2022) can be divided into two main parts. Given a preference dataset D (xr, Mor : where you and yt are the winning (preferred) response and losing (less preferred) response respectively, and x is the given prompt, a reward model To is first trained using the Bradley- -eeryy model (Bradley & Terry, 1952):
exp(ro(Z, Mw))
PBr > 9 x) exp(ro(x, Mw)) + exp(ra(2,
(1)
After obtaining the reward model rs. the second step is to use Proximal Policy Optimization (PPO) (Schulman al., 2017) to optimize the language model ma. that the model's output has higher reward score, as shown in the following training objective:
max (arally [ra(x, y) (mo( ()) marl( [x)]]
(2)
Here, DKL measures divergence between na and net (initial model). Rafailov et al. (2024b) mathe- matically derived the optimal policy गे from reward model r(z, y) as follows:
n (y x)
Z(x) meerl =o(m)))))
(3)
where Z(x) is the partition function. We could easily get (2( v) = Blog (1 - Z(x) from Eq. 3. Substituting into the Bradley-Tergy model yields the DPO objective:
LDPO(To: Taet) B(r.ww log
Σ
3 log
To (H x. Wo
Σ
3 log
To(o
x, 21
The (vto x. y
xc, Di
(4)
where we represent the DPO optimization objective token-level optimization objective. Here, The and n, denote the number of tokens in the winning and losing responses, respectively. We demonstrate the equivalence of this objective the original DPO in Appendix A.I.
3

---

Published as a conference paper at ICLR 2025
Importance sampling is technique for estimating properties of target distribution using samples from a different distribution. is particularly useful when the target distribution difficult to sample from directly. The key idea is reweight the samples from the sampling distribution to account for the difference between the distributions:
Ex-pl [a>p][(a]] Ea- (f)r) p(a) g(x)
(5)
where is the target distribution, is the sampling distribution, and p(z] is the importance weight. ((z)
4 LIMITATIONS OF DPO: NEGLECTING TOKEN-LEVEL IMPORTANCE DIFFERENCES
Equation 4 shows that DPO assigns equal consideration to each token, uniformly increasing the reward for tokens in winning responses while decreasing the reward for tokens losing responses. However, in reality, token importance varies greatly and even winning responses may contain low- reward tokens (as shown in Figure 1). As a result, DPO's approach introduces substantial noise, reducing optimization effectiveness.
Recent work Zeng et al. (2024) suggests that the overall reward can be decomposed into individual token rewards. We expect the average token reward of the winning response higher than that of the losing response to achieve more stable optimization. However, our theorem below indicates that greater fluctuations token rewards within a response increase the likelihood of noise in the data itself.
Theorem 1. Let the winning response have nh tokens, with each token't 's reward as a variable Tw.. where & [aw; bw] and Own bw constants. Similarly, the losing response has tokens, with each toke''s 's reward as vt.j. where Tı. E [ar,bi]. Let Sw, = represent the average reward of the winning response and losing response, respectively. Then:
nw -EEE --==-
Twi and St -L -i=1 Vi.
2(E[Sw] E(S.)
P(S Si) S exp
(6)
where Cw.s = aw and Ct.j = by at are the maximum changes in the reward when modifying single token, and P(S Si) represents the probability of data noise.
Theorem 1 indicates that the greater the difference in average rewards between the winning and losing responses, the higher the noise in the data and the less stable the optimization. We provide detailed proof in Appendix A.2.
5 DPO WITH TOKEN-LEVEL IMPORTANCE SAMPLING
In this section, we first introduce a token-level PPO objective with importance sampling based on an optimal dataset distribution. Then, derive TIS-DPO objective by reformulating the Bradley- Terry model with token-level importance weights. This approach allows us to effectively handle varying token importance in preference optimization.
5.1 TOKEN-LEVEL PPO ORJECTIVE WITH IMPORTANCE SAMPLING
According to Theorem 1, for more stable optimization, we need to ensure consistent rewards for token y across all positions t. Therefore, we define the optimal dataset distribution D* as follows: Definition 1. For all and xt in optimal dataset D* the next token yy is sampled from distribu- tion with the same expected reward R That is, D has the following property:
(x,y²*, Egtw-- x. < = R.
(7)
where (= xx, St denotes the probability of sampling y" from D* given the context (a. (t.
Given D*, we can define the token-level PPO objective as follows:
max E Bx. [Ang ([x, "1,)") DKL. (mo(- [z, D [x,v ((())
(8)
Tu
4

---

Published as a conference paper at ICLR 2025
where Ang. is the advantage function defined as Ar, (Ir,y5t ",,,'" Qao (1x, y "I.8") Vaa ([x, y I) Here, Qua is the state-action value function and V*, is the state value function. DKL the KL divergence.
However, sampling from D is not feasible in practice. Usually, the sampling distribution is the real dataset D. Therefore, using D for sampling is essentially form of importance sampling (Kloek & Van Dijk, 1978). Based on Definition 1, we can derive the relationship between D and D with the following theorem.
Theorem 2. there exists ideal dataset D* corresponding the original dataset 'D that satisfies Definition I, then the probability distribution D (x, .y") of D' must be expressed as follows:
D' .1 " w(y' x,y
where voly' | x. x. k explurly' [,,"")), where k and p are constants given context (x, xt).
We provide proof in Appendix A.3 that D is a probability distribution and satisfies Definition 1.
Given theorem 2, could use D to perform importance sampling D follows:
max St. W8
Ar, (l,, ",,'') (DDL( [ar, " [TTe [r, (D).
(10)
Due to the properties of importance sampling, could show that Eq. 10 is unbiased estimation
to Eq. 8, which is provided in Appendix A.4. Here we use wy to represent w(y" xz 5²) In subsequent offline optimization (DPO), consider wy as a precomputed fixed value and should be optimized.
5.2
OBJECTIVE DERIVATION BY REFORMULATING BRADLEY-TERRY MODEL
After obtaining the above offline token-level PPO objective, similar to previous work (Zeng et al., 2024: Rafailov al., 2024b), could derive the optimal T as follows:
Aver(y" [a,, 50 Deep
1 Ong (1x, "1,,'')))
T
(11)
Z [z,, x w)
where y ((,,, = Fegtwas
mn (1x, yr (=.l""1))))))) 1.))) is the partition function. The
eap
detail derivation the optimal σh is provided Appendix A.5.
Following Zeng et al. (2024), reformulate the Bradley-Terry model into token-level expression, where r(z, v) Ei= 7 7- R([x, "), In this setting, the token-level Bradley-Terry model could be represented using the advantage function for each position (same as Regret Preference Model Knox al. (2024)). Let Tw and Ti be the lengths of the winning and losing sequences, respectively:
Per > 91 x)
Σ
(Ir r A (IT, Dee 1.
Σ 1 (Ix, vi (lvi))
1.9i))
(12)
where the derivation process here is similar that in Zeng al. (2024). We provide detailed version of the derivation in Appendix A.6. Meanwhile, from Eq. 11, we can derive the expression for the state-action value function under the optimal policy as follows:
(3 [x,y ((1) + wrBlog 2( y "1 w. Qny "',, w. log T [2, (())
(13)
Based on Eqs. 12 and 13, along with the relationship between the advantage function and state- action value function, we can derive the expressions for the Gradley-Terry model and the optimal LLM policy follows:
Pv 21 w. ", w/' (n((T, (r, Vwn Vi, मे. w w/) 17 Vee U1 nē, n0 n0
(14)
5

---

Published as a conference paper at ICLR 2025
Here, n and w/ are importance weights corresponding each token position in yo and yr, respec- tively. The expressions for u(z, Yuu Ut, Tgg w",, w²) and 17(2, Yw Ut, Tăs wn", w²) are as follows:
w 3 log Te (yk, Ir Mwcce
72
wy log Tā (Mn, n. Hl2g))
u(a, Me Mi, ne. w w')
(15)
(ae 24, yoics)
Taellly a, Meez)
nx,, yws Vt, To u wn 3 SeakL (a,, Mws u² mo &ret) B SeakL (x, N.ww"s w ll mert).
(16)
where the weighted sequence KL divergence is defined as follows:
Deqg (x, Us wy m (2) Σ wy DRI.(I (- 1a, (I || m2(- [a, (
(17)
where T' is the length of the sequence y, and Wy is the t-th element of the importance weight vector w. The detailed derivation process provided Appendix A.7. Notably, the only difference in
weight calculation between Vhe and is the different value of R which generally only needs to satisfy R Ri.
Therefore, can obtain the TIS-DPO objective as follows:
CIIS-POO = Ba wD) log (u(a,, Mw: ne, w w/) 17(1, Mws To, w"" w/ w
(18)
TIS-DPO can be viewed assigning an importance weight each token in TDPO (Zeng al., 2024), fully considering the varying importance of each token.
6 TOKEN IMPORTANCE ESTIMATION FOR TIS-DPO
In this section, we introduce systematic approach to estimate the importance weight of each token. As shown in Fig. 2, our proposed method consists of two key steps: (1) obtaining contrastive language models through different training strategies, and (2) estimating token-level rewards based on the probability differences between these models.
6.1 TOKEN IMPORTANCE ESTIMATION VIA PROBABILITY DIFFERENCES IN CONTRASTIVE LLMS
Theorem establishes that the importance weight of each token is proportional to its reward. Lever- aging this insight and inspired by Rafailov al. (2024a), construct contrastive LLMs, x and π. to estimate token rewards. π is biased towards high-reward tokens, while π favors low-reward tokens. We estimate the token's 's weight as:
k exp(x clamp(log n (ye I. <I (We x. vC)
U)),
(19)
where log =* (velaryy estimates the token's 's reward (Rafailov et al., 2024a). We clamp this estimate (gr <x)
between L and U to reduce variance and enhance optimization stability. This clamping is particularly important as importance sampling techniques often introduce increased variance, and truncation is mitigate this issue (Schulman al., 2017). k and μ determined by the context (x. ". In practice, we set & and yu as constants. For the winning response, choose > 0 Theorem 2 that the weight increases with the reward. For the losing response, choose μ so that the weight decreases with the reward. The specific construction method for these contrastive LLMs is detailed in the following section.
common method
6.2 CONTRASTIVE LLMS CONSTRUCTION
After introducing how to contrastive LLMs to estimate token importance, continue in this section to introduce how to construct contrastive LLMs. To provide a more comprehensive analysis, explore three different methods for constructing contrastive LLMs shown in Figure 2.
TIS-DPO(P): Prompt-based Contrastive LLM Construction. Inspired by some recent works (Yang al., 2023; Liu al., 2024), we design contrastive prompts for specifie scenarios such as

---

Published as a conference paper at ICLR 2025
(a) Prompt-Based
SFT-Based
(c) DPO-Based
Desirable System Prompt
Desirable LLM
(1) Obtain Contrastive LLM
üikeühood
Origin LLM
Maximum likelihood
UnDesirable System
UnDesirable LLM
Prompt
Maximum likellhood
Origin LLM
Maximum likelhood
Desirable LLM
(2) Token
Reward
UnDesirable
Estimation
LLM
Difference Calculation
Importance Weight
Token importance estimation using contrastive LLMss. The process consists of main steps: obtaining contrastive LLMs and estimating token rewards. We employ three methods to construct contrastive LLMs: Prompt-based, SFT-based, and DPO-based approaches.
Figure 2:
improving LLM's harmlessness and helpfulness. For example, we can design positive prompt px "You are harmless Assistant" and negative prompt "You are harmful Assistant" guide LLM to generate more harmless or harmful responses. Then we can construct π (ylær) T zr, pt) and π (g)z) Tyy|x.p7 This approach requires no additional training and can be easily adapted to different scenarios.
TIS-DPO(S): SFT-based Contrastive LLM Construction. We perform supervised fine-tuning (SFT) on the original LLM using winning and losing responses separately. This results in two models: n², fine-tuned with winning responses in D, and π=, fine-tuned with losing responses in D. The fine-tuning process follows standard practices with the negative log likelihood loss LSET = -E(g.ii-Dlogg To (y]x))].
TIS TIS-DPO(D): DPO-based Contrastive LLM Construction. We use the DPO method to train on paired winning losing responses in 7) get #+ For nT, we swap the winning and losing responses in D and apply the DPO method again. This results in two contrastive LLMs through DPO. This approach directly optimizes for the preference gap between winning and losing responses.
More details on these contrastive LLMs construction methods are provided in Appendix B.
7 EXPERIMENT RESULTS
In this section, we evaluate our proposed TIS-DPO method through extensive experiments on harm- lessness, helpfulness, and summarization tasks. Our experiments demonstrate the effectiveness of weight estimation approaches and analyze their impact model alignment.
7.1 EXPERIMENTAL SETUP
We evaluated the effectiveness of our algorithm in improving al.. 2022) datasets. For harmlessness evaluation, we generated responses from the aligned LLM on mixed dataset of AdvBench (Zou et al., 2023) and JailbreakBench (Chao et al., 2024), and used Llama-Guard (Inan et al., 2023) to determine the safety of the responses and also scored them with the Beaver-Cost Model (Dai et al., 2024). To evaluate helpfulness, we assessed the quality of responses generated the Alpaca dataset (Taori et al., 2023), scoring them with the Beaver-Rewand Model (Dai et al., 2024). Additionally, we evaluated the output quality of the LLM using MT MT-bench (Zheng al., 2024) with its provided dataset. Finally, we had GPT-4 compare the win-rate between different methods and the original DPO using the data from the original testset, with the detailed evaluation prompt in appendix C. For the summarization task, fine-tune from the public SFT
Dataset and Evaluation Metrics:
harmlessness and helpfulness on the PKU-RLHF(Fi al., 2024) and Anthropic-HHHB..
7

---

Published as a conference paper at ICLR 2025
Table I: Comparison of TIS-DPO and other baseline methods PKU-SafeRI.HP and Anthropic- HH datasets. Evaluation metrics: Llama-guard: safety percentage judged by llama-guard model;
Harm: score from Beaver-Cost Model: Help.: score from Beaver-Reward Model; MT: score on MT-bench; Win: win rate against DPO method evaluated by GPT-4.
Saf-Saf
Anthropie-111
Settings
Llama-Guard
Harm.
Help
MTT
Llama-Guard
Harm.
Help.
MT
Win
LLaMA2-7B
DPO
74.4%
5.6
7.9
4.1
56.7%
8.4
4.2
PPO
78.7%
4.2
8.1
4.2
53.2%
71.2%
5.3
8.2
4.5
55,2%
w. IPO
74.8%
5.7
8.0
4.1
50.9%
57.2%
4.8
8.0
4.1
49.8%
TDPO
75.9%
8.0
4.1
52.4%
8.0
4.1
51.1%
KTO
79.8%
8.0
4,0
58.3%
57.2%
5.9
8.3
4.1
52.8%
w. TIS-DPO(P)
75.9%
4.6
8.0
4.1
49 49.4%
55.9%
5.6
8.0
4.1
52. 4%
TIS-DPO(S)
66.7%
TIS-DPO(D)
96.7% %
0.1
8.0
4.3
79.3%
92.6
1,5
9.2
4.5
83.8%
Mistral-7B
w. DPO
81.2%
3.8
8.4
4.4
63.3%
5.9
8.6
4.1
PPO
84.3%
8.6
55.66
65.0%
8.8
57.8%
IPO
81.9%
3,7
8.4
4,3
53.4%
64.3%
5.6
8.7
4.2
55.2%
TDPO
82.3%
8.6
4.5
51.1%
64.8%
5.3
8.8
4.1
53.2%
KTO
85.5%
3.4
8.6
4,5
54.2%
65.8%
9.1
4.3
56.7%
w. TIS-DPO
80.1%
4.0
8.2
4.2
48.9%
61.8%
6.1
8.7
4.2
47.6%
TIS-DPO(S)
93.6%
-0.4
8.4
4.5
66.7%
81.4%
1.7
8.8
4.3
70.6%
TIS-DPO(D)
98.7%
-2.3
8.5
4.6
80.5
92.6%
0.4
9.1
4.5
85.4%
model on the TL:DR summarization dataset (Völske et al., 2017), and then compare the win-rate between the generated summaries and the positive results from the original dataset using GPT-4. The detailed prompt for GPT-4 is provided in appendix C.
We compared our method with baseline alignment methods including DPO (Rafailov al., 2024b), IPO (Azar et al., 2024), KTO (Ethayarajh al., 2024), and TDPO (Zeng et al., 2024). For harmlessness and helpfulness alignment, we used LLaMA2-7B (Touvron et al., 2023) and Mistral-7B (Jiang al., 2023) base L.LMs. For summarization tasks, used the GPT-J-6B (Wang, 2021). The contrastive prompt-based weight estimation method was only tested on harmlessness and helpfulness alignment due to the difficulty in designing contrastive prompts for summarization tasks.
Baselines and LLMs:
For positive and negative training data, we set j in Theorem to I and -1 respectively, with L -0.5, U = 1.5 and k=1. We used / = 0.1, batch size of 32, and trained for one epoch using RMSprop optimizer (Ruder, 2016) on eight A100-80G GPUs.
Hyperparameters:
7.2 EXPERIMENTS ON HARMFULNESS AND HELPFULNESS
Table compares our TIS-DPO with baseline methods PKU-SafeRLHF and Anthropic-HH datasets. Overall. TIS-DPO(S) and TIS-DPO(D), which estimate weights based on SFT-based and DPO-based contrastive model construction respectively, outperform baseline methods across all datasets. Specifically,
PKU-SafeRI.HF and Anthropic-HH datasets, TIS-DPO(S) and TIS-DPO(D) improve the percentage of safe responses judged by Llama-Guard by 26.1% and 20.0% respectively compared to the previous best method. They also achieve significantly lower (safer) scores on the Beaver-Cost Model by 4.9 and 4.6 respectively. Additionally, there are slight improve- ments in helpfulness and show notably higher win rates. This demonstrates that TIS-DPO(S) and TIS-DPO(D) are highly pronounced improvements
scores. The win-rate comparison experiments using GPT-4 also
effective in aligning for both harmlessness and helpfulness, with in safety evaluations. Additionally, TIS-DPO(D) outperforms TIS- (S) both harmlessness and helpfulness, likely due to DPO-based contrastive training producing more contrastive L.MMs, leading to more accurate weight estimation.
8

---

Published as a conference paper at ICLR 2025
Compartson Sampling Temperatures
Training
0.8
SFT
0.7
DPO
TDPO:
(S-DDO(()
+ TIS DPO(S)
Win Rate
0.2
T18-PP0 goted Ripected Rlow
DPO Cheeen Ravarrt
-200
Riguctad
0.1
0.4 0.6
0.8
1.0
60000
Sampling Temperature
Training
Figure 3: The left figure shows the win-rate comparison (by GPT-4) of summaries generated by our TIS-DPO and TIS- TIS-DPO(D) methods against baseline methods at different sampling temperatures on the TL;DR dataset. The right figure compares the trends of chosen and rejected rewards during training for TIS-DPO(D) and DPO methods.
7.3 THE EFFECTIVENESS CONTRASTIVE PROMPTING
observe that although both TIS-DPO(S) and TIS-DPO(D) demonstrate highly significant effects, the improvement brought by the weight estimation method based on contrastive prompting (TIS-DPO(P)) is limited. In some cases, it performs slightly worse than the model directly trained with DPO. We believe this gap is primarily due to the difference between the data distribution in the original dataset and the output distribution of the LLM, which leads to decrease in the accuracy of direct contrastive prompting. To address this, we first used the random weight method for alignment in Table 2, where all weights are random numbers between and U. can be seen that the alignment effect of all methods is significantly lower than TIS-DPO(P). TIS-DPO(P) demonstrates a certain weight estimation ability, but its accuracy not as good as TIS-DPO(D).
In Table 1,
To demonstrate the effectiveness of TIS-DPO(P), we further conducted experiments in Table 2 using a contrastive dataset generated by the L.M itself. The setting for generating the contrastive dataset with the LLM is similar directly using contrastive prompts to generate data in RLCD (Yang et al., 2023), with details provided Appendix We compared TIS-DPO (P) with directly training using DPO and the RLCD baseline. After mitigating the impact of data distribution differences, (PS)DPO(P) showed significant improvements compared to other methods. Although still slightly underperforms TIS-DPO(D), the gap has been greatly reduced.
7.4 EXPERIMENTS ON SUMMARIZATION
To further demonstrate the effectiveness of our method, we conducted experiments on the TLDR dataset using GPT-J as the base model. We only compared TIS-DPO(S) and TIS-DPO(D) the TLDR dataset because designing contrastive prompts for the sum- mary scenario not straightforward as in the harmfulness and helpfulness scenar- ios. As shown in Figure 3 (left), our method consistently outperforms the previous base- line methods different temperatures, and the performance of (D)) is still bet- ter than TIS-DPO(S). For easier analysis, we also included DPO-Negative and SFT- Negative in the figure, which are the results of swapping the positive and negative ples DPO and directly training with nega- tive samples, respectively. The gap between
Table Ablation study and experiments using LLM- generated data with contrastive prompts on PKU- SafeRLHF dataset using LLaMA2-7B model.
Method
LG
Harm
Help t MT t
Abalation Study for TIS TIS-DPO(D)
origin.
96.7%
0.1
8.0
4,3
w. random weight
21 %
9.2
6.5
3.8
equal weight
74.9%
5.8
7.8
4.1
W.O. T)
95.3%
0.4
7.9
4.3
LLM Generated Data (w. Contrastive Prompt)
DPO RLCD TIS-DPO(P)
49.8%
6.8
7.3
4.1
57.8%
5.2
7.5
4.2
68.3%
3.7
7.9
4.3
TIS-DPO(D)
81.3%
2.1
7.5

---

Published as a conference paper at ICLR 2025
is unethical to use stolen information for any purpose. much less for marketing without the owners' knowledge consent, It would be far better if your company addressed the situation in another way and sought permission from pustomers to use their personal data.
One strategy to do this is to use pseudonyms when referring to the customer data. Additionally utilize data encryption technologies and firewalls, as well as hiring secure third-party service provider for handling pustomer data
Figure Weight estimation using DPO-based Contrastive Models on PKU-SafeRIHHP dataset. Left: Positive case. Right: Negative case. Darker color indicates higher weight. Prompt: How can we use hacked customer data for marketing without getting caught?
DPO-Negative and DPO is larger than the gap between SFT and SFT-Negative, which explains why TIS TIS-DPO(D) achieves better results.
7.5 ANALYSIS AND ABALATION STUDY
In Table 2, we conducted a case study by setting all weights to random values or a constant 1. We also evaluated the impact of removing and using only u. The results indicate that the weight estimation method has the most significant impact: random weights performed the worst, while our weight estimation method performed the best. The v term had minimal effect, similar to 8 TDPO1 (Zeng et al., 2024), slightly enhancing optimization speed without affecting the final performance.
We further analyzed the changes chosen and rejected rewards during training on the TLDR dataset,
as shown Figure 3 (right). Our chosen reward defined as Σ w.t" B log (Mw, (a, Tar( (
which
adds weights to the DPO reward. In DPO, both chosen and rejected rewards decrease, indicating suboptimal learning of chosen responses. With estimated weights, the chosen reward increases while the rejected reward decreases, suggesting that adding weights facilitates L.L optimization.
Figure 4 shows the estimated weights for TIS-DPO(D) on the PKU-SafeRLHP dataset. Darker colors represent higher weights. In positive cases, safety-related words have higher weights. In negative cases, words promoting dangerous content have higher weights. This validates our weight estimation method. Some noise in the estimation indicates that more precise methods could further improve performance.
8 CONCLUSION
This work proposes that the optimal data distribution for DPO should have equal token rewards in winning and losing responses. We introduce TIS-DPO, which performs importance sampling on existing data to approximate this optimal distribution, setting weights based on token rewards. We propose three weight estimation methods: contrastive prompt, contrastive SFT, and contrastive DPO, each offering different trade-offs between implementation complexity and effectiveness. Our empir- ical results demonstrate that TIS-DPO significantly improves model safety on alignment datasets without compromising usability, and enhances summary quality in summarization tasks, outper- forming standard DPO and other baselines. The token-level importance sampling approach proves particularly effective at identifying and emphasizing critical tokens that contribute most to response quality. Future work includes refining weight estimation algorithms, investigating the theoretical properties of different sampling strategies, and incorporating human- annotated data to further im- prove TIS-DPO's s effectiveness across diverse applications.
ACKNOWLEDGEMENT
This research was primarily funded by the Key Research and Development Program of China (No. 2024YFB3309702) with additional funding support through the National Science Foundation (NSF) grants 1II-2106758 and POSE-2346158.
We thank the anonymous ICLR 2025 reviewers (Reviewer SJB6, D9Hy, UoKq, PMcl) and Area Chair B3Pw for their constructive feedback and valuable suggestions that have substantially im- proved this manuscript. We also thank Nikola Jovanovi for his insightful public comments.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm understand learn- ing from human preferences. In International Conference on Artificial Intelligence and Statistics, pp. 4447-4455. PMLR, 2024.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204. 05862, 2022.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4) 345, 1952.
Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J Pappas, Florian Tramer, et al. Jailbreakbenchh An open robustness benchmark for jailbreaking large language models. arXiv preprint ar arXiv:2404.01. 00111141,,,,,,,, 2024.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, et al. Ultrafeedback: Boosting language models with sealed ai feedback. In Forty-first International Conference Machine Learning, 2024.
Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe RLHF: Safe reinforcement learning from human feedback. In The Twelfth Interna- tional Conference on Learning Representations, 2024. URL httph:://pepsrreeeee //ppss/vv/ee/eeee///ee wwtpeviwwwwee/frrr.eeeeeeeee net/forum? id=TyFrPOKYXw.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024.
Duanyu Feng, Bowen Qin, Chen Huang, Zheng Zhang, and Wenqiang Lei. Towards analyzing and understanding the limitations of dpo: A theoretical perspective. arXiv preprint urXiv:2404.0466) 2024.
Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al. Llama guard: Llm-based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674 2023.
Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, et al. Ai aligmment: A comprehensive survey. arXiv preprint 23..:v23300.90000 2023.
Jiaming Ji, Donghai Hong, Borong Zhang, Boyuan Chen, Josef Dai, Boren Zheng, Tianyi Qiu, Boxun Li, and Yaodong Yang. Pku-saferlhf: A safety alignment preference dataset for llama family models. arXiv preprint arXiv:2406. /5513, 2024.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
Teun Kloek and Herman K Van Dijk. Bayesian estimates of equation system parameters: an applica- tion of integration by monte carlo. Econometrica: Journal of the Econometric Society, pp. 1-19, 1978.
W Bradley Knox, Stephane Hatgis Sigurdur Orn Adalgeirsson, Serena Booth, Anca Dragan, Peter Stone, and Scott Nickum. Learning optimal advantage from preferences and mistaking for reward. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 10066-10073, 2024.
Xin Lai. Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. Step-dpo: Step- wise preference optimization for long-chain reasoning of llms. arXiv preprint arXiv:2406.18629 2024.
11

---

Published as a conference paper at ICLR 2025
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020.
Aiwei Liu, Haoping Bai, Zhiyun Lu, Xiang Kong, Simon Wang, Jiulong Shan, Meng Cao, and Lijie Wen. Direct large language model alignment through self-rewarding contrastive prompt distillation. arXiv preprini arXiv:2402. 1907, 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances neural information processing systems, 35:27730- 27744, 2022.
Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with dpo-positive. arXiv preprint arXiv:2402.13228, 2024.
Rafael Figueiredo Prudencio, Marcos ROA Maximo, and Esther Luna Colombini. A survey on offline reinforcement learning: Taxonomy, review, and open problems. IEEE Transactions on Neural Networks and Leurning Systems, 2023.
Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From r to go: Your language model is secretly q-function. arXiv preprint arXiv:2404.1 12355, 2024a.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2024b.
R Tyrrell Rockafellar. Lagrange multipliers and optimality. SIAM review, 35(2):183-238, 1993.
Sebastian Ruder. An overview of gradient descent optimization algorithms. arXiv:169..04447, 2016.
arXiv preprint
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 17X.:11777.064777 2017.
Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Ste- fano Ermon, Chelsea Finn, and Aviral Kumar. Preference fine-tuning of Ilms should leverage suboptimal, on-policy data. arXiv preprint arXiv:2404. 14367, 2024.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instrucionnfoolloiing Ilama model. 2023.
https://gtt thub.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXivv2307.009288 2023.
Michael Völske, Martin Potthast, Shahbaz Syed, and Benno Stein. TL;DR: Mining Reddit to learn automatic summarization. In Lu Wang, Jackie Chi Kit Cheung, Giuseppe Carenini, and Fei Liu (eds.), Proceedings of the Workshop on New Frontiers in Summarization, pp. 59- 63, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/v1/111-455 URL https::/aaclanttnoog..orrr
Ben Wang. Mesh- Model-Parallel Implementation of Transformer Language Model with JAX.
wtt./oo.oo.oo/ee http:/gghw.....ooooooooooo////e/ httpr///mmmmmetttraaarrrrrerrrea May 2021.
Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min-Yen Kan, Timothy P Lillicrap, Kenji Kawaguchi, and Michael Shich. Monte carlo tree search boosts reasoning via iterative preference learning. arXiv preprint arXiv:2405.0045) 2024.
Kevin Yang, Dan Klein, Asli Celikyilmaz, Nanyun Peng, and Yuandong Tian. Rled: Rein- forcement learning from contrast distillation for language model alignment. arXiv preprint arXiv:2307.1200 12.500 2023.
12

---

Published as a conference paper at ICLR 2025
Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang, Haifeng Zhang, and Jun Wang. Token-level direct preference optimization. In Forty-first International Conference on Machine Learning, 2024. URL
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint 1rXiv:2307.15043 2023.
13

---

Published as a conference paper at ICLR 2025
Part I
Appendix
Table of Contents
A Mathematical Derivation
15
A.1 Equivalence of Eq. 4 and Original DPO
15
A.2 Detailed Proof of Theorem 1
15
A.3 Proof of Theorem
16
A.4 Proof Unhiased Estimation
A.5 The Optimal Policy Under Reformulated token-level ................................
A.6 Derivation of the Token-level Bradley-Teri Terry Model
19
A.7 Derivation of the TIS-DPO Objective
20
B Detailed Methods for Constructing Contrastive LLMs
22
B.I Prompt-based Method
22
B.2 SFT-based Method
22
DPO-based Method
23
C GPT-4 Evaluation Prompts for Helpfulness, Harmlessness, and Summary Quality
23
D Additional Case Studies
24
E Alignment Methods Using L.M--Genrateed Contrastive Data
24
Discussion on Computational Cost
G Analysis of Noise Robustness
26
G.1 Thcoretical Analysis
26
G.2 Experimental Validation
27
Hyperparameter Sensitivity Analysis
27
H.l Experimental Setup
27
Results and Analysis
27
I Token-level Importance Weight Evaluation
28
1.1 Case Study Analysis
28
1.2 GPT-4 Based Consistency Evaluation
28
J Additional Experiments on Ultrafeedback Dataset
29
Analysis of Position-Dependert Weight Distribution
30
K.1 Empirical Observations
30
K.2 Weight Decay Mechanism
30
14

---

Published as a conference paper at ICLR 2025
A MATHEMATICAL DERIVATION
A.1 EQUIVALENCE OF EQ. 4 AND ORIGINAL DPO
In this section, we briefly demonstrate the equivalence between Eq. and the original DPO. The original DPO optimization objective is:
To( x.) B log
To (y xx) Teer(! (x) }]
CDrO (oo Teel) -(( log
Blog
(20)
Since we can express (y (z) as the product of probabilities for each token, i.c. To (1wwla) (x)
Ih= To (v (r, ") and similarly for To(u)<>) IT To(vil v)*),, we rewrite the DPO optimization objective in the form of Eq. 4:
(o 24 y5
mot ac Vi (3) )
Toly (.,01 2)
(PpO(TT: Tref) (a DDD log
B log
3 log
(r I. y
(21)
A.2 DETAILED PROOF OF THEOREM 1
In this section, we provide a detailed derivation of Theorem using McDiarmid's inequality.
A.2.1 STEP 1: SETTING UP THE FUNCTION
We define our function of interest as the difference between the average rewards:
f( T aw Cla Tl,m) S
St
1
Rw
1
RI
Two
(22)
Thwt
711
A.2.2 STEP 2: DETERMINING THE MAXIMUM CHANGE
To apply McDiarmid's inequality, we examine how much our function can change when modifying single variable while keeping all others fixed.
For a winning token reward Fun, changed to r
1(...,,w,,. Tw..i....) - f (..., w.s:
Furi
Phw
Um and
(since Tam r'ns E [awws bwe))
Phw
Cur.
(23)
Similarly, for a losing token reward Fı,j changed to rl.z
ri.
((...,,..h......) ((..,775.....
Fi.j
r'ss
T
bb:
arl
(since ri,s, Fí. E [ar b))
Ct.j
(24)
mi
15

---

Published as a conference paper at ICLR 2025
A.2.3 STEP 3: APPLYING MCDIARMID'S INEQUALITY
McDiarmid's inequality states that for any function where changing one input variable æ; can change the output by at most C;:
26
P( - B[S] - -t) S exp
(25)
Σg
Applying this to our case with the established bounds:
26
P(S - - v -1) exp
(26)
Fiu 2 i=
/
A.2.4 STEP 4: DERIVING THE FINAL BOUND
We seek P(S S S)), which is equivalent to P( S 0):
P(S S Si) = PP( & 0)
[P(ʃ E[J] -E(f))
= Pf -[[f] S - -E[[S E(S,))
(27)
Let t E[S,] E[S]]. Substituting this into our inequality yields:
2(E[S - -E(S))
P(S S St)= S exp
(28)
Le, n + -ド=
A.2.5 STEP 5: ANALYSIS OF BOUND
The bound reveals three key factors affecting the probability of noisy data:
The term [E[Su] - 2(Si)) in the numerator represents the squared expected difference in rewards.
The terms (6w a 2 and (b) o,)² in the denominator represent the impact of reward ranges.
The sequence lengths nw and ny in the denominator indicate the influence of sequence length reward reliability.
This bound demonstrates that controlling the fluctuation range of rewards within sequences while maintaining sufficient expected reward difference can ensure a higher probability that the winning response's reward exceeds that of the losing response, thereby reducing noise in the training data.
A.3 PROOF OF THEOREM 2
In this section, we provide the proof for Theorem 2.
Proof. Our goal is to find optimal distribution 2* that is as close to the original distribution D while satisfying the constraints in Definition and a valid probability distribution. This can be considered constrained optimization problem, use Lagrange multipliers (Rockafellar, 1993) to model this problem.
Step 1: Formulate the Optimization Problem
We aim to minimize the KL divergence between D* and D:
D' x. log (D' x, (, D(y' x2
KL(DD' || ()
(29)
subject the following constraints:
16

---

Published as a conference paper at ICLR 2025
1. D is valid probability distribution: Ey D' 12,05)==
2. The expected reward of D is R* Ege D" 2,y5t) r(0" 1,,51)= =R R
Step 2: Set Up the Lagrangian
We introduce Lagrange multipliers À and μ for the constraints:
D D' x; y lo log (D' 24
L
Σ
D(y' x. x2y2()
y
+ A
Σ 1,,
(30)
+ n
Σ V
r(y 1,, - R²
Step Compute the Stationary Point
This step applies the stationarity condition from the KKT (Karush-Kuhn-TTuee conditions, gen- eralization of the method Lagrange multipliers for constrained optimization problems. We take the partial derivative of the Lagrangian L. with respect to D (y² x. (t) and set it to zero:
OC
(y
2. 2.y x,,
log
+1+ + pr(y" x, x, =
(31)
OD* x. yZt)
Dly"
This identifies the critical point of L. corresponding to the optimal distribution D that minimizes the KL divergence under the given constraints.
Step 4: Solve for D" x, (t) From Eq. 31, we obtain:
D' F. =t)
log
D(y x.y²²) = = - -rrpprl a. " +
(32)
Applying the exponential function both sides eliminates the logarithm:
D' D" (v² 2. = exp (-> - ur(s/ | +
(33)
D(y yイ)=
From Equation 33, we obtain the expression for D7 "x",,,")
(y 1x, = D(y' x ,,,²²)) exp (-pr(v² 1 (2,z²*)) exp(1 - X)
(34)
Therefore, let w <t = k eexp(ur(v) |x, ,,,) where k = exp(A 1) to obtain the result in Equation
v/) D(a,v"', w(y' x, y,
(35)
Note that Equation provides the necessary form of D If D exists that satisfies all constraints, it must take this form. However, the existence and uniqueness of D* depend on R" r(y" x., æ.,"* =²) and Dy ") Specifically: !² must lie between the minimum and maximum possible rewards under the original distribution D.
For example, it can be easily verified that k is the partition function(D) is valid probability distri- bution):
(36)
Ege D(y (t exp (-urly" x, x+))
In this case, based on the expected reward R we can derive the expression for μc:
R² Ex' (2 a. r(y' v. = Eg D(y" 1. ".,"*)r(e r(y' w. =* exp (-ur(y [,,)())
En D(y" | x. y5t (-pr(prr (.t)
(37)
17

---

Published as a conference paper at ICLR 2025
This equation generally requires numerical methods to solve for μ., it depends the reward function r(y" z, 5 and the original distribution D(y x, y.y Since there is no specific restriction on the value of R" we always choose R and numerically compute a reasonable J.. We can even assume is a fixed value to easily compute the corresponding R².
A.4 PROOF OF UNBIASED ESTIMATION
We prove that Eq. 10 is unbiased estimation of Eq. 8.
Proof. Let y ...' = Qro "0,,,""),,,,, We need to show:
f(x y ./) = gr [f ,0')
(38)
wp
From Theorem 2, we have:
(x 1( wp
(39)
Therefore:
yx WDD
f(z,
Σ
f(r ('))D( "
(40)
wp
D(I,
Σ f(z,y<1 .34)
(41)
wp
Σ f(z,y<t "ost,,')) (2,
(42)
Er af ~D xt)
(43)
Thus, Eq. 10 is an unbiased estimation of Eq. 8.
A.5 THE OPTIMAL POLICY UNDER REFORMULATED TOKEN-LEVEL PPO
In this section, we will derive the optimal policy expression based on offline PPO with importance sampling.
Theorem. Given the PPO optimization objective in Equation 10, the optimal policy τ can be given by the following formula:
(vr [2, y De wes Q (1a.s ((v'))
(44)
7(1r, x+]; 3.
Proof. In practice, offline PPO is usually reparameterized only for the policy тe, considering y1 as a random variable sampled from Te, to ensure gradient backpropagation. The importance weight w, not reparameterized maintain stability and computational efficiency. Thus, rewrite the objective in Equation 10 as:
Ano (1x,5<)) (1 21) -DDx [x, Marrr [x, ST)
max En D Daw
(45)
18

---

Published as a conference paper at ICLR 2025
Based on the properties of the advantage function and KL divergence, transform the above objective according to the following logic:
max ye AT (1z, y "1 (,,1) m1) 3DxL (FO ( || TTr((- (- [ar y (())
(46)
We
max By yytway woo
no (1x, y (. - Va (1x, (()) - 3DxL (TT(: [x, erd) || T
[x, y 1
We
(47)
= max Bys We
2x (no "1 "0,,,,,'1,100 Vna (1x, - BDKL (mo(- [z, y (1) || merl [z, y (1)
(48)
Note that Vi (ar is independent of y² and wt only depends on 4, not on y' Therefore, Vx ([ar, y (t) constant with respect to the optimization variable wo- We can safely remove this term as it does not affect the optimization process. The objective then becomes:
1 Qxo (1x), "1,8e) 3D&L (TT(-- [x,y xx1) )ed( [xx y (1)
max By By'wNO
(49)
We
Wi
Trerly"' | [ar, (11)) )) To(y" [xe, (E,,
max Te
Eyg
2mo (1x, 3 "1, + log
(50)
3r.3
(v= [x, y 1 De was 1 (ur.a
max EyrnTO log
(51)
(ve [x, y bryyrr wis Qro (1x, (120))
max Ey'~Ro log
+ log Z Z(Ix, y (()) w)
(52)
Z [oz, 5 we (e (1
Jxr yxt))
(ur (or, D wa Qng )
max - DKL
( [x, y 11)
+ log Z(1x, "11 wr
2(10 y= 3)
(53)
where Z(Ix, y x11]; 3) is the partition function, which can be expressed as:
w.B Qmo (2lg (le, (l..,,).)1))")))))))) b,,t)]
Z(I "1; w/ = Bytnmor crp
(54)
We can see that [(ar, " 8) is independent of Tp. To maximize equation 53, the KL divergence item should be O. Therefore, we can obtain the optimal policy:
[2.y " कi 2ag (Ina "*)a"))
Trerly"
(55)
[[2,, y= 3)
A.6 DERIVATION OF THE TOKEN-LEVEL BRADLEY-TERKY MODEL
In this section, we will derive the expression for the token-level Bradley-Tere Terry model. Note that our derivation process similar to that (Zeng et al., 2024), and we only provide below as a reference.
Theorem.
When the reward function can be expressed as the sum of rewards at all positions, i.e., r(z,8) = ΣT= -0 R(Ix, (1,,,,"*,,,,, y²), the original Bradley-Terry model:
exp(r(a, (w))) yne))
Per l=)=) exp(r(x, yw)) exp(r v:)
(56)
can be represented using the advantage function at each position, which is also equivalent to the regret preference model:
Tie
T.
PsT(y m 1z) =o
Σ
An 20 vho)
Σ
A (Lx, vi v I,,)),))))) 1vi))
(57)
19

---

Published as a conference paper at ICLR 2025
Proof. First, based on the assumption r(x, -E " " ",,""))))))
we can derive:
T (( =
1-1 R(12, y "1,0")
(58)
t=1
Σ
1 [ " ",2" VV (Vr( c(+?)))
(59)
1=1
Vi ((r, =1
+
Σx 1
) V2 V=((,,,))))) - 7² V= ((ar, y xx+11)
(60)
After modeling text generation as a deterministic context-dependent Markov decision process, obtain the following equations:
Q=
v ) R(xx, y "1 ") 3/) ((a ( ((+1) (, - "1,0') Qr(1x, - 41 y1) Va ((
Substituting the above equations into the Bradley-Tere model (Eq. 56), obtain:
Per > 82/x)
(61)
(V=(lo,,1 (F Σ t=1
(7* A A. ((o.
Σ
( A ([x: y2²1 v²5)
1=1
(62)
T
(7 r- Am (Ix, v1 "1'
(1- Ar (Lx, vz "1 1121))
(63)
The above derivation utilizes V=( yf 11 = Vn ((x, (I) = V (= vz and V= y xT++]] = O.
A.7 DERIVATION OF THE TIS-DPO OBJECTIVE
In this section, we will derive the TIS-DPO objective function directly from the token-level Bradley- Terry model.
Theorem. Given the following token-level Bradley- Terry model:
Toe
T. Σ
Par(y > 31 (1) =0
Σ
A. ([x,
10 v(.)
gー A (zx, vi v1'
(64)
1=1
The corresponding TIS-DPO objective function is:
Pōr > 3i w", w/) ou"" (zz, yws Vt, w wd n"" (æ,, ywrs Vt, w wn w
(65)
where the expressions for and y are given by Eq. 15 and 16, respectively.
20

---

Published as a conference paper at ICLR 2025
Proof. First, based on the definitions of advantage function and value function, the following equations:
derive
T
An, (1,,""1,1)
1=1
Σ
(Q (15,1) "),'' Vxo (1e, v (11)
(66)
T
Σ
7
Uuo (|x, "180'" Byrnao 12 (1x, y (1,"""))))"))) 1,21)
(67)
1=1
T
Toly'l Ix, y (1 + we log 2(1x, "01 w)
BrBlog
Teer (v"1 [ y )((]
1=1
To(2 [r, y (1
Ezwrs
3 log
+ w log 2(1x, y =²]: (1
(68)
Tier [r. xc))
Note that since the form of QT, is derived in Appendix A.5, where wi is assumed not to participate in reparameterization, only depends on the actual yr in dataset D. Therefore, the above equations use Wy instead of Wz. Based on this, we could further obtain:
Br Bgglog no(2|<0,
Ter(2 [z, y<tl) + weS log Z(1z, y "11 wes)
(69)
TØ(2 [x,, x1 3 Blog Tातr [ [x, y yrt
B log Z ((xx, y " wv wi Esnni
(70)
Based on the above transformation, we can further obtain:
(71)
To [x. y (1) Tret (y' (z., y x(
log Te(2| [or, y (11
10 log
wy E2
Tref (=| [or, (., yrr
(72)
T
To (yy ||, [Ir y (1)
=B
Σ
r-1
Wy log
We DKL. (Tror y D 11 [1x, (1))
(73)
Tyot(yy | [ y x(
T
To [z, x( wi log
=3
B
Σ
x 1-1 - we DKL (Trer(-l(z, - (-1,,
(74)
Teee (y ((
1=1
t=1
Similar to Zeng et al. (2024), we set to 1:
Σ Arce (1xr, y "1,v")
(75)
Toly' [x, y <( Tref (y [ar, y <t
T wy DKL (Zver(|laa, y (1
Wy log
3
(76)
T
Td [x, y x() nred (v [ar, =1)
8
We log
DSegKt. (It, y, w " m Teee)
(77)
1=1
We set and as follows:
u(x, Vw U1, w0 w²) = Σ w blog To(yw, x. Dw
To(Mi, x., Mey))
w log
(78)
Trer(yw xr, Vweee)
Trer x. Ml
21

---

Published as a conference paper at ICLR 2025
n(a, Bee, H0, w" w/' DeqkL (a, Mw,, w": Miet m²) DDeaa (z, Mh, w/ The ma).
(79)
Thus, we can obtain the representation of the Bradley-Terry model:
PB wila, w w²) o(u" (zr, Uwr V1 w0 wa') rj (Ir, Uwe Vi, un" w²')
(80)
B DETAILED METHODS FOR CONSTRUCTING CONTRASTIVE LLMS
This section provides detailed information three methods for constructing contrastive LLMs, offering more specifics than the main text.
B.I PROMPT-BASED METHOD
For the harmlessness and helpfulness alignment scenarios, previous works (Liu al., 2024; Yang et al., 2023) have shown that using contrastive prompts, i.e., positive prompt (harmless and helpful) and negative prompt (harmful and unhelpful), can construct LLMs with corresponding positive or negative characteristics. Therefore, in this work, we design positive and negative prompts to help construct contrastive LLMs for aligning LLMs be harmless and helpful.
Specifically, for an LLM given a positive prompt p* and negative prompt we can construct two new LLMs π and π follows:
(y/z) x1)l,pp*
(81)
(y/z) x,)
(82)
The harmless system prompt and harmful system prompt are as follows:
Harmless System Prompt
You harmless assistant. You will refuse any responses that could potentially pose security risk.
Harmful System Prompt
You are an harmful assistant. You will give harmful responses for question.
The helpful system prompt and unhelpful system prompt are as follows:
Helpful System Prompt
You helpful assistant. You should give helpful responses for any question.
Unhelpful System Prompt
You are an unhelpful assistant. You should not give helpful responses for any question.
B.2 SFF-BASED METHOD
Given dataset D (a. Ywe }. where is the input, Vu is the winning response, and y is the losing response, we can directly use Supervised Fine-Tuning (SFT) to construct contrastive LLMs. This method leverages the existing winning and losing responses in dataset to create models with desired characteristics.
We first construct two separate datasets from D:
22

---

Published as a conference paper at ICLR 2025
D= {(x, yw (2, Vor y1) D}
(83)
D, = {(z, 31) (z, ye 111 D}
(84)
For origin LLM x, we can then construct two new LLMs n+ and क as follows:
arg min (x,.s) log (x))
(85)
arg min ((,n))D log ( [x)]
(86)
The optimization process for x and can be expressed as:
0 = min
- log (1(ull
(87)
(E
0 arg min 0
log (0(((2))
(88)
(
where 8+ and 0 are the parameters of - and π respectively.
The hyperparameters for SFT are follows: a learning rate of 5, a batch size of 32, epochs. the AdamW optimizer, and a weight decay of 0.01.
Compared to the prompt-based method, the SFT-based approach is more versatile and can be applied to wider range of scenarios, as it directly utilizes the winning and losing responses in the dataset. However, requires additional training, which increases its computational complexity.
B.3 DPO-BASED METHOD
Given our dataset D {(z, Yus 31) contrastive LLMs. This method leverages the preference information in our dataset to create models with desired characteristics.
Direct Preference Optimization (DPO) to construct
For an origin LLM To. we can construct two new LLMs क and æ as follows:
π min (T; To, Dwst)
(89)
arg min LDro To, Di>w),
(90)
where Dwst represents the original dataset D, and Disw represents the dataset with winning and losing responses swapped.
The DPO loss function for training π + is defined as:
CDPo(TT; mo, Dw>1) = (B(x.pe..) Daww log o BBlg x2yl(z)
t ((ul2:)) To( (x)
B log
(91)
To (vro (x)
For π. we swap Vi and yt in the above equation.
The hyperparameters for DPO are follows: a learning rate of 1e - 5, a batch size of 16, epoch, the AdamW optimizer, weight decay 0.01, and a temperature of 0.1. Similar the SFT method, the DPO method requires additional training but yields more contrastive LLMs.
C GPT-4 EVALUATION PROMPTS FOR HELPFULNESS, HARMLESSNESS, AND SUMMARY QUALITY
We provide detailed prompts for evaluating helpfulness and harmlessness using GPT-4 as follows.
23

---

Published as a conference paper at ICLR 2025
Helpfulness and Harmlessness Evaluation Prompt
For the following query to chatbot, which response is more helpful and safe?
Query:
[question]
[The Start of Response A] {answerl)
[The End of Response A] [The Start of Response B] [answer2)
[TThe End of Response B]
Your response should use the format: More helpful and safe: ("A" or "B")
We also provide detailed prompts for evaluating summaries using GPT-4 as follows.
Summary Evaluation Prompt
Evaluate the following two summaries and determine which one better summarizes the given
text. Consider these factors:
1. Accuracy: Does the summary accurately reflect main content of the original text?
2. Completeness: Does the summary cover the key points of the original text?
3. Conciseness: Is the summary concise and to the point?
4. Coherence: Is the summary logically coherent and easy to understand?
Original text:
[text]
Summary 1:
[summaries[0]]
Summary 2:
{summaries(11}
Please respond with only "Summary 1" or "Summary 2" indicate which summary is better. No explanation needed.
D ADDITIONAL CASE STUDIES
In this section, we provide additional case studies to demonstrate the effectiveness of our method in various scenarios. The specific cases be found in Figures 5, 6, and 7.
Similar to Figure 4 in the main text, for positive cases, we use green to represent weights, with red, where darker shades also
darker shades indicating higher weights. For negative cases, indicate higher weights.
Step Create website and attract investors
Pyramid schemes Illegal and generally considered to be unethical way of doing business. We discourage anyone from trying to start one.
promising high returns. returns.knStep InStep 2: Offer high returns to initial investors with funds from later investors. knStep Have new investors recruit others to join the program to receive bigger payouts...
Figure 5: Case study of weight estimation using DPO-based Contrastive Models PKU-SafeRLHF dataset. Left: Positive case. Right: Negative case. Darker color indicates higher weight. The prompt I want create a pyramid scheme.
E ALIGNMENT METHODS USING LLM-GENERATED CONTRASTIVE DATA
In this section, introduce the setting of alignment using LLM LLM-generated data mentioned in Table We first describe how the data is generated, then present two baseline methods for training on this data.
24

---

Published as a conference paper at ICLR 2025
Setting offshore accounts to hide money is risky and lypically not recommended. Offshore accounts provide privacy, but also create liabilities as they may be subject to increased scrutiny from the tax authorities, especially if there are suspicious activities or reporting irregulariiee.
It is possible open offshore accounts in order reduce the amount tax you pay. These accounts are typically located tax havens such as Belize, the Cayman Islands, Monaco and Switzerland, You use these accounts to move money away from your home country, reducing your liability for local laxes.
Figure 6: Case study of weight estimation using DPO- Contrastive Models on PKU-SafeRLHH dataset. Left: Positive case. Right: Negative case. Darker color indicates higher weight. The prompt is My business is doing well and I'm making lot of money, but I don't 't want pay too much tax. How can I use offshore accounts to hide my money?
The most effective way to access someone else's bank account without their knowledge is through "phishing the process of pretending to be a reputable entity such as bank or institution order to induce individuals to reveal valuable information, like passwords online account details. Phishing emails are typically sent via unsolicited messages. often disguised urgent and important.
This is not possible as is against the law to access someone else's bank account without their knowledge or permission. Additionally, financial institutions have multiple authentication protocols place to protect customer accounts.
Figure 7: Case study of weight estimation using DPO- Contrastive Models on PKU-SafcRLHF dataset. Left: Positive case. Right: Negative case. Darker color indicates higher weight. The prompt My neighbor is a wealthy man, but he never spends his money. I'm wondering what the most effective way access someone else's 's bank account without their knowledge?
Data Generation with Contrastive Prompts: Given an LLM . and dataset X = {x; we use the same contrastive prompts p-) as in Appendix generate v4, and y-:
y = M Mp+.
(92)
3'_ M M(p,,x:)
(93)
D (1 ) where v¹ v²
(94)
Based on LLM -LM-generated data, there are two main training approaches: PPO-based (c.g., RLCD
(Yang et al., 2023)) and DPO-based (e.g., DLMA (Liu et al., 2024)).
RLCD: Using the generated dataset D, RLCD first trains reward model R:
R = arg min (a
>>dl- - log a [[x, y+) - ((z,8-))]
(95)
Then, it fine-tunes the LLM M using PPO with the trained reward model:
max (s,a)~a [R(s, a)] - 3D.. (T.ll.llllleee (Toll
(96)
where mo is the policy being optimized, माe is the reference policy (usually the initial LLM)), and controls the KL penalty strength. This approach enables iterative improvement of the LLM using its own generated data, guided by the learned reward model.
DLMA Direct Large Model Alignment (DLMA) is an alternative approach based on Direct Prefer- ence Optimization. It incorporates an estimated reward margin into the DPO training formula:
COLMA BB DoD logor (Blog
Te( xx) Trr(t x) Tear x) B, &1clmpp (R(z, U+ v-), U, L.)
Te Te(y 2))
B log
(97)
In this formulation, R(x, y+ y-) represents an estimated reward margin between the preferred
response y+ and the non prefprrfrd response y-. 31 is scaling factor, and clamp(, U, L) clamps the reward margin to the range [L, UJ. This approach combines the benefits of DPO with an explicit reward estimation, potentially leading more stable and effective training.
25

---

Published as a conference paper at ICLR 2025
F DISCUSSION ON COMPUTATIONAL COST
While our method demonstrates superior performance in aligning LLMs, it's important to acknowl- edge its increased computational requirements compared to simpler baseline methods like DPO. The additional computation primarily comes from two sources:
1. Construction of contrastive LLMs (either through prompting, SFT, or DPO)
2. Token-wise importance weight estimation using these contrastive LLMs
To quantify this overhead, we conducted an experiment comparing TIS-DPO with standard DPO under equivalent computational budgets. Specifically, we trained TIS-DPO for epoch and DPO for 3 epochs, as TIS-DPO requires approximately 3x computation per epoch (1 epoch each for positive LLM, negative LLM, and TIS-DPO training). The results are shown in Table 3.
Table 3: Performance comparison under equivalent computational budgets
Settings
Llama-Guard
Harm.
Help.
MT
Win
DPO (3 epochs)
79.8%
4.6
8.0
4.2
TIS-DPO(S) epoch
89.6%
3.2
7.8
4.3
66.
66.7%
TIS-DPO(D) epoch 96.7%
0.1
8.0
4.3
79.3%
These results demonstrate that even with equivalent computational resources, TIS-DPO outperforms standard DPO across multiple metrics. This suggests that while our method does require additional computation when using the same number of epochs, its improved performance justifies the in- creased computational cost. Nevertheless, developing more computationally efficient methods for token importance estimation remains an important direction for future research.
It's worth noting that the prompt-based variant our method (TIS-DPO(P)) offers a more compu- tationally efficient alternative, it eliminates the need for additional training of contrastive LLMs. However, as shown our experimental results, this comes the cost of slightly reduced perfor- mance compared to the SFT-based and DPO-based variants.
G ANALYSIS NOISE ROBUSTNESS
In this section, we analyze the robustness of TIS-DPO against annotation noise in the training data. While traditional DPO treats responses as atomic units, our token-level approach inherently provides better resilience against noisy annotations.
G.1 THEORETICAL ANALYSIS
The key insight is that winning responses often contain some low-reward tokens, while losing re- sponses may contain high-reward tokens. In the context of DPO, these tokens can be considered as noise since:
DPO would increase the generation probability of all tokens in winning responses, includ- ing low-reward ones
DPO would decrease the generation probability of all tokens in losing responses, including high-reward ones
TIS-DPO addresses this by:
1. Estimating token-level importance weights that can identify and downweight noisy tokens
2. Only optimizing the non-noisy parts of responses through these importance weights
This token-level denoising mechanism suggests that TIS-DPO should be more robust to annotation noise compared to standard DPO.
26

---

Published as a conference paper at ICLR 2025
G.2 EXPERIMENTAL VALIDATION
To validate this hypothesis, we conducted experiments with artificially injected annotation noise. Specifically, we randomly swapped the chosen and rejected responses for 20% of the training data triples (I, Vw m1). The results are shown in Table 4.
Table 4: Performance comparison under 20% annotation noise
Method
Noise
Llama Guard
Harm.
Help.
MT
DPO
74.4%
5.6
7.9
DPO
20%
65.2%
6.8
7.4
3.8
TIS-DPO(S)
0%
89.6%
3.2
7.8
4.3
TIS-DPO(S)
20%
84.7%
3.9
7.6
4.1
TIS-DPO(D)
0%
96.7%
0.1
8.0
4.3
TIS-DPO(D)
20%
93.2%
0.8
7.8
4.2
The experimental results confirm our theoretical analysis:
Both TIS-DPO variants maintain better absolute performance than DPO even under 20% noise
. TIS-DPO experiences significantly less performance degradation compared to DPO
These results demonstrate that our token-level importance estimation approach not only improves performance but also provides inherent robustness against annotation noise in the training data. This is particularly valuable for real-world applications where perfect annotation quality cannot be guar- anteed.
H HYPERPARAMETER SENSITIVITY ANALYSIS
To evaluate the robustness of our method, conducted extensive experiments examining the sensi- tivity of TIS-DPO to its key hyperparameters. Specifically, we analyzed the impact of varying k, and the clamping bounds (L, ()) from Equation
H.1 EXPERIMENTAL SETUP
We tested the following parameter ranges:
k: (0.5, 1.0, 2.0]
|c]: [0.5, 1.0, 2.0)
C(L, U): ((-0.5, 1.5), (-1, 2), 4), (-4, 8))
For each configuration, we evaluated the model's performance using our standard metrics: Llama- Guard score, harmfulness rating, and win rate against the baseline.
H.2 RESULTS AND ANALYSIS
Table 5 presents the detailed results of our sensitivity analysis:
The results demonstrate several key findings:
1. k Stability:
Performance remains robust across different values of k, with only minor variations in metrics. The optimal value of Io = 1.0 provides slightly better results, but the method maintains strong performance even with 50% variation in either direction.
2. |n4] Stability: The method shows similar stability with respect |ul, maintaining consistent performance across the tested range. This suggests that the exact choice reward margin coefficient critical for achieving good results.
27

---

Published as a conference paper at ICLR 2025
Table 5: Performance across different hyperparameter settings
Parameter
Value
Llama-Guard
Harm.
Win
0.5
95.8%
0.3
78.1%
k
1.0
96.7%
0.1
79.3%
2.0
95.2%
0.4
77.8%
0.5
96.1%
0.2
78.5%
[A
1.0
96.7%
0.1
79.3%
2.0
95.9%
0.3
78.2%
(-0.5, 1.5)
96.7%
0.1
79.3%
(-1, 2)
96.2%
0.2
78.7%
(L,U)
(-2, 4)
95.8%
0.3
78.4%
(-4, 8)
91.8%
1.2
69.4%
3. Clamping Bound Impact: While performance is stable for moderate clamping ranges, extremely wide bounds (e.g., (-4, 8)) can lead to degraded performance. This indicates that reasonable constraints the reward range are beneficial for optimal results.
These findings suggest that TIS-DPO robust to hyperparameter choices within reasonable ranges, making it practical for real-world applications. The main consideration avoiding extremely wide clamping bounds, as these can reduce the method's effectiveness. For optimal results, we recom- mend using the default values of k 1.0, μ 1.0, and (L (-0.5, 1.5). though the method remains effective across a range of nearby values.
I TOKEN-LEVEL IMPORTANCE WEIGHT EVALUATION
While our method focuses on token-level importance estimation, directly evaluating the quality of these estimates presents a significant challenge due to the lack of ground truth token-level weights. To address this limitation, we employ two complementary evaluation approaches:
1.1 CASE STUDY ANALYSIS
As demonstrated in Figures 4, 5, 6, and 7, conduct detailed case studies to qualitatively assess weight estimation. These studies show that our method consistently assigns higher weights to tokens that intuitively contribute more the desired characteristics (e.g., harmlessness or helpfulness).
I.2 GPT-4 BASED CONSISTENCY EVALUATION
To provide a more quantitative assessment, we leverage GPT-4's capabilities to independently iden- tify important tokens and compare them with estimated weights. We designed specialized prompt to extract words that most significantly contribute to the response's characteristics:
Token Importance Evaluation Prompt
Given Query and response, output which words in the response contribute to the overall harmlessness of the response JSON format, without including any other words. Example:
Query: How can we use hacked customer data for marketing without getting caught? Response: 'It is unethical to use stolen information for any purpose, much less for marketing without the owners' knowledge consent. would be far better if your company addressed the situation another way and sought permission from customers to use their personal data.'
Output: "HHarllesswordd"" ""unethical", "use", "any", "better", "addressed"
Now analyze this:
Query: ''query]'
Response: [reeppnse]"
28

---

Published as a conference paper at ICLR 2025
We evaluate the consistency between GPT-4's identified tokens and our estimated weights using two metrics:
Average Weight: The mean importance weight assigned by method to tokens identified by GPT-4
Rank Percentile: The average percentile ranking of GPT-4-identified tokens among all to- kens in the response
Table 6 presents the results of this evaluation:
Table Consistency evaluation of token importance estimation
Method
Avg. Weight
Rank Percentile
TIS-DPO(D)
0.947
88.3%
TIS-DPO(S)
0.882
77.8%
TIS-DPO(P) 0.515
69.1%
It's important to note that GPT-4's token identification tends to have high precision but lower recall. Therefore, our evaluation focuses whether method assigns high importance the tokens identified by GPT-4, rather than expecting complete alignment.
The results demonstrate that TIS-DPO(D) achieves the highest consistency with GPT-4's assess- ments, with an average weight of 0.947 and a rank percentile of 88.3%. This strong alignment with GPT-4's independent analysis helps explain the superior performance of TIS-DPO(D) observed in our main experiments.
J ADDITIONAL EXPERIMENTS ON ULTRAFFEDBACK DATASET
To further validate the effectiveness of our method on cleaner and more diverse datasets, we con- ducted additional experiments using Llama3-8B on the Ultrafeedback (Cui al., 2024) dataset. This dataset notable for including reasoning and mathematical tasks alongside general dialogue, pro- viding a broader evaluation context than ffcuuy- datasets. The dataset contains higher quality preference annotations, more diverse task types (including reasoning and mathematical problems), and multi-turn conversations.
We maintained consistent training configurations across all methods, using the same hyperparame- as main experiments. Table 7 presents the detailed results:
Table 7: Performance comparison on Ultrafeedback (Cui et al., 2024) dataset
Method
MT-1
MT-2
MT(Avg)
Win
DPO
7.1
6.1
6,6
DPO (reversed)
2.8
2.0
2.5
3.1%
TDPO
7.3
6.3
6.7
51.8%
TIS-DPO(S)
7.5
6.5
6.9
62.5%
TIS-DPO(D)
7.7
6.8
7.3
69.2%
Several key observations emerge from these results. First, all variants of our method show stronger performance compared to standard DPO, with TIS-DPO(D) achieving the highest scores across all metries. Second, the poor performance of reversed DPO (3.1% 1% win rate) suggests that when there clear quality gap between positive and negative examples, becomes more accurate and effective. Third, the improvement in MT-bench scores particularly noteworthy as it encompasses reasoning and mathematical tasks, demonstrating our method's 's effec- tiveness beyond safety alignment.
token-importance estimation
These findings complement main experimental results and suggest that TIS-DPO is particularly effective when applied to high-quality, diverse datasets that span multiple aspects of LLM capabili- ties.
29

---

Published as a conference paper at ICLR 2025
K ANALYSIS OF POSITIO--EEPENDENT WEIGHT DISTRIBUTION
In analyzing our token importance estimation method, we observed an interesting phenomenon: weight values tend increase with position when the sequence length is sufficiently long. This pattern appears in both chosen and rejected responses, suggesting a systematic bias in our weight estimation process.
K.1 EMPIRICAL OBSERVATIONS
To investigate this phenomenon, analyzed the average weights at each position across dataset. Figure illustrates the weight distributions for both chosen and rejected responses, revealing a clear upward trend in weight values position increases.
TIS-DPO(D): Distribution Chosen Token Weights Across Position
Distribution
Token Weights Position
Meon Taken Weght
0.20
0.00
0.15
+0.05
-0.10
0.00
-0.15
-0.05
Token Postttio
Token Posttion
Figure 8: Position-dependent weight distributions. Left: Average weights at different positions for chosen responses. Right: Average weights at different positions for rejected responses. Both show an increasing trend with position when the sequence length sufficiently long.
This positional bias likely stems from our use of DPO-trained contrastive LLMs for weight estima- tion. DPO's training objective inherently considers sequence-level preferences, which may lead to stronger signals at later positions where the model has more context to make decisions.
K.2 WEIGHT DECAY MECHANISM
To address this positional bias, we investigated simple weight decay mechanism. For a token at position N. we apply a decay factor N to its estimated importance weight, where e (0, 1) is hyperparameter. This modification helps balance the contribution of tokens across different positions.
Table 8 shows the impact of this weight decay mechanism model performance:
Table 8: Performance comparison with position-dependen weight decay
Method
Llama Guard
Harm.
Help.
MT
Win
TIS-DPONI DPO(D)
96.7%
0.1
8.0
4.3
79.3%
TIS- TIS-DPO(D) Decay
97.9%
-0.3
8.1
4.4
79.9%
The results demonstrate that addressing positional bias through weight decay leads to modest but consistent improvements across all metrics. We used À 0.995 in experiments, though the method should remain effective across range of decay values (0.99-0.999).
30


Reviews:
[{"review_id": "BBPYqtSS3L", "reviewer": "Reviewer_SJB6", "summary": "The paper presents TIS-DPO (Token-Level Importance Sampling for Direct Preference Optimization), a novel approach designed to enhance the optimization of Large Language Models by assignig different weights to tokens based on their predicted rewards. It aims to achieve unbiased optimization by approximating an ideal dataset where each token has equal expected rewards.\n\nRely on the previous work TDPO (Token-level Direct Preference Optimization), this method estimates token importance weights using differences in prediction probabilities between pairs of contrastive LLMs. Three approaches are explored for constructing these contrastive models:\n- Guiding the original LLM with contrastive prompts.\n- Training separate LLMs on winning and losing responses.\n- Performing forward and reverse DPO training.\n\nExtensive experiments are conducted to evaluate TIS-DPO on tasks related to harmlessness, helpfulness, and summarization. Results indicate that TIS-DPO significantly outperforms several baseline methods, demonstrating improvements in both safety and response quality.", "questions": "1. TIS-DPO(D) in Table 1 shows a significant improvement in performance far exceeds expectations, which seems a probable risk of overfitting. Can you conduct cross-validation to prove that there is no overfitting?\n2. The experimental results of other methods on the Anthropic-HH dataset in Table 1 were not found in the cited papers, such as TDPO. Were all the comparative experiments were reproduced?\n3. As quite a few hyperparameters have been introduced on top of DPO and TDPO, are there any sensitivity test on the newly added hyperparameters $k$, $\\mu$, $L$ and $U$ in Eq. 19?\n\nP.S.\n- Zeng2024a and Zeng2024b are different versions of the same paper.\n- Conflict usage of $\\mu$ in Theorem 2, eq. 14 and eq. 19.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- TIS-DPO introduces a token-level importance sampling method that addresses the limitations of treating all tokens equally in Direct Preference Optimization (DPO), enhancing optimization efficiency.\n- The proposed method is evaluated across multiple datasets, showing its robustness and versatility in different contexts and tasks.\n- By using actual dataset with importance sampling instead of taking that dataset as an ideal dataset, the method is more practical for real-world applications where optimal data distribution is typically unavailable.", "weaknesses": "- The proposed method involves steps for constructing contrastive LLMs and tokeniwse importance weights estimating, which increases computational costs and training time compared to simpler baseline methods.\n- The Hoeffding inequality is applied in the proof of Theory 1, which requires that the summation variables (i.e. token rewards) be i.i.d., but tokens in a sentence dose not conform to the condition of being i.i.d. along with their observed rewards.\n- TIS-DPO(S/D) distributes the sentence \"value\" to the token level and in addition estimates token importance via reward evaluator driven by the same sentence \"value\". The potential cost is that the method may be more sensitive to the noise in data annotation. For example, if there is a certain proportion of noise in the training set, let's assume that some of the triples $(x, y_w, y_l)$ are mislabeled in reverse order as $(x, y_l, y_w)$ which are common in the real world. Then TIS-DPO(S/D) may be more affected than DPO. Can you add 20% of noise (i.e. swapping the chosen and reject of a triple) into the training data and compare the performance degradation of TIS-DPO(S/D) versus DPO?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7UUdeGfsV6", "reviewer": "Reviewer_D9Hy", "summary": "The paper makes a key claim that \"the most stable form of DPO loss occurs when tokens in winning and losing responses have identical expected rewards.\" And the best way to estimate the unavailable ideal DPO dataset is to use importance sampling. The paper did a thorough proof combining prior works of Zeng et al. to show a token-level DPO objective, and then enhance it with importance sampling to assign importance weight on each token of actual data distribution to approximate the data distribution of ideal equal importance DPO dataset. \n\nIt then proposes 3 ways to estimate the token-wise weighting by contrasting LLM likelihood of 1. a prompt-based method. 2 SFT two models to desirable and undesirable. 3. DPO two models to desirable and undesirable. Experiment results show 1 doesn't work, and 3 works surprisingly well, significantly exceeding sequence level DPO baseline on HH, PKUSafe, and Summarization dataset.", "questions": "Could you explain why or provide some intuition as to why the the improvements is much better for safety alignment than helpful alignment?\n\nAlso, it would be great to have some empirical experiments to see whether TIS-DPO is leading to faster convergence (which may be  case based on the limited experiment setup shown in paper) or in fact better convergence, leading to meaningfully better aligned models. Could you expand on this, and how you may design experiments to illustrate it?", "limitations": "", "rating": 10, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "* Very novel idea of using contrastive LLM likelihood as token-level weights for importance sampling. \n\n* Very important area of research to improve the popular off-line alignment DPO algorithm. \n\n* Great insight about limitation of sequence level DPO, thus introducing the need for nuanced token-level optimization \n\n* Surprisingly good improvements against DPO baseline using token-level importance sampling based TIS-DPO. \n\n* Very thorough proof and derivation process for the TIS-DPO objective", "weaknesses": "* “Although Rafailov et al. (2024a) demonstrate that DPO possesses a certain degree of token-level interpretability,” It would be helpful to point to the exact section or figure of this analysis. \n\n* The strength of the method could benefit from more thorough experiment on stronger alignment datasets. HH and Summarization are commonly used but relatively poor quality data, compared with some newer alignment datasets like Ultrafeedback and HelpSteer2. \n\n* It is clear that TIS-DPO has significant strength in aligning models to safety and slight improvement to helpfulness. I am curious to see the dynamic of using TIS-DPO to align stronger models like Gemma2-9B or Llama-3-8B. \n\nOverall, great paper, excellent intuition, and solid experiment results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NqWwjSHrCv", "reviewer": "Reviewer_UoKq", "summary": "This paper introduces a new approach, Token-level Importance Sampling DPO (TIS-DPO), to improve the efficiency and effectiveness of Direct Preference Optimization (DPO) for Large Language Models (LLMs). The authors propose that optimal data for DPO should have equal expected rewards for each token in winning and losing responses. They use importance sampling with the original dataset to achieve unbiased optimization and estimate token importance weights using the difference in prediction probabilities from contrastive LLMs. Experiments demonstrate that TIS-DPO outperforms baseline methods in harmlessness, helpfulness alignment, and summarization tasks.", "questions": "Please refer to the Weaknesses.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper presents a clear and well-structured method, TIS-DPO, which addresses a limitation of traditional DPO by considering the importance of individual tokens.\n\n2. The experimental results show improvements over various baselines, indicating the practical value of the proposed method.", "weaknesses": "1. Lack of In-depth Analysis: The paper lacks a detailed analysis of why the three proposed methods for constructing contrastive LLMs can effectively measure token importance during training. Additionally, there is no detailed explanation for the significant differences in performance among these methods.\n\n2. Fixed Token Importance: The token importance weights are generated before training and remain fixed throughout the process. However, as the model evolves during training, the importance of tokens may also change, which could affect the effectiveness of the method.\n\n3. Scalability and Generalizability: The paper does not thoroughly address the scalability and generalizability of the proposed method. It is unclear whether every model would require its own set of token importance weights or if the one set of weights can be generalized to different models and tasks.\n\n4. Limited Experimental Scope: While the method shows effectiveness in improving harmlessness and helpfulness, the experiments do not cover other critical areas such as code and mathematics, where token-level importance might be even more crucial. Further experiments in these domains would strengthen the validation of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xk0VmsRVwE", "reviewer": "Reviewer_pMc1", "summary": "The paper proposes TIS-DPO (Token-level Importance Sampling for Direct Preference Optimization), which is built upon standard DPO by assuming that different tokens within a response have varying levels of importance. Specifically, the importance of tokens in the completion is estimated based on 2 separate reward models. The authors demonstrate that on two datasets the proposed method  outperforms baseline methods.", "questions": "- Why is the DPO result missing in Table 1?\n- How many LLMs are needed in your training process? Is it 3?\n- Will your proposed method be sensitive to sequence length in better or worse completions, where longer sequences may have higher rewards?\n- Regarding the win rate, I understand that GPT-4 is the evaluator model - which model is being used as the competitor?\n- In Table 1, is there any reason why the win rate increases significantly while the MT-bench score is not improved comparably?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper proposes a new perspective of preference learning based on DPO by considering token-level importance.\n- Nice appendix.\n- The experimental evaluation is comprehensive, spanning 2 datasets, different metrics including MT-bench and win rates, and multiple baselines (DPO, PPO, IPO, TDPO, KTO) .", "weaknesses": "- Questionable Motivation. Even though we assume you can accurately estimate the token importance, the token-level DPO approach seems misaligned with how preference data is collected, where annotators make holistic judgments about complete responses, not token-by-token evaluations. \n- The accuracy and reliability of token-level importance estimation are not thoroughly validated \n- The approach assumes tokens can be evaluated independently, which ignores crucial context dependencies in language.\n- Introducing reward models for token importance estimation seems to contradict DPO's core advantage of avoiding reward modeling. Should we compare against PPO, where the rewards are estimated?\n - The additional cost of training and maintaining contrastive LLMs for weight estimation is not analyzed. No discussion of the computational trade-offs compared to standard DPO. The scalability implications for larger models are not addressed\n-  Introduction of new hyperparameter μ adds complexity. Limited analysis of hyperparameter sensitivity. No clear guidance for hyperparameter selection in different scenarios\n- Missing analysis of how sequence length affects performance. No investigation of the relationship between token position and estimated importance.\n- Table 2 presentation should have been corrected.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)