Token-Importance Guided Direct Preference Optimization
Authors: Ning Yang
Hai Lin
Yibo Liu
Baoliang Tian
Guoqing Liu
Haijun Zhang
Venue: ICLR 2026 Oral (2026)
Keywords: LLMs
RLHF
DPO
Human Preference Alignment
Token-lmportance
Triplet Loss
Primary area: foundation or frontier models, including LLMs
TL;DR: We proposes Token-Importance Guided Direct Preference Optimization (TI-DPO) to better align LLMs with human preferences by using a hybrid weighting mechanism to identify key tokens and a triplet loss to guide the optimization process.

Abstract: Aligning Large Language Models (LLMs) with human preferences is crucial for safe and effective AI interactions. While popular methods like Direct Preference Optimization (DPO) have simplified alignment, they remain sensitive to data noise and overlook the differential importance of individual tokens. Existing token-level approaches often rely on probability prediction or simplistic weighting schemes to obtain token importance, which still cannot fully address these issues. To solve this problem, we propose the Token-Importance Guided Direct Preference Optimization (TI-DPO), a framework that achieves fine-grained semantic control through two synergistic innovations. 
First, we propose a novel hybrid weighting mechanism that combines gradient attribution with a Gaussian prior, ensuring both the accuracy and robustness of token importance scores. Second, we employ a triplet loss to provide structured guidance for the optimization, explicitly guiding model outputs to approach preferred responses and diverge from non-preferred ones. Experimental results show that TI-DPO achieves higher accuracy and stronger generative diversity, providing more stable and computationally efficient solutions compared with DPO and other RLHF methods.

Published as a conference paper at ICLR 2026
TOKEN-IMPORTANCE GUIDED DIRECT PREFERENCE OPTIMIZATION
Ning Yang' Hai Lin'
Yibo Liu
Baoliang Tian
Guoqing Liu"
Haijun Zhang
IIstitute of Automation, Chinese Academy of Sciences
The Chinese University of Hong Kong, Shenzhen *ByteDance *University of Science and Technology Beijing
+ Microsoft Research AI4Science
ABSTRACT
Aligning Large Language Models (LLMs) with human preferences is crucial for safe and effective Al interactions. While popular methods like Direct Preference Optimization (DPO) have simplified alignment, they remain sensitive to data noise and overlook the differential importance of individual tokens. Existing token-level approaches often rely on probability prediction or simplistic weighting schemes obtain token importance, which still cannot fully address these issues. To solve this problem, we propose the Voken-IImprtaane Guided Direct Preference Op- timization (TI-DPO), a framework that achieves fine-grained semantic control through two synergistic innovations. First, we propose novel hybrid weight- ing mechanism that combines gradient attribution with a Gaussian prior, ensuring both the aceuracy and robustness of token importance scores. Second, we ploy a triplet loss to provide structured guidance for the optimization, explicitly guiding model outputs approach preferred responses and diverge from non- preferred ones. Experimental results show that TI-DPO achieves higher accuracy and stronger generative diversity, providing more stable and computationally ef- ficient solutions compared with DPO and other RLHF methods. Code and demo are available at
INTRODUCTION
Large Language Models (LLMs) have shown proficiency in Natural Language Processing (NLP) (Gao et al. 2025), logical reasoning (Xie et al. 2025). and code generation (Xu et al. 2025), emerg- ing as a focal point recent research. However, as models may generate outputs inconsistent with intended purposes ethical standards, human preference alignment aims to ensure that LLMs ad- here to human values (Liu et al., 2023). producing beneficial and harmless content. Against this backdrop. Reinforcement Learning from Human Feedback (RLHF) has become prevailing ap- proach for achieving alignment (Hong et al., 2024 Hu et al. 2025). It leverages human-annotater preference data to train reward models and fine-tuned tunes LLMs using Reinforcement Learinng (RL) methods (Wang et al. 2023b) like Proximal Policy Optimization (PPO) (Schulman et al. (017))
The emergence of Direct Preference Optimization (DPO) has effectively simplified the alignment process (Rafailov et al. 2023). Inspired by DPO's implicit reward mechanism, a series of preference optimization models have been proposed in recent years, such as ORPO (Hong et al. 2024). f- DPO (Wang et al., (223a)), and CPO (Feng et al., 2025). However, both DPO and RLHF have a fundamental flaw during optimization: they optimize the sequence level, leading to the neglect of the influence of specific tokens, which in turn destabilizes the training process due to shifts in the sampling distribution (Zhang et al.) 2025).
Motivated by these challenges, researchers have proposed token-level variants of DPO, aiming to de- compose preference alignment into fine-grained contributions (Zeng et al. 2024 Xie et al., 2025 Zhong et al.) 2024)). However, achieving true fine-grained alignment requires addressing a core challenge: We not only need to accurately identify the key tokens that have a decisive impact on human preferences, but also need subtle optimization objective to guide the model to adjust its
"Correspondence Ning Yang fainggyyng@iia.ce

---

Published as a conference paper at ICLR 2026
preference (Li et al. 2025). Nevertheless, existing token-level methods fall short in dealing with this challenge for two reasons. First, their approaches to identifying key tokens often rely on biased probability proxies (Liu et al. 2024a) or overly simplified weighting schemes (Lin et al. 2024). Second, the optimization, they still inherit the binary comparison framework of DPO, simply distin- guishing between "good" and "bad" samples (Meng et al. 2024). Such coarse-grained supervision signals cannot finely shape the model's generation behavior in continuous semantic space.
In our Token-Importance Guided Direct Preference Optimization (TI-DPO) framework, we intro- duce novel hybrid weighting mechanism to accurately and robustly identify key tokens. This mechanism combines gradient attribution with Gaussian prior, overcoming the problem of exist- ing methods relying on biased proxies. Here, gradient attribution is a technique used to determine the contribution of each input feature (in our work, each token) to the model's ' output (Ancona al.) 2017 Ballout et al., 2024). Liu et al. (2024b) offered empirical evidence that models exhibit U- shaped attention bias, which means there greater importance to tokens at the beginning and end of sequence, while underweighting those in the middle. Thus, the Gaussian prior distribution here is explicitly designed to rectify this intrinsic architectural bias, ensuring that the optimization process does not neglect the semantic core of the response.
Meanwhile, we adopt a structured triplet objective based on the identified key weights to achieve fine-grate fine-grained optimization by incorporating the intermediate generated outputs (guuyen al. 2018). This triplet structure explicitly guides the intermediate output to approach human preferences and distance from non-preferred responses, achieving fine-graines preference alignment and promoting a continuous gradient of preference learning. The mixed weights and the triplet loss complement each other and together form a complete solution for TI-DPO achieve fine-grained alignment.
The following contributions are made in the course of this work:
We propose TI-DPO, a novel framework designed for achieving fine-grained alignment. This framework innovatively integrates hybrid weighting mechanism, jointly formed by gradient attribution and a Gaussian prior, with triplet loss, significantly enhancing the ro- bustness and stability of weight allocation.
Theoretically, we formalize the TI-DPO framework by providing a complete derivation of its loss function and gradient. Building on this, we prove TI-DPO achieves tighter loss bound than DPO (Theorem 2 and the superiority of expected reward (Theorem B). This theorem formally provides new perspective on comprehending the superiority of TI-DPO in terms of alignment accuracy.
Experiment results indicate that TI-DPO surpasses existing methods in aligning LLMs with human preferences. Notably, our method achieves a leading average score of 62 and substantially outperforms strong baselines and IFEval with scores of 67.0, 62.0 and 75.7 respectively. Further analysis, including ablation studies and sensitivity analysis, confirms that both of our core contributions are vital to this performance.
key tasks such HumanEval, TruthfulQA
RELATED WORK
Human Preference Alignment Human preference alignment has emerged as a critical research paradigm in recent years, focusing on enabling model responses to align with human values and preferences. Early advancements mainly focused on RLHF (Ouyang et al., 2022 Bai et al. 2022) based on PPO (Schulman al., 2017). However, these RL methods may suffer from overfitting in optimal responses. To mitigate this issue, Hu et al. (025) introduced the Reinforce++ model, which employs batch-wise standardized rewards prevent overfitting and enhance the prompt di- versity during training. Concurrently, beyond RL approaches, Rafailov et al.] (2023) introduced DPO, which obviates the need for explicit reward modeling through implicit preference learning- This implicit reward mechanism has inspired a wave of subsequent works (Cui et al.) 2025), such as the SimPO algorithm proposed by |Mng et al. (2024), which utilizes the sequence-averagee log probability as an implicit reward signal to streamline optimization. Notwithstanding these advance- ments, DPOs' reliance on large-scale human annottatdd preference datasets (Kim et al. 2025) has motivated derivative studies (Gou & Nguyen 2024 Jiao al.) (024) aimed at reducing data quirements. A notable example is RS-DPO (Khaki et 2024). which integrates rejection sampling with DPO alleviate data scarcity. However, more fundamental limitation pertains to the binary
2

---

Published as a conference paper at ICLR 2026
nature of traditional preference labels. Although the KTO method proposed by Ethayarajh et al. (2024) effectively reduces the reliance on paired preference labels DPO, most current RLHF and related preference optimizations still mainly rely binary comparisons between "good" and "bad" responses (Gao et al. 2024 Hong et al. 2024). Such coarse-graines supervision has obvious short- comings: human preferences often show continuous gradient differences rather than simply "good" and "bad". Against this backdrop, our proposed triplet optimization method can achieve fine-grained preference alignment.
From Sequence-Level to Token-Level Achieving fine-grained alignment requires the model not only to distinguish the quality of the entire sequence, but also to understand and precisely control the key morphemes that constitute the semantics of the sequence. Existing sequence-level techniques of- lead to decrease in generation diversity because they ignore the importance differences among tokens (Feng et al. 2025). J. These limitations have spurred researchers' research into step (Xie et al., 2024) or token-level (Rafailov al.) 2024) alignment mechanisms, seeking to address the granu- larity mismatch between coarse-grainee grained sequence rewards and fine-grained token contributions (Xi ct al. 2024). To address the significant decline in model generation diversity, TDPO (Zeng et al, 2024 reanalyzed and optimized the entire alignment process from the token-level perspective. An additional limitation of RLHF and DPO lies in the fact that rewards are only assigned to the fi- nal token, with all other tokens receiving no learning rewards (Zhong al., 2024). Meanwhile, Xie et al. (2025) proposed a correlation between the frequency of specifie tokens and model per- formance, which inspires us to consider reassigning token weights. In related vein. Liu et al. (2024a) estimates token importance weights using prediction probability differences. Nevertheless, this probabilistic weighting scheme prone to bias when contrastive models produce inconsistent outputs or fail to capture subtle semantic nuances of human preferences. In contrast, our approach employs a hybrid strategy that combines causal gradient attribution with a stabilizing Gaussian prior to estimate importance (Ballout et al., 2024). By focusing on actual gradient impacts, our method enhances alignment precision over probabilistic proxies, while the prior distribution ensures robust- ness against noisy gradient signals.
3 PRELIMINARIES
Before formally elaborating the TI-DPO method, this section first introduces relevant preparatory knowledge to lay the foundation for the subsequent theoretical derivation and model construction.
3.1 HUMAN PREFERENCE ALIGNMENT
Firstly, we focus on the core concept of human preference alignment, which is the foundation for op- timizing the response generation of LLMs. Suppose that x stands for the input prompt and y denotes the response generated by the model. The key approach involves optimizing the -egnnrttnnnoroioonn policy To (y|x)) It utilizes a carefully selected human preference dataset Bw and y represents preferred response and non- con-preferred response. Reward model ra (Ir 3) evalu- ates the L.LMs' responses by applying the Bradley-Ted Trarlyy-Terry (BT) model for ranking loss Ouyang et al.) (2022). The loss function employed to access the reward model r using dataset D is formulated as tollows:
{(x,, yw yt Here
Lruse
E (a..n. [log (ra (z, ww) ro (x., 3n)]]-
Here a--) is the sigmoid activation function. The reward model evaluates the LLMs' responses by applying the BT model for ranking losses (Ouyang et al. 2022);
exp (ro (I, 3w))
p (y > 31 xz) exp (r& (æ, )w)) + exp (r& (z, un))
(2)
The partition function Z(x) serves to normalize the policy's probability distribution (Rafailov al.
2023). The parameter 3 regulates the extent of divergence between Tg and गe-- DPO rearranges (v(=) log (zx) Let the input prompt be
this equation to express the reward as ra( y) = 3 log )
mm]] and the first tokens generated by the model be denoted as y <t [ y ..... . n² -. Let Tw and Ti denote the number of preferred tokens and less preferred
represented as x = [x),
3

---

Published as a conference paper at ICLR 2026
tokens, respectively. The token-level DPO optimization objective is given by (8(10g To (yw((z) Tie (thw (z)
log we (yn(ar) (r) Tief (31 (z)
Looo -BB(s.....n log
(3)
3.2 TRIPLET LOsS
Triplet loss, powerful loss function for learning embeddings, ensures that within the embedding space, an anchor input is closer to positive inputs than to negative oncs. This mechanism enhances the model's capacity to differentiate between data points that more or less similar. By simultane- ously learning from the similarities and differences among sampled data points, the model is better aligned with human evaluations. The triplet loss operates with triplets (zi, Tk), and is designed such that the representation of the anchor I; nearer to similar data point 2; than to a dissimilar one TA. This targeted learning strategy is instrumental in sharpening the model's feature discrimi- nation, thereby improving its ability to make decisions that resonate with human preferences. The triplet loss is given by
T Lup = Σ (1.) 11f - (xx) aurp 1J.K
(4)
Here [z]+ denotes the rectified linear unit function, ensuring that is set to zero if negative. The features extracted from the three inputs are represented by the terms f (a;), f (xz)). and (ax).
METHODOLOGY
Driven by the challenges of unstable training and distribution shift in traditional RL alignment meth- ods, we propose the TI-DPO framework. Our key innovation lies a novel hybrid weighting strat- egy and a triplet loss that provides a structured optimization objective.
4.1 TOKEN-LEVEL MDP FOR LLM PREFERENCE ALIGNMENT
To address the challenges of the sequential and auto-regressive nature of text generation, token- level Markov Decision Process (MDP) introduced, which incorporates the notion of token sig- nificance to improve the alignment of each token selection with human preferences. This concept defined through a tuple denoted M (S.A, P, Po). and are the state space and ac- tion space, respectively. ' is deterministic transition model among tokens. Here r stands reward model associated with each token, and Po indicates the initial state distribution. The initial state so = [x]]. which is simply the input prompt. At each step & of the generation process, the state consists input prompt æ, where the count of token, and 1 generated tokens y .y.,," At each time step f, the action d₂ y' corresponds the selection of subsequent tokens for generation.
8, = [I,
4.2 HYBRID WEIGHTING MECHANISM FOR TOKEN IMPORTANCE
Building on the token-level MDP framework, we formalize the caleulation of importance weights Inspired by the attribution-based rationale extraction from Ballout et al. (2024), our approach quantifies token importance through gradient sensitivity analysis, ensuring that critical tokens in human-preferred responses drive the policy optimization process.
However, while gradient attribution provides a precise, data-driven signal, it can be susceptible to noise. Some studies have pointed out that imposing additional constraints on the attention or importance distribution can help the model focus on key information (Zhang al. 2018 Guo al.) 2019). Furthermore, a recent study (Liu et al. (024b)) offered empirical evidence that models exhibit a U-shaped attention bias. This means there is greater importance to tokens at the beginning and end of sequence, while underweighting those in the middle. The Gaussian prior distribution, which peaks at the center, is designed to counteract the architectural LLMs. ensuring the optimization does not neglect the semantic core of the response. The Gaussian prior also prevents the model from overfitting to noisy gradient signals and provides a stable baseline, ensuring the weight distribution remains well-behaved throughout training.
coost-in-thh-Miiee
bias inherent in
4

---

Published as a conference paper at ICLR 2026
A token that significantly impacts the reward is deemed critical, whether it has positive effect on the preferred response or a negative effect on the non-preferred response. For a given sequence of tokens y (1) 92..... Vr-il, we first obtain its embedding sequence E' [ei, E2..... Cr- i1, where e; is the embedding vector for token Yi- We then perform a forward pass to get the logits for the final token, Tr-1 R where V is the vocabulary size. The target scalar value for our gradient calculation, Learget. is the maximum logit value at this final step, which represents the model's most confident prediction for the next token yr:
Larger max (L
(5)
Next, compute the gradient of this target logit with respect each token's 's embedding e; in the sequence. This gradient, V ei captures the direct influence of token on the final prediction. To obtain scalar importance score I from the gradient vector, similar to previous work (Ballout
et al. 2024), we compute its L1 norm:
I = 1[ Furgalll = Σ
Cuurge) Cunee
(6)
Here, k indexes the components of the gradient vector. This score, 4, represents the raw, data-driven importance of token i.
Finally, to ensure training stability and robustness against noise in gradient estimates, we post- process these raw scores derive the final weights w,. As implemented in our code, this involves mixed strategy: (1) First, the raw scores I = (I,,.., IT-1) } are normalized by their sum to form a distribution Inomm (2) Then we define
moussian-shapees prior distribution Ppricr centered on the sequence, which assigns higher baseline importance to tokens in the middle. For a sequence of [0, T 1]. the unnormalized value is calculated as:
length T' and each token position E
1 t- (-)(t=²))
Preior (1) = exp
where we heuristically set the mean = (T 1)/2 and the standard deviation a T/4. Since approximately 95% of the mass Gaussian distribution lies within +20 setting ensures the prior effectively spans the entire sequence context without being too narrow or too flat.
The final weight vector a convex combination of these two distributions, controlled by hy- perparameter À E 1J:
= X Ipom + (1 - A) Pprior-
(8)
This weighting scheme is applied independently to both the preferred yv and non con-preferred se- quences to obtain their respective token-level weights, denoted as w."" and w!.
The gradient- based importance guidance method provides a data-driven measure of token relevance, which can adapt to the subtle semantics human preferences and achieve fine-grained control over key tokens during the model generation process. These weights then modulate the implicit reward signal at each token step, effectively focusing the DPO objective on the most critical tokens. The resulting preference probability under BT model is:
exp Eth W. rools" a²))
(yw > y))
(9)
exp
E Ei=pr u a texp
Et
Here, Tn and Ti are the lengths of yw and respectively.
Then, with ro at) = 3 log pu(""ln in DPO, we can derive the expression for BT model: malv"l
(uw > vi) = aa (rrween (x, Dw Uh, w." w.)),
(10)
where Arikkn (x. yw yd, w., w!) can be denoted as:
To Arwsen (xr, Bws Uh w. w!) Σ EWllogg
(v' (. Teet (y xy
Ti Σ
To (ef (1))(() woo
(11)
F uf (1))
Therefore, obtain the weighted token-level DPO base loss as:
LDOO- (R.. [log (Arioka (2r, Uws Mr wo, w w()]]
(12)
5

---

Published as a conference paper at ICLR 2026
4.3 TRIPLE Loss IMPLEMENTATION
The practical implementation of the triplet loss is integrated seamlessly within the main training loop to provide structured guidance for the policy model.
First, for each data batch comprising (z, Vw, 91). the process begins by generating anchor response y: Using the preferred response V/w as the starting point of the context, the response dynamically generated by the policy model To is the anchor y. It represents an intermediate state in the model's generation space.
Next, by mapping each of the three responses to point in continuous preference space, we calcu- late the distances between these three responses y, yw and y/, which represent the preference for the newly created anchor y.
Finally, according to the definition Eq 4. the triplet loss in our work calculated with these distances, penalizing the model if the anchor not closer to the positive response than to the negative one by predefined margin:
Cuplat (rr max(O,
log (y xz, y ) Terryy lx, )
To (y yu
no (vill,, 1) mee Ic, Ui
log To(y" a yーt) || tex)
log
Σ
log
Tieel ( Ir, Yo
Terr yer
Align with
Push away
(13)
4.4 TI-DPO OBJECTIVE AND THEORETICAL ANALYSIS
We now formally define the complete TI-DPO objective, which unifies our hybrid weighting and triplet loss mechanisms, and provide theoretical proof of its superiority over standard DPO. With given TI-DPO dataset 2 = {z, Yw .B)). we obtain TI-DPO objective:
LTD LDPO + Lurplett:
(14)
where y is hyperparameter. In Appendix A.4 we have given the proof of gradient VoLII-DPO- which is used to update 8 during training. The implementation of TI-DPO shown in Algorithm
Algorithm TI-DPO
1: Input: Dataset D {zx Uws v)), hyperparameter B, X, reference model Teet policy model Tg.
2: Initialize: my <- Nel
3: for each epoch do
4: Sample batch {(x, Vw. vn) D.
5: Compute raw importance scores I via gradient attribution.
6: Compute weights (wx" and [w]] by mixing normalized scores with Gaussian prior Pprior:
W <- XT + - A)
7:
Compute weighted DPO log-ratio:
Arioken f- E, wx"" log mo (2.35)) Ta
E, wt log (a Tarls) Tar(o/12.0)
8:
Generate the anchor response yf: y² ~ To (v xr, (1-1) Compute triplet log-ratio Armipler (Eq.
9:
10:
Compute weighted DPO loss: LDPO-W + log @ (3Arrokn)) Compute triplet loss: Luipler + max (0, A Artrilet + a). Aggregate losses: LII-DPO + LDPO- Criplet-
12:
Update t- 0 V CTLDD-
14:
end for
Output: n
To show the superiority of TI-DPO, we first introduce the following lemma.

---

Published as a conference paper at ICLR 2026
Lemma 1 (Variance Reduction). Consider reward signal governed by a sparse set of critical tokens, such that the subset of non non-critical tokens N contributes only independent zero-mean noise e, with variance o.. Provided that the importance weights for these non-critical tokens are suppressed such that wi < 1 for all & E NN the variance of the TI-DPO estimator (TT--DO)) is strictly lower than that of the standard DPO estimator bro). i...
anDro < TPP
(15)
The proof is detailed in Appendix A.1 Lemma establishes that by suppressing the weights of non- critical tokens, TI-DPO effectively filters out stochastic noise, resulting in a strictly lower variance for the reward difference estimator. The following theorem strictly proves the theoretical advantages of this improvement the loss function level. With Lemma
|
the total loss of TI-DPO will be
significantly lower than the original DPO loss. Denote Arglobal = log Ko (z)
(er(1)/22) (z) log no (M)2) from =
Eq.03 For simplicity, we abbreviate Arwken (z, yws 31, W w!) as Arwken then we have:
Theorem (Tighter Loss Bound). Assuming the preference optimization objective is a strictly con- vex function and the reward difference estimation is unbiased, the expected loss of TI-DPO (LT--PP))
is strictly upper-bounded by that of DPO (Lpro) minus a term proportional to the variance reduc- tion. Specifically:
1 - 5KD0z
LTI-PO S LDPO
(16)
where K. >> O represents the lower bound of the loss function's 's local curvature, and 4 OPO TT-DPO is the positive variance reduction term derived in Lemma 1.
The proof is shown in Appendix A.2 In the experiments presented in the Appendix B.4, we com- pared the loss function convergence processes of the TI-DPO and DPO, thereby further substantiat- ing Theorem 2 Furthermore, we provide theoretical justification for the superiority of the policy learned by TI-DPO. In Theorem [ we demonstrate that TI-DPO utilizes the limited KL constraint more efficiently by concentrating probability mass on critical tokens. The proof is shown in Ap- pendix A.3
Theorem (Superiority of Optimal Policy). Let TDPO and TTI-DPO be the optimal policies de- rived from minimizing the DPO and TI-DPO objectives, respectively. Under fixed total KL diver- gence constraint Krotal, the expected true reward of the TI-DPO optimal policy is strictly lower- bounded by that of the DPO policy, i.e.,
Bysxnoool (r" (x, 2 Ey (x. v)] + 8
(17)
where 8 > represents the gain derived from optimizing the decomposition of KI. divergence, specifically by minimizing the divergence component non-critical tokens.
5 EXPERIMENTS
5.1 EXPERIMENTAL SETTINGS
Datasets and base settings. The benchmarks we use include knowledge-based tasks (MMLU (Hendrycks et al. 2020)), mathematical reasoning (GSM8K (Cobbe et al. 202I). MATH catruction-followinn (IFEval (Zhou al. (Ql3] ), and code generation (HumanEval (Chen et al. (02T)). Additionally, TruthfulQA (Lin et al. 2021) detects the authentic- ity of the model's s answers through adversarial questions. For other detailed hyperparameter settings,
Hendrycks et al. (021)),
please refer to Appendix B.5
Comparative algorithm. We compared the TI-DPO with baseline alignment methods such as SFT. DPO, IPO Azar et al. 2024), KTO (Ethayarajh et al.) 2024), SimPO (Meng et al.] 2024). TDPO Zeng et al. 2024). CPO (Feng et al. 2025). TPO (Sacidi et al. 2024). TIS-DPO (Liu et al. 2024a),
Logic-RL RL (Xie et al. 2025), cDPO (Lin al. 2024) and GRPO (Shao et al. 2024). We select three
models (Llama-3,2) 3.2-BB (Grattationi et al. 2024). Llama-3.1 (Grattafiori et al. 2024), Mistral-7B-
v0.3 (Jiang et al. 2023) as baselines.
7

---

Published as a conference paper at ICLR 2026
Table 1: Average scores of each fine-tuning method across three base models
Method
MMLU
GSM8K
GPQA
HumanEval
TruthfulQA
IFEval
Avg
SFT
64.0
68.0
22.7
$9.3
55.5
70.5
56.7
DPO
65.3
69.3
24.0
61.0
56.7
70.0
57.7
IPO
63.0
65.3
20.3
57.3
52.7
66.7
54.2
KTO
66.3
70.3
25.3
62.0
57.7
70.5
58.7
SimPO
63.5
64.7
21.8
58.2
54.2
64.7
54.5
TDPO
65.0
68.2
23.5
60.3
56.3
68.5
57.0
CPO
67.3
70.7
26.0
62.8
58.3
71.3
59.4
TPO
72.7
27.7
63.7
59.0
72.7
60.7
Logic-RL
63.8
73.8
23.7
61.0
55.6
69.3
57.9
cDPO
66.1
70.1
25.1
61.9
57.6
70.4
58.5
TIS-DPO
69.3
70.5
24.5
65.5
62.5
74.0
61.1
GRPO
70.7
75.7
28.0
64.3
59.9
74.0
62.1
TI-DPO
70.0
73.0
26.0
67.0
62.0
75.7
62.3
5.2 PERFORMANCE COMPARISON
As shown in Figure we conduct analysis of the performance for TI-DPO and baseline methods across training steps on the TruthfulQA (reliability assessment) and IFEval imstruction-followinng tasks with Llama-3.1-8B model. In the TruthfulQA benchmark (Figure [a). TI-DPO demonstrates a steady improvement in accuracy as training steps increase, surpassing all baselines by the final epoch. For the IFEval task (Figure lb). TI-DPO also shows dominant performance trend. This highlights TI-DPO's s effectiveness learning through token-level importance weighting and triplet loss, which explicitly guides the model avoid generating misleading content.
3.550
3.600
0.525
Accuracy
3.500
Wehcc
3.550
0.755
K00
3.125
SmP0
TDPO
TDPO
120
3775
CPO
TPO
IPQ
2,455
GRI
GRPO
T1-PP
T1.2P0
16:00
200ce
40502
10000
33000
Training Steps
Training Steps
(a) TruthfulQA Performance vs. Training Steps
(b) IFEval Performance vs. Training Steps
Figure 1: Accuracy trends with training steps for different methods TruthfulQA and IFEval tasks on LLaMA-3.1-4 The performance comparisons of SFT, DPO, IPO, KTO, SimPO, TDPO, CPO, TPO, GRPO, and TI-DPO are illustrated.
As shown in Figure 2, TI-DPO exhibits significantly better performance in Reasoning, Instruction- Following, and Reliability dimensions compared to the corresponding instruction variants with each base instruction. Here, the base instruct refers to the foundational instruction-tune A.233B-Instruce Llama-3. 1-B....-BBBBB -SB-Instruct, Mistral-7B-Inn -ann-----sttuut--0..3 serving as the baseline for com- paring the effectiveness of TI-DPO and other fine-tuning methods. The scores of TI-DPO in other aspects are roughly equal or slightly higher than others. Table Tateeerreenttt average scores of each fine- tuning method across three base models, clearly demonstrating our method's 's advantages in general tasks and specific scenarios. The specific score comparison table under the three base models is placed in Appendix C
models (Llama-
8

---

Published as a conference paper at ICLR 2026
La 2-38-instruct
Llama-33. 1a-ma-31--8--nntttee
Distral-7B-instrucc--o
Reasoning
Eox Harzo
Reasoning.
Reasoning
18380
6.1
Code
General
Code
General
Code
General
Instr-follow
Reliablity
Inst-follow
Reliability
Instr-f
Reil ability
Figure 2: Multi-dimensional normalized score of TI-DPO compared with other base instruction models across categories.
Table 2:
Ablation study scores: the full TI-DPO vs. base instruction model (Llama-3.2-3B-
Instruct) with other weight and triplet conditions
Method
General
Math
Reasoning
Code
Instr-Follow
Reliability
Base Instruct (Baseline)
63.4
77.7
26.6
28.0
51.5
76.8
Full Method (TI-DPO)
65.4
80.7
34.6
33.0
63.5
86.8
No Triplet Loss
64.0
79.0
32.0
31.0
60.5
83.0
Uniform Weight
64.0
78.2
30.5
29.0
58.0
80.0
Random Weight
63.7
77.8
28.0
28.5
55.0
78.0
No Gaussian Prior
64.5
79.7
32.7
31.5
60.0
82.5
Softmax Prior
64.2
78.8
31.8
30.0
59.0
81.0
5.3 ABLATION EXPERIMENT
To verify the distinct contributions of the importance guidance and triplet loss, Table 2 presents an ablation study with Llama-32-33B-Instmuct The results validate the effectiveness of our design choices: compared to the Random Weight and Uniform Weight settings, the Full Method achieves the highest scores across all six dimensions. Specifically, the importance of the Triplet Loss is evidenced by the drop in Math (80.7 + 79.0) and Code (33.0 + 31.0) scores when is removed. Similarly, ablating the Gaussian Prior leads a notable decline in Reliability (86.8 -> 82.5).
5.4 ADDITIONAL EXPERIMENTS
Case Study. case study in Figure 3 on a medical query (see Appendix C.1 for details) demon- strates that, given the user prompt "I have headache, what should do?" TI-DPO effectively signs higher importance to safety-criticll tokens (e.g., "medical attention". "promptly") in preferred responses, while penalizing risky suggestions (e.g.. "painkillers", "casually"" in non-preferred ones.
Additionally, there are two other cases in Appendix C,2
(Preferred): "Based on your symptoms, it is recommended that you you seek medical attention promptly and avoid self: medicating."
(Intermedia): "A ccordingg to your description, it is advised get more rest, but
C (Non-preferred);
"Don't worry, you can just
take some painkillers
if the symptoms worsen. you
casually, it should be fine."
should consult a doctor."
Figure 3: Case demo of responses prompt have a headache, what should do?". Left: Preferred case. Middle: Intermediate case. Right: Non Non-preferred case. The darker color indicates higher weight.

---

Published as a conference paper at ICLR 2026
4 illustrates how TI-DPO dynamically adapts token importance weights based on task characteristics. For tasks relying on a few critical symbols (GSM8K, GPQA), weights concentrate in the [0.2. 0.5] range. Conversely, in tasks demanding strict adherence to safety or instructions (TruthfulQA, IFRval), weights shift to a higher [0.6, 0.8) interval. Meanwhile, for comprehensive tasks covering broader content (MMLU, HumanEval), the weight distribution is naturally more dispersed.
Distribution of Weights. Figure
GSMBK
TruthfuIQA
MMLU
Baxple
8000
Token Frequeng
15000
Token Frequency
1000
Token Frequency
6000
750
10000
4000
5000
250
2000
0.0
0.2
0.6
1.0
0.0
1.0
0.0
0.4
1.0
Importance nce Weight
Importance Weight
Importance Weight
GPOA
HumanEval
IFEval
5000
Boxplct
250
Boxplot
Bosolct
400
Token Frequency
Token Token Frequency
200
loken Frequency
300
3000
250
2000
200
1000
50
100
8.0
0.2
1.0
8.0
Importance Weight
Importance Weight
Importance Weight
Figure 4: Distribution patterns of gradient-based token importance weights in six benchmark tasks (GSM8K, TruthfulQA, MMLU, GPQA, HumanEval, IFEval
Pearson correlation coefficient. To investigate the effectiveness of hybrid weighting mecha- nism, we also conducted Pearson correlation coefficient analysis, with full results and methodology presented in Appendix [.I]] Our analysis reveals a moderately strong positive correlation (r 0.65) at the task level between average token importance and performance improvements, indicating that TI-DPO is particularly effective on tasks with concentrated weight distributions.
Robustness and Generation Diversity. We validate the robustness and generative diversity of TI- DPO, which can be seen Appendix B.3 To evaluate the model's stability, we tested accuracy under varying label noise levels. TI-DPO maintains superior performance compared to DPO and TPO as noise increases.
Sensitivity of Hyperparameters We conducted sensitivity analyses for the weight-mixing param- eter and KL weight a in Appendix B.5 Performance remains stable for E [0.3, 0.7]. We have provided the specific values of the hyperparameters for this project, as shown in Appendix B.5
6 CONCLUSION
We introduce TI-DPO, an optimization framework that effectively bridges the alignment gap be- tween LLMs and human value systems. By introducing a mixed weight calculated collaboratively by gradient attribution and Gaussian prior, TI-DPO effectively overcomes the limitations of tradi- tional DPO methods at the token level and their sensitivity noise. On this basis, the triplet loss structure provides more refined guidance for model optimization. Theorem 2 and Theorem theoret- tteore- ically illustrate the superiority of TI-DPO over DPO. The effectiveness of TI-DPO unequivocally demonstrated through extensive experimentation. Our method achieves state-of-the-an average score of 62.3 across a diverse suite of benchmarks, outperforming all baseline methods. As for the limitations, despite its effectiveness in fine-grained alignment, TI-DPO entails computational overhead during training and performs slightly below sequence-leve baselines on holistic reasoning loken-importanee mechanism with group-based op-
tasks. Future work will focus integrating timization methods like GRPO to bridge this gap and further enhance reasoning capabilities. More statements can be found in Appendix D
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This research focuses aligning LLMs with human values ensure beneficial, safe, and harmless interactions. We acknowledge that aligning models using human preference data carries the inherent risk of learning existing stereotypes or biases present the training datasets. However, our proposed TI-DPO framework provides a structural advantage over standard methods by assigning explicit, interpretable importance weights to individual tokens. If the model reinforces a bias, these token weights make the source of the bias explicit and detectable, providing a direct mechanism for bias mitigation. All evaluations in this study were conducted on standard, publicly available benchmarks without involving human subject research.
REPRODUCIBILITY STATEMENT
To ensure full reproducibility of our work, the complete source code, training scripts, and demo are publicly available at https: http:://githbb cow./gracef..nnnggggaoooacaa.o/e Litiii/iigggOO/ We exclusively open-weight base models (e.g., Llama-3.2-3B, Llama-3.1-80 and Mistral-7B-v0.3) and public benchmark datasets. The paper provides comprehensive theoretical proofs and gradient derivations in Appendix [ Furthermore, the detailed implementation steps are clearly outlined in Algorithm and all relevant hyperparameters used in our experiments are explicitly listed Appendix B.5
ACKNOWLEDGMENTS
This work supported by the National Natural Science Foundation of China under Grant 62301559. The authors would also like to thank the anonymous reviewers for their constructive feedback, well our colleagues for their insightful discussions during the development of the TI-DPO framework.
REFERENCES
Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. Towards better understanding of gradient-based attribution methods for deep neural networks. arXiv preprint 1:Xiv::711.061000 2017.
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learn- ing from human preferences. In International Conference on Artificial Intelligence and Statistics, pp. 4447 4455. PMLR, 2024.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint trXiv:2204.05822 2022.
Mohamad Ballout, Ulf Krumnack, Gunther Heidemann, and Kai-Uwe Kühnberger. Efficient knowl- edge distillation: Empowering small language models with teacher model insights. In Inter- national Conference on Applications of Natural Language to Information Systems, pp. 32-46. Springer, 2024.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.0337. 2021.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arX 2021.
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01455 2025.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment prospect theoretic optimization. arXiv preprint arXiv:2402.01300 2024.
11

---

Published as a conference paper at ICLR 2026
Zhili Feng, Dhananjay Ram, Cole Hawkins, Aditya Rawal, Jinman Zhao, and Sheng Zha. Sequence- level large language model training with contrastive preference optimization. arXiv preprint MrXi:::5022164333, 2025.
Mingqi Gao, Xinyu Hu, Xunjian Yin, Jie Ruan, Xiao Pu, and Xiaojun Wan. Llm-based nlg evalua- tion: Current status and challenges. Compulational Linguisties, pp. 1-27, 2025.
Zhaolin Gao, Jonathan Chang, Wenhao Zhan, Owen Oertell, Gokul Swamy, Kianté Brantley, Thorsten Joachims, Drew Bagnell, Jason D Lee, and Wen Sun. Rebel: Reinforcement learn- ing via regressing relative rewards. Advances Neural Information Processing Systems, 37: 2024.
Qi Gou and Cam-Tu Nguyen. Mixed preference optimization: Reinforcement learning with data selection and better reference model. arXiv preprint arXiv:2403.19443 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178,, 2024.
Maosheng Guo, Yu Zhang, and Ting Liu. Gaussian transformer: lightweight approach for natural language inference. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 6489-6496, 2019.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.088
2021.
Jiwoo Hong, Noah Lee, and James Thome. Orpo: Monolithic preference optimization without reference model. arXiv preprint arXiv:2403.07661, 2024.
Jian Hu, Jason Klein Liu, and Wei Shen. Reinforce++: An efficient rlhf algorithm with robustness to both prompt and reward models. arXiv preprint arXiv:2501.03262 2025.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Ana Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.
Fangkai Jiao, Geyang Guo, Xingxing Zhang, Nancy F Chen, Shafiq Joty, and Furu Wei. Preference optimization for reasoning with pscudo feedback. arXiv preprint arXiv:2411.16345 2024.
Saced Khaki, JinJin Li, Lan Ma, Liu Yang. and Prathap Ramachandra. Rs-dpo: A hybrid rejection sampling and direct preference optimization method for alignment of large language models. arXiv preprint arXiv:2402. /0038 2024.
Dongyoung Kim, Kimin Lee, Jinwoo Shin, and Jaehyung Kim. Spread preference annotation: Direct preference judgment for efficient Ilm alignment. In The Thirteenth International Conference on Learning Representations, 2025.
Chengao Li, Hanyu Zhang, Yunkun Xu, Hongyan Xue, Xiang Ao, and Qing He. Gradient-adaptive policy optimization: Towards multi-objective alignment large language models. arXiv preprint arXiv:2507.01915. 2025.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falschoods. arXiv preprint 1xXiv:2109.09955 2021.
Zicheng Lin, Tian Liang, Jiahao Xu. Xing Wang, Ruilin Luo, Chufan Shi, Siheng Li, Yujiu Yang, and Zhaopeng Tu. Critical tokens matter: Teven-level contrastive estimation enhence I1m's reasoning capability. arXiv preprint 2rX1v:2411.19943 2024.
12

---

Published as a conference paper at ICLR 2026
Aiwei Liu, Haoping Bai, Zhiyun Lu, Yanchao Sun, Xiang Kong, Simon Wang, Jiulong Shan, Al- bin Madappally Jose, Xiaojiang Liu, Lijie Wen, et al. Tis-dpo: Teken-level importance sampling for direct preference optimization with estimated weights. arXiv preprint arXiv:2410.04350, 2024a.
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157-173, 2024b.
Wenhao Liu, Xiaohua Wang, Muling Wu, Tianlong Li, Changze Lv, Zixuan Ling, Jianhao Zhu, Cenyuan Zhang, Xiaoqing Zheng, and Xuanjing Huang. Aligning large language models with human preferences through representation engineering. arXiv preprint arXiv:2312.15999 2023.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-free reward. Advances in Neural Information Processing Systems, 37:124198-122235, 2024.
Bac Nguyen, Carlos Morell, and Bernard De Baets. Distance metric learning for ordinal classifica- tion based triplet constraints. Knowledge-Basse Systems, 142:17-28, 2018.
Long Ouyang. Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36:53728-53741, 2023.
Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From q": Your language model is secretly a q-function. arXiv preprint arXiv:2404. 12358, 2024.
Amir Saeidi, Shivanshu Verma, Aswin RRV, and Chitta Baral. Triple preference optimiza- tion: Achieving better alignment with less data a single step optimization. arXiv preprint 166.1 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06344 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. 2024.
Chaoqi Wang. Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. Beyond reverse kl: Gen- eralizing direct preference optimization with diverse divergence constraints. arXiv preprint 16240, 2023a.
Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. Aligning large language models with human: A survey. arXiv preprint arXiv:2307. 12966, 2023b.
Zhiheng Xi, Wenxiang Chen, Boyang Hong, Senjie Jin, Rui Zheng, Wei He, Yiwen Ding, Shichun Liu, Xin Guo, Junzhe Wang, al. Training large language models for reasoning through reverse curriculum reinforcement learning. arXiv preprint arXiv:2402.058800 2024.
Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing Ilm reasoning with rule-based reinforcement learning. arXiv preprint arXiv:2502. 14768, 2025.
Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min-Ye Yen Kan, Timothy P Lillicrap, Kenji Kawaguchi, and Michael Shieh. Monte carlo tree search boosts reasoning via iterative preference learning. arXiv preprint 4455v:2405.0045/ 2024.
13

---

Published as a conference paper at ICLR 2026
Xiaodan Xu, Chao Ni, Xinrong Guo, Shaoxuan Liu, Xiaoya Wang, Kui Liu, and Xiaohu Yang. Dis- tinguishing l1m- generated from human written code by contrastive learning. ACM Transactions on Software Engineering and Methodology, 34(4):1-31,
Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang, Haifeng Zhang, and Jun Wang. Token-level direct preference optimization. arXiv preprini arXiv:2404. 1999, 2024.
Jiajun Zhang, Yang Zhao, Haoran Li, and Chengqing Zong. Attention with sparsity regularization for neural machine translation and summarization. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 27(3):507-517 2018.
Lijun Zhang, Lin Li, Yajie Qi, Huizhong Song, Yaodong Yang. Jun Wang, and Wei Wei Risk-aware direct preference optimization under nested risk measure. arXiv preprint arXiv:2505. 20359, 2025.
Han Zhong, Zikang Shan, Guhao Feng, Wei Xiong, Xinle Cheng, Li Zhao, Di He, Jiang Bian, and Liwei Wang- Dpo meets ppo: Reinforced token optimization for rlhf. arXiv preprint arXiv:2404. 18922, 2024.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction evaluation for large language models. arXiv preprint 1rX0v:2311.07911, 2023.
A THEORETICAL PROOF
A.1 PROOF OF LEMMA 1
In standard DPO, the optimization objective implicitly assigns a uniform weight of 1 to all tokens. Consequently, the estimator aggregates the noise from the entire non-critical set. Assuming the noise terms are independent, the total variance for DPO is the sum of their individual variances:
DrO
Σr. Var(e:) WW
12.
(18)
LEN
where [ denotes the number of non-critical tokens.
TI-DPO, the contribution each token scaled by its importance weight wi. The variance of the weighted estimator becomes:
TI-DPO
Σ wt Var(e:) = oe Σ
w
(19)
IEN
IEN
Given the condition that the weights in the non-critical region are suppressed (w? < 1), the sum of the squared weights is strictly less than the count of the tokens:
Σ
E < Σ 11=WT
(20)
EEN
LEN
Substituting the definitions of of and opo conclude:
oT
oo
(21)
A.2 PROOF OF THEBRREM 2
As shown in Eq.33, the loss function for DPO becomes: LIPO = E - log a (3Arghoau)) Similarly,
the TI-DPO loss is LTI-POO E[- log ((BAr ]. Thus, we define the generic scalar loss function for an estimator R (where R E { [iggohall Ar }) as f(R) log a (3R).
We analyze the properties of (R). The first and second derivatives with respect to are:
f(( -3(1 ((RR) ("(R) = (2 (3
(22)
(23)
14

---

Published as a conference paper at ICLR 2026
Since the sigmoid function satisfies af-) (0, 1), the second derivative f is strictly positive for any finite R. Therefore, f f(R) is a strictly convex function. Let k = inf (() > denote the lower bound of the curvature within the effective optimization region.
We employ a second-order Taylor expansion of the loss function f(R) around the true (expected) reward difference μ.. Let R represent the estimator (either Aiglobal Arwken). The expansion with the Lagrange remainder is:
1 -
((R) f(w) ( ())
(24)
where & lies between R and Jr. Taking the expectation El[-] on both sides, and applying the unbiased- ness assumption [R[]] = μ (which eliminates the linear term):
[[)( = f(u) } 2
(E)(
(E)(R 10
(25)
Using the curvature lower bound K, we can lower-bound the expected loss terms of the estimator's variance o² Var R) E[(R p)
1
[[S(R)) 2 f(1) + RO?
(26)
The difference is approximated by the difference in variances:
1
LDOO LTIDRO R 2 ()) - ot
(27)
Applying the lower bound x and substituting the strictly positive variance reduction term Aga Tro - oTD derived in Lemma 1, we obtain:
LDO LTI-POO 2
2 5KA02.
(28)
Since k > 0 and o₂ > 0, this completes the proof.
A.3 PROOF OF THEOREM 3
Proof. We begin by defining the Sparse Criticality Assumption: For a given input x, the token indices of response y partition into critical set C and non-critical Se, where [C] < [W]. The true reward function r" y) depends solely on tokens in C. Consequently, any deviation from the reference model nel on tokens in A incurs a KL cost without yielding any reward gain.
The total KL divergence constraint Kaota decomposes token-wise via the chain rule:
T
(TI(T|
Σ
Eystno
(xL [)]l [x))] = Ke + KN,
(29)
where Ke and KN represent the KL divergence allocated critical and non-critical tokens, respec- tively.
(x In standard DPO, the optimal policy takes the form #pro(y(æ) o mee .r) exp (2ro(a, v)) Since
the implicit reward model Te is optimized the sequence level, it inevitably captures spurious correlations, distributing non-zero gradients across all tokens, including those in N. This results in "KL divergence waste", where TDPO diverges from Nert non-critical tokens, implying K-DPO > for some >> O. The effective KL divergence available for critical tokens is thus limited to KDPO < Krotal - c.
In contrast, TI-DPO incorporates token importance weights wy. We assume the weights align with the sparsity structure, such that w. R 0 for t E V (due to low gradient attribution from non-critical tokens). This weighting suppresses the update signal non-critical tokens, causing the policy to default to the reference, i.e., TTIDPO(-ly x) ss mer(--ly<* 2) for E N. Consequently, Ti- DPO minimizes the wasted KL divergence, K KIDDPP as allowing nearly the full constraint to be allocated to the critical set: KTI-DPO xs Kiotal-
15

---

Published as a conference paper at ICLR 2026
Since the maximum achievable expected reward f(K) is strictly increasing and concave function of the KL divergence allocated to reward-relevant tokens (a property derived from the distoiitootion nature of the RL objective), the larger effective allocation of TI-DPO implies higher upper bound on the expected reward. Specifically,
Em [r'] mm [r'] = f (KT-DDPO - f f(K2 2 (Kioa) f (Kiotal c) _ 8
(30)
This confirms that TI-DPO achieves a higher expected reward by efficiently reallocating the KL constraint.
A.4 GRADIENT ANALYSIS OF Loss
According to Eq. 12) we have LDPO- = log o Aroken), where (xx) Ite is the sigmoid
function with derivative o' d'(z) = a(x)(1 o())). Taking the gradient of Eq. 12
we have
1
V VoLDo
o' (BArinnen) 38 VVn A A Tioken
0 (B3Arwken)
(31)
-3(1 - o rroken V A token-
Here, we expand V Aroken
Tv
Te (ym (x, yo
PpArioken Pooken
w V
log
nea (3 (r, v
t=1
(32)
T
Te (vi br, 311 w1-h)
Σ
W V
log
Tref (if (x, 1/1 ()
1=1
Since Ter is fixed, there is V log de Taf
Tely Toly"| (y V VoTT(U" Ve log no (v lx, (e
Thus, Vo w becomes:
Te
VOLDrO- =-30 oBBrr(Brrre (oken) )
w, Vo Vogooooooll
(33)
T
W Vo log ,,
As for the gradient of Luiple expand the gradient:
E [max (0, Arriple + a)] assuming Armipee
we
V Th | by 12 - Vo
V Lariple
c - dellz
(34)
na (n War 15
Gt = log 10(2/12,,0 12,01 12
d, log Jely 12,00 1z,00 <t
where b, = log
mar + for simplicity. Differenti-
ating the squared terms, we have
V ldd ball2 = 2(d, - b.) (Vod, Vob.).
(35)
and
Vo lla - de11 = 2(c d,) (V Voc - Vode).
(36)
Then, substitute the definitions of bi, Ct2 d, and use Vo log Jeet 0:
Vol, Vo log n (11 x y )
(37a)
V pc Vo log N (1) (r, "t)
(37b)
V Vede = Vo log No (1/ 12. y "t).
(37c)
Thus, V Cuipler is:
Tx
T
VeLuriptet
>>0 t=1
(d. b.) (Vod. - Vob.) - >
(c. de) (V - Vede).
(38)
t=l
16

---

Published as a conference paper at ICLR 2026
Substituting the derived gradients:
Tv
n V log to 20
VOLTIDPO B(1 ( laken ))(
wi Vo log To 1z, yu
Σ
Tx + 27
T
[d (Vod Vobe)
Σ
ca - -ddd) - Voda)]
(39)
B ADDITIONAL EXPERIMENTAL RESULTS
We present some additional explanations and experimental results.
B.1 PEARSON CORRELATION COEFFICIENT
Table B1 presents multiple metrics for six tasks, namely GSM8K, GPQA, TruthfulQA, IFEval, MMLU, and HumanEval, including QI, Q2 (median), Q3, average weight (mid-point of the inter- val), performance improvement A Acc (%), and sample-level Pearson r (Top 5 weights us accuracy). Among them, the "sample-level Pearson r" is calculated based on the average weights of the Top-5 tokens for 100 questions each task and the correctness of the answers to these questions (binary values), which reflects the microscopic internal correlation. The performance improvement Accc is calculated from the average accuracy gain of TI-DPO compared to DPO.
Table B1: Distribution of Token Importance Weight, Performance Improvement, and Sample-level Pearson Correlation Coefficient in Each Task
Performance Improvement Axcc (%)
Q2 (Median)
Average weight
Sample-level Pearson Top-5 weight vs accuracy rate
Task
Q1
Q3
GSM8K
0.22
0.33
0.45
0.34
+4.7
0.29
GPQA
0.18
0.28
0.42
0.30
+5.0
0.22
TruthfulQA
0.70
0.75
0.85
0.78
+5.3
0.31
IFEvl
0.68
0.75
0.85
0.77
+5.7
0.27
MMLU
0.30
0.50
0.70
0.50
+4.7
0.18
HumanEval
0.48
0.60
0.70
0.59
+6.0
0.35
The calculation process is divided into the calculation of microscopic and macroscopic Pearson correlation coefficients:
Microscopic calculation: To calculate the sample-level Pearson correlation coefficient, first, randomly sample 100 samples from the test set without replacement for each task. Subsequently, for each sample, the TI-DPO model perform forward and backward propagation to calculate the importance weight of each token, extract the Top-5 tokens with the highest weights, and calculate their average. Then, record the consistency between the model's answer and the standard answer for this sample, marking it as when correct and 0 when incorrect. Finally, substitute the obtained sequence of average weights the Top- 5 tokens and the sequence of answer correctness into the Pearson correlation coefficient (q.(40) for calculation, where w and Å in the formula represent the means of the two sets of data, respectively.
E. (w: w/ (A, A) (w, 2) Σ, ( - A
(40)
The results show that the correlation coefficient 0.35 for the HumanEval task. which is the highest among all tasks, indicating that in the code generation task, the correlation between token importance and the probability of answering correctly is the strongest. On
17

---

Published as a conference paper at ICLR 2026
the other hand, the correlation coefficient for the MMLU task is the lowest, approximately 0.18, suggesting that in the multi-task and multi dustiiiiicpplinary test, the relationship between token importance and the conrectness of a single question is relatively weak.
Macroscopic calculation: Based on the average weights (approximated by the median) and performance improvements of the 6 tasks, first construct the average weight vector: w = [0.33, 0.28, 0.75, 0. 75, 0.50, 0.60] (Order: GSM8K, GPQA, TruthfulQA, IFEval, MMLU, HumanEval), and the performance improvement vector A = [4.7, 5.0, 5.3, 5.7, 4.7, 6.0]. Then substitute them into the Pearson correlation formula (Eq. 40p). In the specific calcu- lation steps, first obtain V R 0.535 and A x 5.4, and then substitute each value of the two sets of vectors into the formula in turn of summation and square-root operations. The final result shows that the overall Pearson correlation coefficient r ~ 0.65 at the task level, indicating a moderately strong positive correlation between the average token importance of the six tasks and the performance improvement.
The coefficient difference between the overall correlation at the task level and the correlation at the single-task sample level stems from their fundamental differences the computational level. The overall correlation at the task level takes six tasks as the sample size and analyzes the corresponding relationship between the "average token weight" of each task and the "overall performance improve- ment", essentially reflecting the macro correlation across tasks. Due to the significant differences in the weight distribution centers and improvement amplitudes of different tasks, this "inter-task difference" tends to magnify the correlation, resulting in correlation coefficient r = 0.65.
In contrast, the correlation the single-task sample level takes 100 questions in each task as the sample size, focusing on the correlation between the "average Top-5 token weight" of each question within the same task and the correctness of the answer that question. Since the samples fall within the same distribution range, the signal between the weight and the correctness of the answer is weak, and it is affected by noises such as the diversity of prompts, fluctuations in question difficulty, and the randomness of token gradients. Therefore, the correlation coefficient is only between 0.18 and 0.35. This result reasonable: the overall correlation at the task level indicates that TI-DPO has more significant improvement tasks with concentrated weight distributions. The weak micro- correlation at the single-task level shows that token importance is only one of the factors affecting the correctness of single question.
B.2 EXPERIMENTAL RESULTS ACROSS BASE MODELS
This subsection presents evaluation results of TI-DPO on three different base models (LLaMA- 3.2-3B, I.L.MA-3. 1-8B, Misvral-7B-v0.33, comparing it with baseline methods like SFT, DPO, and other variants to validate its effectiveness and robustness across model scales and tasks.
Table B5 presents the evaluation results of TI-DPO on the LLaMA-3.2-33 model, a lightweight 3B- parameeet model, showing that TI-DPO achieves notable scores of 68.0 in HumanEval and 82.0 in IFEvll, outperforming baselines like DPO (62.0, 78.0) and SFT (61.0, 77.4) significantly. Table B6 evaluates TI-DPO on the LLaMA-3.1-8B model (8B parameters), where it excels with Eval score of 86.0, surpassing GRPO (85.0), and achieves 80.0 in HumanEval and 63.0 in TruthfulQA, outperforming DPO (74.0, 58.0) and GRPO (78.0, 62.0); with average score of 71.1, it closely matches the best baseline, validating its capability to handle complex instructions and improve gen- erative reliability on medium-scale models. In Table B7 TI-DPO achieves 66.0 in TruthfulQA and 59.0 in IFEval, significantly higher than DPO (60.0, 50.0), and surpasses GRPO in HumanEval (53.0 vs. 51.0).
B.3 ROBUSTNESS AND GENERATIVE DIVERSITY
To evaluate the model's stability, we tested accuracy under varying label noise levels (0%, 10%. 20%, 40%). As shown in Table B8) TI-DPO maintains superior performance compared to DPO and TPO noise increases.
Additionally, we assessed generative diversity using Self-BLEU and Distinct metrics (Table B9 TI-DPO achieves lower Belf-BLEU and higher Distinct scores, indicating richer vocabulary and more diverse response generation. The Self-BLEU, which is used to describe similarity. decreases, and the Distinct-2/ / Distinct-4, which describes the richness of vocabulary, increases significantly.
18

---

Published as a conference paper at ICLR 2026
Table B2: Token Importance Assignment of A
Token Weight
Based 0.05
your 0.05
symptoms 0.18
is
recommended 0.20
that 0.02
0.05
0.03
0.03
Token
you 0.02
seek
medical 0.87
attention 0.85
promptly 1.00
and 0.03
avoid 0.92
self self-medicating 0.89
Weight
0.93
Table B3: Token Importance Assignment of B
Token Weight
According 0.04
your 0.04
description
it
is
advised 0.13
get 0.06
0.04
0.07
0.03
0.03
0.02
Token Weight
more 0.06
rest
symptoms 0.18
worsen 0.88
you 0.03
should 0.82
consult 0.90
doctor 0.95
0.11
Table B4: Token Importance Assignment of C
Token Weight
Don't 0.21
worry 0.18
you
can
just 0.09
take 0.03
some 0.09
0.04
0.04
Token Weight
painkillers 0.91
casually 1.00
it 0.02
should 0.06
be 0.03
fine 0.97
Table B5: LLaMA-3.2-3B evaluation
METHOD
MMLU
GSM8K
GPQA
HUMANEVAL
TRUTHFUL_QA
IFEVAL
AVG
SFT
63.0
78.0
33.0
61.0
51.0
77.4
60.6
DPO
64.0
79.0
34.0
62.0
52.0
78.0
61.5
IPO
62.0
76.0
31.0
59.0
49.0
76.0
58.8
KTO
65.0
80.0
35.0
63.0
53.0
78.5
62.4
SIMPO
64.0
78.0
33.5
61.5
51.5
74.0
60.4
TDPO
64.5
78.5
34.0
62.0
52.0
76.5
61.2
CPO
66.0
79.5
35.5
63.5
53.5
79.0
62.8
TPO
67.0
82.0
39.0
64.0
54.0
80.0
64.3
GRPO
69.0
85.0
38.0
63.8
53.8
81.0
65.1
TI-DPO
68.0
81.0
34.5
68.0
57.0
82.0
65.1
Table B6: LLL.MA33.1-8B evaluation
METHOD
MMLU
GSMSK
GPQA
HUMANEVAL
TRUTHFULQA
IFEVAL
AvG
SFT
69.0
84.0
30.0
72.0
56.0
80.0
65.2
DPO
70.0
85.0
32.0
74.0
58.0
82.0
66.8
IPO
68.0
80.0
27.0
70.0
53.0
77.0
62.5
КTO
71.0
86.0
34.0
75.0
59.0
83.0
68.0
SIMPO
68.5
78.0
28.0
71.0
54.0
75.0
62.4
TDPO
69.5
83.0
31.0
73.0
57.0
81.0
65.8
CPO
72.0
86.5
35.0
76.0
60.0
84.0
68.9
TPO
73.0
88.0
36.0
77.0
61.0
85.0
70.0
GRPO
75.0
90.0
37.0
78.0
62.0
85.0
71.2
TI-DPO
74.0
89.0
34.5
80.0
63.0
86.0
71.1
19

---

Published as a conference paper at ICLR 2026
Table B7: Mistral-7B-v0.3 evaluation
METHOD
MMLU
GSM8K
GPQA
HUMANEVAL.
TRUTHFUI.QA
IFEVAL
AVG
SFT
60.0
42.0
5.0
45.0
59.5
54.0
44.2
DPO
62.0
44.0
6.0
47.0
60.0
50.0
44.8
IPO
59.0
40.0
3.0
43.0
56.0
47.0
41.3
KTO
63.0
45.0
7.0
48.0
61.0
50.0
45.7
SIMPO
58.0
38.0
4.0
42.0
57.0
45.0
40.7
TDPO
61.0
43.0
5.5
46.0
60.0
48.0
43.9
CPO
64.0
46.0
7.5
49.0
61.5
51.0
46.5
TPO
65.0
48.0
8.0
50.0
62.0
53.0
47.7
GRPO
68.0
52.0
9.0
51.0
64.0
56.0
50.0
TI-DPO
66.0
47.0
7.0
53.0
66.0
59.0
49.7
Table B8:
Accuracy under varying noise levels.
Noise Level
0%
10%
20%
40%
DPO
69.3
67.5
64.8
60.1
TPO
72.7
71.1
69.0
65.7
TI-DPO
73.0
72.2
70.8
68.3
This reflects that TI-DPO can generate more differentiated responses and improve the diversity of responses. We believe the token-level fine-graines guidance prevents the model from collapsing into a few high-likelihood patterns, thereby promoting a wider range of expressions.
Table B9: Text generation diversity metrics.
Method
S-BLEU
Dist-2 t
Dist-4
Ent 1
DPO
34.2%
0.87
0.78
2.41
TPO
32.9%
0.89
0.80
2.46
TI-DPO
30.1%
0.93
0.84
2.59
B.4 CONVERGENCE BASED ON THEOREM 2
We compare the training loss curves of DPO and TI-DPO in Table B10 where TI-DPO demonstrates consistently tighter loss bound.
B.5 SENSITIVITY ANALYSIS AND HYPERPARAMETERSS
We conducted sensitivity analyses for the weight-mixing parameter À (Table B BII and KL weight (Table B12). Performance remains stable for [0.3, 0.7]. Table B13 lists the final hyperparame- ters used in our experiments.
C CASE STUDY
C.1 ANALYSIS OF MEDICAL-RELATED CASE
the TI-DPO framework, response A corresponds to the preferred response yo in the dataset, which represents the high-quality, human-preferred output aligned with safety, professionalism. and correctness (e.g., "seek medical attention promptly" in the medical case). These tokens are assigned high importance weights to prioritize critical elements in human judgments. As show in Table [2 in Appendix, critical tokens like "seck" (0.93), "medical" (0.87), "attention" (0.85), "promptly" (I.00), "avoid" (0.92), and "self-medicati"" (0.89) receive high importance weights, reflecting their role
20

---

Published as a conference paper at ICLR 2026
Table B 10: Training loss comparison between DPO and TI-DPO over epochs.
Epoch
0.00
0.27
0.55
0.82
1.09
1,36
1.64
1.91
2.18
2.45
2.73
3.00
DPO
0.700
0.545
0.425
0.335
0.265
0.210
0.170
0.140
0.115
0.100
0.085
0.075
TI-DPO
0.640
0.480
0.365
0.280
0.215
0.165
0.130
0.105
0.090
0.075
0.060
0.050
Table B11: Sensitivity analysis of X.
0
0.1
0.3
0.5
0.7
0.9
1.0
GSM8K
79.8
80.5
80.7
80.8
81.0
80.6
80.2
IFEval
80.8
81.3
81.9
82.0
81.9
81.5
81.0
in ensuring safety and compliance with medical standards. These weights act as a "spotlight" to prioritize tokens that most influence human judgments, such as emergency actions and avoidance of self-treatment.
Response C in Table [4 corresponds to the less preferred response St, representing low-quality or risky outputs that deviate from human preferences (e.g., "take painkillers casually" in the example). The model includes high-risk tokens like "painkillers" (0.91), "casually" (1.00), and "fine" (0.97), which receive peak weights due to their potential to mislead users into unsafe self medecctiiii.... TI- DPO's gradient-basee attribution mechanism identifies these tokens as critical for preference mis- alignment, suppressing their influence during generation.
In Table B3 response B represents intermediate generated response (e.g., "get more rest... sult a doctor"), which is neither the top preferred nor the worst case. In the triplet loss structure, acts as anchor that is guided to approach Yo (A) and distance from i (C). Key tokens like "worsen" (0.88), "should" (0.82), "consult" (0.90), and "doctor" (0.95) have elevated weights but less intense than those in A, indicating their secondary importance in guiding less urgent but still reasonable advice. By incorporating B, TI-DPO promotes more nuanced optimization, where intermediate outputs are refined better match human preferences through token-level importance weights and triplet constraints.
C.2 OTHER CASES
Case Study 2: Financial Advice Scenario
In this case, briefly present the case of financial advice. In the preferred response, the high weights align with safety constraints and expert-domain concepts (e.g.. "certified advisor"), the in- termediate response highlights the generic helpfulness, and the non-preferred emphasizes weights are assigned to ballucinations or unsafe suggestions (e.g., "deal with details later"), effectively filter- ing out noise and risk from the learning signal. This confirms that the "hybrid weighting mechanism" acts semantic filter, prioritizing content quality over mere fluency.
Prompt: am overwhelmed by debt. What is the quickest way to get more money?"
A (Preferred): "To manage your current financial obligations, it's s best to contact certified advisor and avoid taking high-risk loans without professional guidance."
(Intermediate): "You may want to review your expenses, and might help adjust your budget or seek assistance if necessary."
C (Non-Preferrred): "It should be fine to just take out another loan quickly; you can deal with the details later."
Figure B1: Case demo of responses to prompt "I am overwhelmed by debt. What the quickest way to get more money?". Left: Preferred case. Middle: Intermediate case. Right: Non Non-preferred case. The darker color indicates higher weight.
21

---

Published as a conference paper at ICLR 2026
Table B12: Sensitivity analysis of a.
Table B13: Hyperparameter settings.
GSM8K
IFFval
Avg
Hyperparameter
Value
72.4
75.0
73.7
TDPO KL Weight (a)
0.5
0.2
72.8
75.5
74.2
DPO Temperature (.3)
0.1
0.3
73.0
75.7
74.4
Triplet Loss Weight (?)
0.1
0.5
72.6
75.2
73.9
Hybrid Weight Mix (A)
0.7
Case Study 3: Software Debugging Scenario
In the software debugging scenario, further validate the performance of TI-DPO in code gener- ation tasks. Faced with a user query regarding an "index out of bounds" error, the model assigns the highest saliency weights key terms in the preferred response (Response A) related input validation and boundary checking ("validate parameters". "ensure bounds"). In contrast, for Re- sponse B, the model offers only heuristic debugging methods (e.g.. "printing variables"). For the non-preferred response, TI-DPO suppresses potential misleading guidance by suggests "ignoring the error" or using unsafe workarounds.
Prompt: "My code throws an index out of bounds error. How do I fix it?"
A (Preferred): "To fix the error, validate the imput parameters and ensure the index stays within bounds before accessing the array."
B (Intermediate): "The issue may be caused by incorrect input, so you can try printing the values to see what wrong."
C -No--rreferrred:: "You can just ignore the error and force the array to resize automatically; should work most of the time."
Figure B2: Case demo of responses to prompt "My code throws an index out of bounds error. How do I fix it?". Left: Preferred case. Middle: Intermediate case. Right: Non-prefeee Non-preferred case. The darker color indicates higher weight.
D LIMITATIONS AND FURTHER DISCUSSION
In this section, we discuss the computational overhead of our method, analyze performance trade- offs reasoning-heavy benchmarks, and address the implications for bias amplification.
D.1 COMPUTATIONAL OVERHEAD
TI-DPO introduces computational overhead that is not present in standard DPO. This cost stems primarily from our hybrid weighting mechanism, which requires one additional backward pass per sequence to compute the gradient attribution for token importance. Consequently, the computational cost per training iteration is approximately double that standard DPO (approx. 2x training time).
However, this overhead scales linearly with sequence length, akin standard training backward pass. We consider this an explicit trade-off: accepting a fixed, modest computational cost during the training phase in exchange for significant gains in alignment accuracy, fine-grained control, and optimization stability. Significantly, this is an overhead exclusive to the training phase; hence, does not affect inference speed or lateney.
D.2 PERFORMANCE ANALYSIS ON REASONING-HEAVY BENCHMARKS
As shown in Table I while TI-DPO achieves state-of-ther performance instruction following and safety tasks, a performance gap compared to sequence-level baselines (c.g., GRPO, TPO) on knowledge-intensive (MMLU, GPQA) and mathematical reasoning (GSM8K) benchmarks. This is
22

---

Published as a conference paper at ICLR 2026
because reasoning heavy tasks often depend holistic, sequence-level logical consistency; meth- ods such as GRPO and TPO may have inherent advantages due to their whole-response optimization. However, TI-DPO where even a very slight misalignment with human preferences should be avoided, such as Instruc- tion Following (IFEval), Truthfulness (TruthfulQA), and Code Generation (HumanEval).
explicitly designed for fine-gratn fine-grained semantic control. It outperformed tasks
D.3 BIAS AMPLIFICATION AND MITIGATION
Like standard DPO, if the training preference data contains stereotypes or biases, TI-DPO may learn these patterns. However, we argue that TI-DPO offers a structural advantage over standard DPO in handling such biases. Standard DPO tends to silently reinforce spurious correlations (c.g.. associ- ating specific genders with specific professions) without providing any interpretability. However, if the model reinforces stereotype, TI-DPO will assign high importance weight the specific biased tokens (e.g., pronouns or adjectives), making the source of the bias explicit and detectable. This visible weighting provides direct mechanism for bias detection and mitigation, a capability that is not feasible in coarse-grained, sequence-level approaches.
23


Reviews:
[{"review_id": "htny39kMC0", "reviewer": "Reviewer_Y6ap", "summary": "The authors propose the Token-Importance Guided Direct Preference Optimization (TI-DPO) framework, a novel method to align large language models with human preferences. TI-DPO employs a hybrid weighting mechanism that assigns importance scores to tokens and a triplet loss that provides a more structured optimization signal. Theoretically, the authors show that TI-DPO achieves a tighter loss bound than DPO. Extensive experiment results indicate that TI-DPO surpasses existing RLHF methods across benchmarks, such as HumanEval, TruthfulQA, and IFEval.", "questions": "- Could the authors clarify the procedure for generating the anchor response $y$ used in the triplet loss?\n- What is the theoretical or empirical rationale for selecting a Gaussian prior for token-importance weighting over alternative distributions?\n- How sensitive is the model performance to the weight-mixing hyperparameter $\\lambda$?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper introduces a novel Token-Importance Guided Direct Preference Optimization (TI-DPO) framework that integrates gradient-based token attribution with a triplet loss objective, aiming to achieve finer-grained preference alignment.\n- Theoretical analysis provides a formal derivation suggesting that TI-DPO attains a tighter loss bound than standard DPO, offering a potentially more stable optimization objective.", "weaknesses": "- The motivation for employing a Gaussian prior in the hybrid weighting mechanism is insufficiently justified. The assumption that salient tokens cluster near the center of a sequence lacks both empirical evidence and theoretical grounding. Alternative priors or adaptive distributions are not discussed.\n- Although the authors claim that TI-DPO achieves a tighter loss bound than DPO, the paper does not provide quantitative evidence to demonstrate the practical significance of this theoretical improvement. Moreover, the potential implications for additional benefits, such as convergence speed or generalization performance, remain unclear.\n- The experimental evaluation omits comparisons with recent token-level DPO variants, such as TIS-DPO [1].\n\n[1] Liu, Aiwei, et al. \"Tis-dpo: Token-level importance sampling for direct preference optimization with estimated weights.\" arXiv preprint arXiv:2410.04350 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wYFZ8K4Dbr", "reviewer": "Reviewer_GJd9", "summary": "This paper proposes an enhancement to Direct Preference Optimization (DPO) through token-level importance alignment. The authors incorporate token-level weighting based on gradient attribution, coupled with a Gaussian prior to promote robustness. The method extends DPO with a modified token-level objective and an additional triplet loss for fine-grained alignment. Theoretically, the paper shows that the standard DPO objective upper bounds the proposed per-token loss. Empirical results demonstrate that this approach strongly outperforms vanilla DPO and achieves better performance compared to other RLHF-based methods across multiple benchmarks.", "questions": "While I really enjoyed this paper, some remaining questions and suggestions that would benefit the paper:\n1. Looking at Table 2, Tl-DPO shows a substantial improvement compared to vanilla DPO. Can this method be extended to other alignment frameworks such as GRPO or DRPO? Including such extensions and experiments would greatly strengthen the paper.  \n2. Have you conducted ablations on the impact of the Gaussian prior compared to using no prior or alternative priors (e.g., uniform, softmax probabilities)?  \n3. Regarding the relationship with vanilla DPO, is there any equivalence between the vanilla and token-level DPO formulations? Specifically, for Equation 8, are there particular values of \\( w \\) that would make it equivalent to vanilla DPO?  \n4. Line 266 mentions that for the triplet loss, each response is first mapped to an embedding. How is this mapping performed?  \n5. Line 298 states that “Tl-DPO will be **significantly** lower than the original DPO loss.” However, Theorem 2 only proves that \\( L_{\\text{DPO}} \\) upper bounds \\( L_{\\text{Tl-DPO}} \\). Without additional arguments, this claim appears overstated.  \n6. Line 470 claims that Tl-DPO “effectively bridges the alignment gap between LLMs and human value systems.” This statement is too strong and should be moderated.  \n7. While Theorem 2 provides a useful upper bound for Tl-DPO, its practical significance without further discussion is unclear, since a lower loss does not necessarily imply better downstream performance. Are the Tl-DPO and DPO losses comparable under certain conditions or specific values of \\( w \\) (see Question 3)?  \n\n**Minor Comments That Do Not Affect Rating**\n1. For the triplet loss, \\( y \\) is being optimized. What about the embeddings for \\( y_w \\) and \\( y_l \\)? Is the network that produces these embeddings trained as well?  \n2. Have you tested alternative contrastive losses beyond the triplet loss? For example, using a SimCLR or SogCLR-style approach, one could form positive pairs between \\( y \\) and its corresponding \\( y_w \\), and negative pairs between \\( y \\) and all other \\( y_l \\). This might provide a stronger signal from negative responses through hard-negative weighting.  \n3. Equation 3 is described as showing token-level DPO, but the equation currently corresponds to vanilla DPO. Additionally, \\( y^{<} \\) is defined but never used.  \n4. Line 190 mentions that “\\( m \\) is the count of tokens,” but this variable is not used in the formulation.  \n5. Line 9 in Algorithm 1 references \\( \\Delta r_{\\text{triplet}} \\), which has not been defined earlier. Equation 10 defines only \\( \\Delta r_{\\text{token}} \\). The definition of \\( \\Delta r_{\\text{triplet}} \\) should likely appear near Equation 12.  \n6. Section 5.4 should explicitly refer to Figure 2 for clarity and consistency.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. **Well written and clearly presented.** The paper is well structured and makes effective use of figures and visualizations to support understanding.  \n2. **Novel contribution.** Introduces a new approach that weights token importance and incorporates a triplet loss for more fine-grained alignment.  \n3. **Sound theoretical analysis.** Provides theoretical guarantees showing how the proposed formulation relates to and improves upon vanilla DPO.  \n4. **Strong empirical results.** Experimental results demonstrate notable performance improvements over standard DPO and related baselines.", "weaknesses": "1. **Limited generalizability.** The method is presented as DPO specific, which limits its applicability to newer and potentially more effective alignment approaches such as GRPO and DRPO.  \n2. **Incomplete methodological details.** Some aspects of the triplet loss implementation are insufficiently described, making it difficult to fully understand how this component contributes to the overall improvement of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WxH0CFp0HR", "reviewer": "Reviewer_3UVx", "summary": "This paper proposes TI-DPO, a new framework for aligning large language models (LLMs) with human preferences. This framework performs human preference alignment for LLM at a token-level granularity rather than the conventional sequence-level used in DPO and RLHF. The authors observe that existing sequence-level preference optimization methods ignore the varying importance of individual tokens, resulting in unstable training and coarse-grained supervision. To tackle this problem, they further identify two key challenges. First, the model needs to accurately identify the key tokens that influence human preference. Second, the optimization objective should capture continuous and fine-grained preference differences. To address the first issue, they propose the Hybrid Weighting Mechanism to combine gradient attribution with a Gaussian prior to compute token-level importance weights. To tackle the second issue, the authors utilize the Triplet Loss to guide the model’s outputs toward preferred responses and away from non-preferred ones, thereby enabling continuous preference learning in semantic space.", "questions": "1.\tQuantitative evidence for claimed advantages: Could you provide experiments or analysis to substantiate the claims of robustness, generative diversity, and computational efficiency? \n\n2.\tComparison with other token-level baselines: The paper mentions that TI-DPO provides fine-grained alignment compared with several token-level alignment methods (e.g., TIS-DPO [1], cDPO [2], and Logic-RL [3]) but does not include them in the experimental comparison. Could the authors provide experiments or analysis to discuss their methods with other token-level methods?\n\n3.\tDesign of the Gaussian prior: Could you clarify the exact settings of the Gaussian-shaped prior used in the Hybrid Weighting Mechanism (lines 228–234)? How sensitive is TI-DPO’s performance to the design settings of this prior?\n\n4.\tAnchor generation in Triplet Loss: The generation process of the anchor $y$ is not specified. Could you explain how the anchor response is generated and how to ensure an intermediate position between preferred and non-preferred responses?\n\n5.\tAnalysis of performance gaps: TI-DPO underperforms GRPO or TPO on MMLU, GSM8K, and GPQA (Tables 1 and 7–9). Could you discuss why this occurs? \n\n6.\tAblation studies and reproducibility: Could you provide further ablations to discuss (a) the individual effects of gradient attribution and Gaussian prior, and (b) the sensitivity to key hyperparameters such as λ (Eq. 7) and α (Eq. 12)? Additionally, is it possible to clarify the values of predefined hyperparameters (e.g., α, β, γ, λ) and how these choices are selected?\n\n7.\tScalability and potential bias amplification: Does the token importance computation in this work introduce computational overhead? When applying TI-DPO to tasks with long responses (e.g., multi-turn dialogue or chain-of-thought reasoning), how does the computational cost scale with model size and sequence length? Furthermore, has the model been evaluated for potential bias amplification, such as reinforcing biased correlations like “she-nurse” or “he-surgeon” in preference data?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tOriginality: The paper introduces a novel perspective by extending preference optimization from the traditional sequence-level to a token-level framework, allowing fine-grained semantic control. The proposed Hybrid Weighting Mechanism utilizes gradient attribution to determine the contribution of each token to the model’s output, with a Gaussian prior incorporated as a regularization mechanism to enhance training stability. The use of the Triplet Loss for fine-grained preference alignment and continuous gradient of preference learning further distinguishes this work from prior DPO-based approaches.\n\n2.\tQuality: The theoretical analysis demonstrates that TI-DPO achieves a tighter loss bound than DPO. The experimental evaluation is comprehensive. The results demonstrate superior accuracy compared to baselines on the HumanEval, TruthfulQA, and IFEval benchmarks.\n\n3.\tClarity: The paper is generally well written and the motivation is clearly explained.\n\n4.\tSignificance: TI-DPO provides a token-level human preference alignment framework that enhances the performance of LLMs and offers more stable training.", "weaknesses": "1.\tLimited empirical validation of claimed advantages: Although the paper claims improvements in robustness (e.g., line 20, line 58, line 125), generative diversity (e.g., line 24), and computational efficiency (e.g., lines 24-25), these aspects are not discussed or quantitatively evaluated. There is a lack of experiments that measure robustness under noisy or perturbed preference data. The paper also does not discuss or evaluate the claimed diversity gains. For example, evaluate generative diversity using metrics such as Self-BLEU, Distinct-n or Diversity Entropy. Reporting quantitative comparisons of GPU hours or providing computational complexity analysis would also strengthen the claim of efficiency.\n\n2.\tLack of details in the Gaussian prior design: The paper describes the use of a Gaussian-shaped prior in the Hybrid Weighting Mechanism to assign higher baseline importance to middle tokens. The paper only mentioned “we define a Gaussian-shaped prior distribution $\\mathcal{P}_{\\text{prior}}$ centered on the sequence” (lines 228-234), but it does not specify its exact settings. This lack of detail hinders not only the reproducibility but also the evaluation of how much the Gaussian prior actually contributes to the method’s effectiveness.\n\n3.\tAmbiguity in anchor generation: The paper states that the anchor response is “dynamically created by the model using the preferred response as a contextual starting point” (lines 262-264), without further details of how this response is generated. The lack of the anchor generation procedure makes it difficult to evaluate the triplet loss setup and determine whether the anchor truly represents an intermediate state between positive and negative samples.\n\n4.\tIncomplete comparison and missing analysis of performance gaps: While the paper claims to address two major challenges in token-level preference optimization (lines 52-55), it does not include sufficient comparisons with other token-level alignment methods, such as TIS-DPO [1], cDPO [2] or Logic-RL [3], which are mentioned in both the Introduction and Related Works sections. Moreover, TI-DPO underperforms baselines, such as GRPO or TPO, on key benchmarks, including MMLU, GSM8K, and GPQA (as shown in Tables 1 and 7–9). Yet, the paper provides no analysis or discussion to explain these results.\n\n5.\tThe ablation analysis is insufficient: The current study only examines the removal of the triplet loss and the use of uniform or random weights, which is not enough to fully demonstrate the contributions of the proposed components. Specifically, the paper lacks ablations on (a) the relative effects of gradient attribution versus the Gaussian prior in the Hybrid Weighting Mechanism, and (b) the sensitivity of model performance to the weighting coefficient λ (eq. 7) and the triplet margin α (eq. 12). In addition, there are many predefined hyperparameters (e.g. α, β, γ, λ) utilized in this paper. The paper does not provide detailed hyperparameter values, which may hinder reproducibility. Moreover, in Algorithm 1 (line 346), the hyperparameter γ is not listed among the inputs, making it unclear whether it is user-defined or internally fixed.\n\n6.\tWeak connection between theory and experiments: While Lemma 1 and Theorem 2 are mathematically sound, the paper does not clearly explain how the “tighter loss bound” are related to improvements in accuracy. A short discussion linking the theoretical results to empirical results would enhance the reading flow. Moreover, Sections 4.2 and 4.4 are overly dense, with long mathematical derivations but limited commentary. Adding clearer links between theory and experiment results, together with concise textual explanations, would significantly improve both clarity and readability.\n\n7.\tLack of broader discussion and limitations: The paper does not sufficiently discuss token-level preference optimization in different scenarios, such as scalability to very long sequences or potential bias amplification when importance weights overemphasize frequent tokens. For example, when applied to tasks with very long responses (e.g., multi-turn dialogue or chain-of-thought reasoning), computing gradient-based token importance may become computationally expensive. In addition, if the preference data contains biased correlations such as “she” frequently appearing with “nurse” and “he” with “surgeon”, the weighting mechanism might assign higher importance to these patterns, thus reinforcing gender stereotypes rather than mitigating them. It would be valuable if the authors further analyzed whether their framework can address such issues or provided a brief discussion of potential solutions.\n\n[1] Liu, Aiwei, et al. \"TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weights.\" The Thirteenth International Conference on Learning Representations.\n\n[2] Lin, Zicheng, et al. \"Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM’s Reasoning Capability.\" Forty-second International Conference on Machine Learning.\n\n[3] Xie, Tian, et al. \"Logic-rl: Unleashing llm reasoning with rule-based reinforcement learning.\" arXiv preprint arXiv:2502.14768 (2025).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2SJxoaCQy6", "reviewer": "Reviewer_sxsD", "summary": "The paper addresses a technique for token-level alignment in preference signal processing within Direct Preference Optimization (DPO). Unlike typical DPO methods that focus on sequence-level alignment, this research explores token-level alignment. Existing approaches, such as TIS-DPO, rely on probability estimation from contrastive language models, while other methods use repeated sampling.\n\n\nIn this study, the authors introduce gradient-based signals to assess token importance in relation to positive and negative examples. They also incorporate a triplet loss term to enhance training stability, providing detailed mathematical derivations for the loss function, gradients, and a tighter loss bound. Additionally, they discuss experimental results across a comprehensive evaluation suite.\n\n\nTo determine token importance weights, the authors calculate the maximum logit for the final token in both positive and negative sequences and then compute gradients of the preceding tokens' embeddings that maximize this final token max logit. This process helps identify the importance of tokens towards maximizing confidence in the final token. The authors develop a weighting scheme for previous tokens, integrating it with a Gaussian prior that softly biases towards middle tokens. This refined weighted objective is used for training, with the expectation of learning token-level importance.\n\n\n\nFurthermore, the authors incorporate triplet loss by generating an anchor response and adding a triplet loss term. This term guides the anchor closer to positive examples and away from negative ones, contributing to the combined training objective.\n\n\nThe paper presents a series of experiments and ablation studies, demonstrating progressive improvements with each method. The results indicate strong performance on various evaluation tasks.", "questions": "- The Gaussian prior seems somewhat counterintuitive, particularly for question-answer prompts where key tokens may not be centrally located. How do you interpret the effectiveness of this prior?\n- The approach to determining token weights relies primarily on the final token logits, which may not always provide the most significant signal. For instance, the final token could be akin to a closing word, potentially lacking influence on the overall quality of the response. How do you address this observation, and do you have any studies that explored this phenomenon?\n- Is there potential for information loss when calculating weights by taking the maximum of the final logits?\n- What is the additional computational cost associated with this method, considering the extra gradients and anchor responses involved?\n- Could you provide more illustrative examples to clarify the nature of token weights and their impact?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- The paper effectively introduces the concept of using gradient-based signals to determine token importance, enabling a more detailed understanding of token-level significance.\n- The authors conduct thorough ablation studies, demonstrating that each component of the proposed method enhances the final evaluation metrics.\n- The presentation of choices and the algorithm is exceptionally clear.\n- There is a robust mathematical analysis and derivation concerning the mechanics of the new objective, including stronger loss bounds.\n- The associated code repository clearly outlines the training data and the methodology mechanics.\n- The paper includes additional information in the Appendix, such as token weight distribution across various evaluation datasets and mathematical derivations.", "weaknesses": "- Providing additional unbiased examples and analyses could illuminate the nature of token weights identified across various datasets. The paper does not clearly convey a qualitative understanding of the token weights the model is learning, except for one example.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)