TGDPO: Harnessing Token-Level Reward Guidance for Enhancing Direct Preference Optimization
Authors: Mingkang Zhu
Xi Chen
Zhongdao Wang
Bei Yu
Hengshuang Zhao
Jiaya Jia
Venue: ICML 2025 poster (2025)
Keywords: Large Language Models
Preference Optimization
Primary area: deep_learning->large_language_models

Abstract: Recent advancements in reinforcement learning from human feedback have shown that utilizing fine-grained token-level reward models can substantially enhance the performance of Proximal Policy Optimization (PPO) in aligning large language models. However, it is challenging to leverage such token-level reward as guidance for Direct Preference Optimization (DPO), since DPO is formulated as a sequence-level bandit problem. To address this challenge, this work decomposes the sequence-level PPO into a sequence of token-level proximal policy optimization problems and then frames the problem of token-level PPO with token-level reward guidance, from which closed-form optimal token-level policy and the corresponding token-level reward can be derived. Using the obtained reward and Bradley-Terry model, this work  establishes a framework of computable loss functions with token-level reward guidance for DPO, and proposes a practical reward guidance based on the induced DPO reward. This formulation enables different tokens to exhibit varying degrees of deviation from reference policy based on their respective rewards. Experiment results demonstrate that our method achieves substantial performance improvements over DPO, with win rate gains of up to 7.5 points on MT-Bench, 6.2 points on AlpacaEval 2, and 4.3 points on Arena-Hard. Code is available at https://github.com/dvlab-research/TGDPO.

TGDPO: Harnessing Token-Level Reward Guidance for Enhancing Direct Preference Optimization
Mingkang Zhu' Xi Chen?
Zhongdao Wang?
Bei Yu'
Hengshuang Zhao? 2 Jiaya Jia
Abstract
1. Introduction
Reinforcement Learning from Human Feedback (RLHF) has become crucial technique for aligning Large Lan- guage models (LLMs) with human preferences and inten- tions (Ouyang et al., 2022; Ziegler et al., 2020). This ap- proach has demonstrated significant success in recent LLMs advancements (OpenAI et al., 2024; Team et al., 2024a; Grattafiori et al., 2024; Team et al., 2024b). In typical RLHF workflows, a reward model is first trained using human feed- back, and then the Proximal Policy Optimization (PPO) algorithm (Schulman et al., 2017) is employed to fine-tune the policy model. Typically, in these methods, sequence- level reward is assigned to the last token of a sequence. However, this approach faces challenges, such as the sparse reward problem (i.e., delayed feedback), which leads to in- stability and sample inefficiency PPO training (Choshen et al., 2020). This issue is particularly pronounced LLM training, where responses are often lengthy and generated at the token level (Yang et al., 2023). Recent research has suggested that leveraging dense token-level reward models (Yang et al., 2023; Yin et al., 2025; Zhong et al., 2024) can help alleviate these issues, improving PPO's performance in aligning LLMs with human preferences.
Recent advancements in reinforcement learning from human feedback have shown that utilizing fine-grained token-level reward models can sub- stantially enhance the performance of Proximal Policy Optimization (PPO) in aligning large lan- guage models. However, it challenging to lever- age such token-level reward as guidance for Di- rect Preference Optimization (DPO), since DPO is formulated sequence-level bandit prob- lem. To address this challenge, this work decom- poses the sequence-leve PPO into a sequence of token-level proximal policy optimization prob- lems and then frames the problem of token-level PPO with token-level reward guidance, from which dlosed-form optimal token-level policy and the corresponding token-level reward can be de- rived. Using the obtained reward and Bradley- Terry model, this work establishes a framework of computable loss functions with token-level re- ward guidance for DPO, and proposes a practical reward guidance based on the induced DPO re- ward. This formulation enables different tokens to exhibit varying degrees of deviation from ref- erence policy based their respective rewards. Experiment results demonstrate that our method achieves substantial performance improvements over DPO, with win rate gains of up to 7.5 points on MT-Bench, 6.2 points on AlpacaEval 2, and 4.3 points on Arena Arrna-Hard. Hard. Code available
Recent developments in RLHF have centered around ating simpler and more efficient algorithms that eliminate the need for a separate reward model. A notable approach in this direction is Direct Preference Optimization (DPO) (Rafailov et al., 2023). DPO reparameterizes the reward function RLHF by directly using preference data to opti- mize the policy model, bypassing the traditionally required step of training a separate reward model. This reparameter- ization streamlines the alignment process, making DPO popular algorithm for LLM alignment. While dense token- level reward guidance has been proved beneficial for PPO (Yang et al., 2023; Yin et al., 2025; Zhong et al., 2024), its extension to DPO is nontrivial, as DPO is formulated as sequence-level bandit problem. reward is expressed through the policy being optimized, and integrating token-level reward guidance into this framework presents significant challenge, especially in eliminating the partition function from the loss function.
'The Chinese University of Hong Kong "The University of Hong Kong 'Huawei 'SmartMore 'The Hong Kong University of Science and Technology. Correspondence to: Mingkang Zhu <mkzhu23@ Srccchu2@@eeecccoom aase.cuuk.edii edu. Jiaya Jia Sja@ccseustt.be
this context, the
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
To fill this gap, we decompose the sequence-level proximal

---

TGDPO: Harnessing Token-Level Reward Guidance for Enhancing Direct Preference Optimization
policy optimization into sequence of token-level proximal policy optimization problems and modify them incorpo- rate token-level reward guidance. We derive a closed-form optimal token-level policy and the corresponding token- level reward for the modified problem. Based on the ob- tained reward and Bradley-Terry model, especially a new theoretical result for eliminating partition function, we pro- pose a preference optimization algorithm framework with token-level reward guidance for DPO, which we refer to as TGDPO. Additionally, we introduce practical token-level reward guidance based on the induced DPO reward.
ment learning. Proximal Policy Optimization (PPO) with on-policy sampling (Schulman et al., 2017) is commonly used for this purpose. However, challenges in effective re- ward modeling and tuning the PPO algorithm to achieve optimal performance have motivated alternative approaches that bypass the reward modeling step and focus on directly optimizing the policy. The direct preference optimization (DPO) algorithm (Rafailov et al., 2023) is a representative one. DPO explicitly represents the reward function with the optimal policy of the proximal policy optimization problem, thereby avoiding the need for separate reward model and fine-tuning LLMs directly with human preference. DPO has proven to be both lightweight and stable, showing strong performance in a range of applications (Ivison et al., 2024; Tian et al., 2024; Miao et al., 2024). Several variants of DPO have since been proposed, improving its performance. For instance, R-DPO (Park et al., 2024) addresses DPO's tendency to exploit token length, while SimPO (Meng et al., 2024) aims to better align the objective with the decoding formula and eliminate the need for a reference model. KTO (Ethayarajh et al., 2024) focuses on optimizing preferences using non-pairwise data. These preference optimization techniques, however, operate at the sequence level and do not shape the reward function of DPO from the token level. In contrast, our approach aims to leverage token-level re- wards to guide preference optimization and better align LLMs. A recent work TDPO (Zeng et al., 2024) tries to provide a token-level understanding of DPO. explains DPO using token-level Markov decision process and pro- poses to incorporate forward KL divergence to the DPO objective. However, like DPO, TDPO still does not consider token-level reward guidance. Our TGDPO, on the other hand, explicitly incorporates token-level reward signals into the preference optimization framework.
Extensive experiments are conducted on three instruction following benchmarks: AlpacaEval 2 (Li et al., 2023), MT- Bench (Zheng et al., 2023), and Arrd (I.i et al., 2024). TGDPO consistently outperforms existing preference opti- mization algorithms, achieving improvements of up to 7.5 points on MT-Bench, 6.2 points on AlpacaEval 2, and 4.3 points on Arena-Hard compared to the best baseline method. We further demonstrate and analyze the unique advantages of TGDPO. We empirically show that TGDPO achieves sat- isfactory policies upon loss convergence, which is not com- monly observed in conventional preference optimization methods. TGDPO also enables control over convergence speed and is robust to variations in token-level rewards. These properties significantly enhance the efficiency and practicality of the algorithm. Our key contributions are outlined below:
We decompose the sequence-level PPO into a sequence of token-level proximal policy optimization problems via the upper-bounding approach and derive a closed- form optimal loken-level policy for the modified prob- lem, with which the corresponding reward can be rep- resented along with the token-level reward guidance.
RLHF with Dense Token-Level Reward. Text generation of LLMs can be modeled as a Markov decision process. Sequence-level PPO treats the entire sequence as an ac- tion and assigns a reward at the sequence's 's end (Schulman et al., 2017), which results in sparse feedback at the token level. This sparsity hinders the model's 's ability to differ- entiate between preferred and dispreferred tokens within a sequence, leading to training instability (Snell et al., 2023; Xia al., 2024). To mitigate this issue, several techniques have been developed to generate dense token-level rewards, including learning from fine-graines human feedback (Wu et al., 2023), fine-grained AI feedback (Ouyang et al., 2024), and grounding preferences at the token or segment level (Yang et al., 2023; Yin et al., 2025; Zhong et al., 2024). PPO leveraging such fine- iine-grained reward models has shown significant performance improvements. However, extend- ing token-level guidance to DPO is challenge, as DPO's reward function is explicitly expressed through the policy being optimized. Incorporating levelllevel reward guidance
With the obtained reward, the Bradley-Terry model, and a new result for eliminating the partition funetion, we propose TGDPO, a preference optimization algo- rithm framework with token-level reward guidance for DPO. We further introduce a practical token-level re- ward guidance based on the induced DPO reward.
Extensive experiments demonstrate that our TGDPO improves win rates by up to 7.5 points on MT-Bench, 6.2 points on AlpacaEval 2, and 4.3 points on Arena- Hard compared to the best baseline.
2. Related Work
Reinforcement Learning from Human Feedback. Rein- forcement learning from human feedback (RLHF) has been extensively applied for aligning LLMs with human prefer- ences and values (Ouyang et al., 2022; Ziegler et al., 2020). The standard RLHF pipeline typically consists of two stages: reward modeling and policy optimization through reinforce-
2

---

TGDPO: Harnessing Token-Level Reward Guidance for Enhancing Direct Preference Optimization
into the DPO framework requires overcoming substantial diffieulties, especially in eliminating the partition function from the loss function, which remains open problem. More discussions on closely related work are presented in Appendix C.
where i is a prompt, 8, and d2 are the state and action defined previously, y [ao. [00:-..,0TT1] is the response generated by та from the given prompt 2. Classically, the sequence-level reward function r0 y) can be set as (2( v) Ei ro(st, ar) (Yang et al., 2023).
Direct Preference Optimization. Direct preference opti- mization (Rafailov et al., 2023) bypasses learning a reward model and aligns directly an LLM to human preference. DPO (Rafailov et al., 2023) expresses the sequence-level reward function explicitly with the optimal policy of Equa- tion (1) as:
3. Preliminary
Given human preference dataset D {2z, Mer vt)], where is prompt, Yw and yt are preferred and dispreferred responses respectively, in RLHF a sequence-level reward model (z, y) is first trained with the preference dataset for assigning higher reward to preferred response and lower reward to dispreferred one. With the trained reward model, sequence-level Proximal Policy Optimization (PPO) solves the following problem to fine-tune LLMs:
To(y|æ) (v)())
ro( y) = Blog
+ 3 log ((x),
(3)
where (xx) is the partition funetion and ,3 is a positive constant. By adopting the Bradley- Terry preference model
max E, (.yao (z, v) - 5IDx TO (-
(Bradley & &eery, 1952)
To (u]l)) max Bew ro(2, Blogg Te (y/x))
exp (ro(z, Mww)
Pr(yw > yn [x)
(4)
exp (ro(z, Mw)) + exp (ra(1r, v1))
no
for specifying human preference distribution, DPO obtains the following loss function:
where [ is the KL-divergence of two probability distri- butions, Ty is the language model policy, Tee is the reference policy, and the positive parameter /3 controls the deviation of no from Tee. Equation (1) can be considered as assigning the reward to a sequence and is referred to as the sequence- level PPO problem in this work. It has the issue of sparse reward (delayed feedback) that challenges traditional deep reinforcement learning (Andrychowicz et al., 2017). To alle- viate the issue, sequence-level PPO with token-level reward guidance is developed to fine-tune LLMs in a fine-grained fashion with dense token-wise rewards (Yang et al., 2023; Yin et al., 2025; Zhong et al., 2024).
no ( (x) (DPO(TT) = -E (a log [3 log
Tree TTee (x:)
To (31 (z) -B Tieel Teee(yn (a:
(5)
which is obtained by substituting Equation (3) into Equa- tion (4), where a is the sigmoid function. DPO minimizes Equation (5) with respect to the policy Te to directly fine- tune the LLM with the preference dataset at the sequence level.
Sequence-Level PPO with Token-Level Reward Guid- ance. Text generation of LLM can be modeled as Markov Decision Process (MDP). Let 8 be the context for generating the token at time step / 2 0, the generated token is denoted as a₂ ~ Ta (-(ss) For a prompt x of the LLM, 80 = x and 8; = [z. <t) where a² = [ag...., 4;-1] are the previously generated tokens. The generated full text- sequence with 7 tokens denoted as a [ao. ..... -.-..,ar-]) -r-1] 1). A token-level reward, for convenience it is also denoted by (alss, a,), is learned so that the reward sequence is dense and can guide the selection of token at any time step, which is called token-level reward guidance (Yang et al., 2023; Yin et al., 2025). Typically, the problem of sequence-level prox- imal policy optimization with token-level reward guidance (Yin et al., 2025):
4. Methodology
Direct preference optimization expresses the reward func- tion explicitly with the optimal policy of the sequence-level proximal policy optimization problem. However, incorpo- rating existing token-level rewards explicitly into DPO to guide fine-tuning is an unresolved problem. To derive a form of DPO with token-level reward guidance, this section first gives the problem of token-level PPO in Section 4.1 from the sequence-level PPO in Equation (2). The token-level PPO problem is further modified to incorporate token-level reward guidance in Section 4.2, the closed-form optimal policy is derived, and the corresponding token-level reward with guidance is obtained. Then with the Bradley-Terry model, we propose the direct preference optimization with token-level reward guidance in Section 4.3.
max E
ay TT- To(a)
Σ
ro(sse, (())),)))
4.1. Token-Level PPO
t=
Te(()x)) Teer(yla))
Note that y = [ao...., aT-1] is the response generated by my from the given prompt x. Using the notations of state
@ log
3

---

TGDPO: Harnessing Token-Level Reward Guidance for Enhancing Direct Preference Optimization
and action in Section can get
is not easy solve the problem since s ~ D, is dependent the policy zo to be optimized (see Equation (1) for comparison, where the distribution D is independent of the policy Ty to be optimized).
T-1 II
To(()z)) The ne((ao. ................................ 0T a1-1112)=
(a(e)))))
t=
T-1
4.2. Modified Token- PPO with Reward Guidance and Optimal Policy
Ter(yll=)= = T Tre((ao...... ar-11/z) (a)()))
Given win and lose responses V (ah...... ..... oT and S (abs.... ... a 1). Rafailov et al. (2024) expressed the per-instance loss of DPO (Rafailov et al., 2023) in the token- level as:
Thus, the objective function in Equation (2) can be decom- posed into the token level as:
(er ax) - -/1log To(u(x) Tre x:)
Pr(yer > v1)
log (a (sr)) Teet (0z (s
T Σ
To ssr) Tea (a.) (s))
3 log
ra(s); 0z) - Blog To (a(())))) Tre(ay
t=
Assuming access to a token-level reward f(t,, az), since the token-level reward f(st, may imply whether the action or is preferred or dispreferred in the state 8t, this work aims to replace 3 in the above equation with (f ot)), a function of the token-level reward f(s,, at), to guide the DPO.
Moreover, according the MDP for language model (Sec- tion 3), y II= To (a)((())) in Equation (2) is equivalent to (-()2)), which is further equivalent to 8o = x ~ D az ~ no (-(ss,), 6 1, T - 1. Then by Equation (6), the problem of sequence-level PPO with token-level reward guidance in Equation (2) becomes
Following DPO (Rafailov et al., 2023), derive this form of Joss function from the token-level proximal policy opti- mization in Equation (8) by incorporating the token-level reward guidance [f((st, at)). First, similar to (Zeng et al., 2024; Yang et al., 2024), we relax 8 ~ Dr to st ~ D and make Equation (8) solvable as
[a(ae(ss)
Σ (ro(st, at) log
max D. Te
Trer (ae((s))))
= max
Σ
(na at)
(t)
max E, - (. [ro(se -Blog
To (e(((())) Tee (a st
To Ta(a st)) n 0r (8))
166
-Blog
(7)
Next, we manage to incorporate token-level reward guidance f(((s; az)) into this formulation, and represent the ground- truth unknown reward function ra (:) with the optimal policy of this equation. The obtained ground-true truth reward r' (st; 0e) is subsequently leveraged construct our DPO's loss function under the Bradley-Terry preference model.
Based on Equation (7), we can show that: Theorem 4.1. The maximum value of the sequence-level proximal policy optimization in Equation (2) is upper bounded by the summation from t = 0. of the maximum value of the problem:
Directly replacing 3 in Equation (9) with (f(((s a:)) might not make the problem easy to solve. To address this issue, by noting that / is a positive constant, Equation (9) is equivalent to
max ED, ro((),a)) az) Blog To (a)(()))) X9
Trel(a (8t)
where 31 De denotes that so = I ~ and ap ~ nT((1sp).
Tr((t, at) - llog
To(as (t)) (er (10)
p 0, 1,...,t.. 1.
max E,, B, B..DDaarnoo(
The proof of Theorem 4.1 is given in Appendix A.I.
Then, we make the following Assumption 4.2 for incorpo- rating token-level reward guidance f(f(st, az)) explicitly into Equation (10).
Equation (8) is the problem of token-level PPO at time step t, which optimizes the policy for action a, given the state st- Theorem 4.1 suggests that, the sequence-level proximal pol- icy optimization in Equation (2) can be upper-bounded with a sequence of token-level PPOs in Equation (8). However, it
Assumption 4.2. Suppose we have an existing reward model f(-), which can generate a dense token-level reward

---

TGDPO: Harnessing Token-Level Reward Guidance for Enhancing Direct Preference Optimization
sequence f(t;, ay), L 0.1,...,,7- T- 1. Moreover, suppose [u) is positive univariate function of u.
Without loss of generality, suppose that trajectories gen- erated by LLMs are bounded by finite number of time steps, or tokens. Then, since LLMs are over-parameterizeed. we may assume without loss of generality that, there exists 0 such that Tp(ae (s)) = To, (0r(8)). t = 0, 1, 1,....77 T 1. Thus, with the notations of the prompt æ and the generated sequence y, Equation (13) can be rewritten in the form
was shown in Rafailov et al. (2024) under the defini- tion of equivalent state-action reward class and invariant re- parameterization that, DPO implicitly learns a token-level reward fste dr) of the form 33 log (ed((e))e)) and the total reward F(z, y) = ET- f(s;,a)) Hence Assumption 4.2 is feasible.
Tu(oelse)
Toly' [[,, y v-1) TTer [z, yrt))
ro( y "
"1,,'")) "),"")))((((() -3[(f((2, log
Modified Token-Level PPO. With Assumption 4.2, we propose to adopt the token-level reward f(st. qz) to guide token-level PPO. First, the parameter 3 in Equation (10) is replaced with 3 (f (4)) and we obtain the modified problem of teven-level PPO with token-level reward guid- ance as follows:
+ BBf(F( " "I ,')) log Z ((1 y ((
(14)
for all sime-step t, where the last term with the partition function does not depend on To according to Theorem 4.3.
4.3.
Direct Preference Optimization with Token-Level Reward Guidance
To Ta( U+)
log To (a(ar) st) Teeel ar (st)
max (aa
Bf(f(st, a:)
(11)
For the proximal policy optimization with token-level re- ward guidance in Equation (11), Section 4.2 has represented the ground-truth unknown token-level reward r5 at) ex- plicitly in Equation (14). Subsequently, the total reward ro (z. y) for the prompt x and its response y can be ex- pressed as:
where (f((st, a.)) with the token-level reward f(s,, a.) is adopted to modify the ground-truth unknown reward func- tion T (st, a.).
Thus similar to (Rafailov et al., 2023). the optimal policy for the action ag time step of the modified token-level prox- imal policy optimization in Equation (I1) can be derived as the following Theorem 4.3.
ro v) = Σ " log To x M |z, y<tl)
T
Theorem 4.3. The optimal policy ne, (aclss) for the action a, at time step of the modified token-level proximal policy optimization in Equation (11) is
T
+
3[(F((z,
- 1,,') log log 2(x "T))
(15)
TTe((e((s)) exp To, (aelss)=
rg(s..a)) (F(((())a))))
where the last term with the partition function does not depend on ne.
((a)
where (ss) BarwTast exp volenar)) ) is the partition function, and St D does not depend on Re.- Moreover, the ground-truth unknown token-level reward can be represented with the optimal policy na, (ae (8)) as:
Next, we derive the loss function with token-level reward guidance for direct preference optimization, as we set the target at the beginning of Section 4.2. Given human prefer- ence dataset D = {ar, yws Dw, v)),, where x: is prompt, yoe and Y1 are preferred and dispreferred responses respectively, we adopt the reward function in Equation (15) and the Bradley- Terry preference model in Equation (4) for specifying hu- man preference. To this aim, we choose different shaping functions fmr() and fi(-) for win and lose responses respec- tively, both of them satisfy the condition in Assumption 4.2. Then by substituting Equation (15) into Equation (4), we can get the per-instance loss detailed as follows.
(f(F())))))
ro(Se: f(F(st,a.) = :Blog (ee((r)s1)
To,
Ta, (ay(s))) + log 7(s)) (12)
The proof of Theorem 4.3 is provided in Appendix A.2.
Modified Token-Level Reward. By Equation (12), we have the token-level reward function
Bradley-Terry Model with Token-Level Reward Guid- ance. From Equation (15), for convenience we let
Te. (ar(st) +
ra(ss, az) =Bf(F(st, log Teet (ae $t)
Bf (s,, at)) log Z (st).
(13)
(T f, x 22 Bw. v1)
where f(((st, at)) satisfies Assumption 4.2, 3 is a constant, 8t D does not depend on no,, 1 = 0, 1, 1,..., ...,.
(f w.w))) log Toer (,z, y T
Te(uiclla I1x, yoe
5

---

TGDPO: Harnessing Token-Level Reward Guidance for Enhancing Direct Preference Optimization
(e [z Vi
preference function as a (yp(To, J, Fr Nw W1), i.e.
2f (,x)) log
(16)
Tee (yr [r, 3/ (])
Pr(y > m/x) - a (Lp(TT, f, 2, Yer 3r)
3l (f Uu ww ),w")w log no (v [xx ww51) Te win [ar, ye "
f P: x. Mro 31)
T
(fo(,( ww'l, vlo)) log ((1x, ()
f.f(f((2, of v("). log To [|
11 31 2) Tee [z; 3/ (1
=
T
Σ Bfi(f((2, vf)) log 2(xx, vi I),,
which specifies the per-instance human preference and is computable. Furthermore, analogous to Equation (5), we formulate the loss function for enhancing DPO by harness- ing token-level reward guidance as follows:
where Tu and Tj are the lengths of the responses yo and y respectively. Then, the Bradley-Terry preference model with token-level reward guidance is
LTGDPO(TO) -B( log
Pr(y > (1(z)) a (vp(TO, J f3,,, yu. m1) + 8(S, (2 you v))
(17)
3. fuc(F((a vat (o""),"()))
log Teet ( [2x, 31511)
T-
To I y1 () Ther [s., Mi (( )]
The proof of Equation (17) is given in Appendix A.3.
(fi(F((r, vi "/ (l)))) log t=
The above function is not computable since it contains parti- tion functions in {ff, f; xr Mwr W). Notably, preference opti- mization aims to maximize the preference function in Equa- tion (17) with respect to Tg, and {(f, f; x. Ywo, Ut) does not depend on the policy Te, we can eliminate 8(f, f; x, yos 31 from Equation (17) based on the following Theorem 4.4.
(20)
The loss function (TaDPO(TT) in Equation (20) provides a framework of direct preference optimization, by leverag- ing f(f(s;, at)) to shape the optimization of the policy on the tokens of win and lose responses. Specifically, with an appropriate choice of f(-). this framework can recover several known direct preference optimization methods. For example, if we take f fi = 1, then Equation (20) is the loss function of DPO (Rafailov et al., 2023) (for others, see Appendix C.2). Nonetheless, the aim of this framework is to use token-level reward f(s,, az) to shape the loss function in Equation (20) directly. In the following, we provide a practical example.
Theorem 4.4. The preference function in Equation (17)
has the same maxima and the same ascent directions the function a (yp(Te, f, fi Vw v.)). Moreover, for two policies Te, and T
a ((p(TT) f, f, ywr v1 + off f; yw: 11) > > (yp(Ter: f. fr yw vi) + 8ff f; Vur v1))
(18)
Practical Method. For convenience, we adopt the induced DPO reward (Rafailov et al., 2023) for the token-level re-
is and only if
ward f(se, dz). Suppose T& is an optimal policy of the loss function of DPO in Equation (5), Rafailov et al. (2024) showed in Theorem 1 that DPO learns implicitly a token- level reward of the form
((p(Te) f; yw: 91) > o (1p(Ter f, Ve Ui))
(19)
The proof of Theorem 4.4 is given in Appendix A.4. Theo- rem 4.4 is due to that, the sigmoid function is strictly increas- ing and it does not change the order of values. Hence Theo- rem 4.4 suggests that, maximizing a (1p(TT, f, f; x. yw y1) with respect to Fp is equivalent to maximizing the preference function in Equation (17) with respect to πy. Furthermore, the equivalence between Equation (18) and Equation (19) demonstrates that, for any two policies To, and maz. cancel- ing the term 8(f, i;; xr yhe, M1 from Equation (18) does not affect the preference order of the responses you and yr.
TalV'l0 ( ( Terrry' (z,, xEE)
A(x y (1,)') = Blogg
Hence for Equation (20), we simply set
fu (f(2,, voll vle) = 1 + a f([z, v vlu) fi(f((z, "1 vi) =1 f([z,
(21)
w1 vf)
where is positive constant. Obviously, this setting meets Assumption 4.2 if a is small enough.
Motivation of the Practical Method. Observing the loss function LTGDEO (Tg)) in Equation (20), below is the motiva- tion for setting fff([z, y ())))) as in Equation (21):
Loss Function. Since we only care about the optimal policy of Equation (17), by Theorem 4.4 we may redefine the
6

---

TGDPO: Harnessing Token-Level Reward Guidance for Enhancing Direct Preference Optimization
For token v in win-response, if f( yo p > O, then it is identified as a preferred token, implying that the state-action should be reinforced, and then it is assigned a larger weight 1 + ar([z, ", Vie) In this way, the gradient of our loss function (TGDPO(TO) at this state-action is
2024), Llama3.2 (annan.2-3B--iie Instruutt and Gemma2-2B-it (Team et al., 2024b). Following (Meng et al., 2024), we use prompts from the UltraFeedback dataset (Cui et al., 2024) and let each model generate 5 responses with a temperature of 0.8. These responses are then ranked using the ArmoRM model (Wang et al., 2024). The highest and lowest-ranked responses are selected as the chosen and rejected samples, respectively. For Llama3-88 ---3-----Instuutt we further utilize the PairRM model (Jiang et al., 2023) annotate response scores, thereby evaluating the robustness of algorithms in handling varying quality of sample annotations. Hyperpa- rameter settings are presented in Appendix D.I.
To(y Jr, yee" Teet (yee [xt, vo()))
8(1 + af[(f, "e'), V xy log
which is scaled up by 1 + af(((a " Vin) optimizing our loss function LTGDPO (T)) encourages the policy to assign a higher probability to this action.
As result,
Evaluation Benchmarks. We primarily evaluate trained models' performance using three widely recognized open-ended instruction-folloving benchmarks: MT-Bench (Zheng et al., 2023), Arena- Hard (Li et al., 2024), and Al- pacaEval 2 (Li et al., 2023), which assess models' response quality across diverse queries. For MT-Bench, we report the MT-Bench score and win rate against GPT-4. For Arena- Hard, we report the win rate against GPT-4-0314. For Al- pacaEval 2, we report the win rate against GPT-4 Turbo. Further details are discussed in Appendix D.2.
Similarly, the token v satisfying f([z, y='l. v(n) < 0 is identified as a dispreferred token, although it is in the preferred response Y/o. Then by assigning weight + (f( " vit)) < L. optimizing our loss func- tion LTGDPO (Te) would progressively assign a lower probability to this action.
For a token of in lose-response, if f( 91 "J," < O, then it is considered as a dispreferred token. Thus since (fx y1 * vf)) > 1, optimizing the loss function CraDro (T)) would assign an even lower probability to this action.
the weight
Baseline Methods. We compare our TGDPO with two state- of-the-art preference optimization methods: DPO (Rafailov et al., 2023) and SimPO (Meng et al., 2024). We also include the pre-trained Instruct model as a baseline.
The token of satisfying f(2z, vi 'I vf) ) > 0 is consid- the dispre- ferred response yı. In this case 1 af((z, vf " v())= < 1, then optimizing the loss function LTGDPO(TT) (T)) would progressively assign a higher probability to this action.
ered as a preferred token, although
5.2. Main Results
The experiment results on AlpacaEval 2 (Li et al., 2023), Arena-Hard (Li et al., 2024), and MT-Bench (Zheng et al., 2023) are summarized in Table 1. Our TGDPO consistently outperforms baseline methods across these benchmarks. No- tably, on AlpacaEvaal 2, it achieves win rate increase of up 6.2 over the best baseline, while on MT-Bench, the win rate improves by up to 7.5. For the challenging Arena- Hard benchmark, our method demonstrates stable superior performance, with win rate enhancement of up to 4.3 com- pared to the best baseline. These consistent performance improvements underscore the effectiveness of our approach. More experiment results and comparisons are presented in Appendix B.
The above analysis indicates that our direct preference op- timization with token-level reward guidance performs in the token-level granularity, and exhibits varying degrees of deviation from the reference policy based on their respective rewards. This property inherently empowers our approach to discover satisfactory policies, leading to better policies than existing approaches. This property should be attributed to the modified token-level PPO with reward guidance in Section 4.2, and the derived loss function LIGDPO(TT) (Te) for direct preference optimization in Equation (20) with the setting of f(f([z, y *),,""))) in Equation (21).
5. Experiments
5.3. Analysis
In this section, we present an empirical analysis of the unique properties of our TGDPO in comparison to con- ventional preference optimization approaches. The analysis is conducted under the Llama3-8B: ---3-88---nsteee PairRM setting.
In this section, we first outline our experiment settings in Section 5.1. Then we show the main experiment results in Section 5.2. Lastly, we provide an empirical analysis of the unique properties of our TGDPO in Section 5.3.
TGDPO Leads to Satisfactory Results upon Loss Conver- gence. A well-known challenge in preference optimization algorithms the misalignment between loss minimization and model performance (Guo al., 2024). Specifically, min-
5.1. Experiment Settings
Models and Training Settings. We conduct experiments on three models: Alamaa38B-IInseee (Grattafiori et al.,
7

---

TGDPO: Harnessing Token-Level Reward Guidance for Enhancing Direct Preference Optimization
Table Experiment results AlpacaEval 2 eLieta, al., 2023), Arena-Hard (I. al., 2024), and MT-Bench (Zheng al., 2023) benchmarks.
-lama3-8B-Instrvee PairRM
Llama3-8B- ArmoRM
Method
AlpacaEval
Arena-Hard
MT-Bench
AlpacaEval 2
Arena-Hard
MT-Bench
Win Rate (%)
Win Rate (%)
Score
Win Rate
Win Rate (%)
Win Rate (%)
Score
Win Rate(%)
SFT
30.6
21.4
7.9
27.5
30.6
21.4
27.5
DPO
41.7
30.4
8.0
37.5
40.8
36.2
8.2
46.3
SimPO
39.8
28.7
7.8
32.5
37.0
28.1
7.8
42.5
TGDPO
43.9
34.3
8.0
41.9
42.5
40.5
7.9
45.0
1 ArmoRM
Gemma2-2B-it ArmoRM
Method
A ApacaEval
Arena-Hard
MTT-Bench
AlpacaEval 2
Arena-Hard
MT-Bench
Win Rate (%)
Win Rate (%)
Score
Win Rate (%)
Win Rate (%)
Win Rute (%)
Score
Win Rate (%)
SFT
23.8
17.1
7.0
16.3
32.8
20.1
37.5
DPO
29.6
23.2
7.9
29.4
40.8
26.4
8.0
4.3.
SimPO
26.2
22.6
7.4
15.7
34.8
21.1
7.8
40.0
TGDPO
35.8
25.4
8.1
36.9
43.0
30.7
8.1
46.9
Analysis of preference optimization methods' perfor- mance upon training loss convergence.
Table 2.
Method
AlpacaEval 2
Arena-Hard
0.5
Win Rate (%)
Win Rate (%)
DPO
Loss
SFT
30.6
21.4
TUSDPOO wi e-1.0
TGDPO a.2
DPO SimPO
41.7
30.4
0.2
39,8
28.7
DPO w/ convergence
30.7
17.9
SimPO w/ convergence
4.6
2.4
TGDPO w/ convergence
43.9
34.3
100
200
300
400
Step
Table 3. Analysis of our TGDPO's performance upon training loss convergence with different convergence speeds.
Figure 1. Training loss curve for DPO and our TGDPO with dif- ferent values of a. Changing the value of a leads to different convergence speeds for our method.
Method
AlpacaEval 2
Arena-Hard
Win Rate (%)
Win Rate (%)
SFT
30.6
21.4
imizing the loss for many preference optimization methods often results in degenerate policies. This issue necessitates extensive hyperparameter tuning identify a sweet spot be- tween the initialization and convergence points, significantly limiting the practicality and efficiency of these algorithms. As shown in Figure 1, the optimal hyperparameters for DPO barely reduce its loss. In contrast, we empirically find that TGDPO enables convergence in much fewer steps than con- ventional preference optimization algorithms. TGDPO demonstrates consistent and stable loss reduction toward convergence. We assume it is because TGDPO's token-level reward inherently distinguishes preferred and dispreferred tokens.
TGDPO w/ 0.5
43.9
34.3
TGDPO w/ a 1.0
42.5
33.9
TGDPO w/ a 2.0
43.3
34.3
results reveal that both DPO and SimPO suffer substantial performance degradation upon convergence, with SimPO's win rates dropping to single digits. Conversely, TGDPO maintains exceptional performance at the convergence point. These findings highlight the necessity of extensive hyper- parameter searches for traditional preference optimization algorithms,, whereas TGDPO simplifies the process, signifi- cantly improving efficiency and usability.
Figure 1,
Furthermore, in Table 2, we compare benchmark perfor- mances by training each method using their default con- figurations and training them until loss convergence. The
TGDPO Enables Control Over Convergence Speed. TGDPO offers the flexibility to control the speed of con- vergence by adjusting the value of Equation (20).
8

---

TGDPO: Harnessing Token-Level Reward Guidance for Enhancing Direct Preference Optimization
admits closed-form optimal token-level policy with which the corresponding token-level reward be represented. Using the obtained token-level reward and Bradley-Terry model, we propose TGDPO, sequence-level DPO algo- rithm framework with token-level reward guidance. More- over, we introduce a practical token-level reward guidance. Extensive experiments on MT--eech, AlpacaEval 2, and Arena-Han demonstrate TGDPO's superiorities.
Table Analysis of TGDPO's rohustness using different leven-level rewards ai).
Method
AlpacaEval 2
Arena-Hard
Win (a)e
Win Rate (%)
SFT DPO w/ 5 = 0.1
30.6
21.4
34.8
26.7
DPO w/ 8 0.01
41.7
30.4
TGDPO w/ 3 = 0.1 for a()).a))
42,8
34.3
TGDPO w/ 3 0.01 for (()
43,9
34.3
Impact Statement
This paper enhances DPO by incorporating token-level re- ward guidance. This integration significantly boosts DPO's performance. Although the current evaluation concentrates on helpfulness, we believe our method would also benefit other important aspects of LLM alignment, such as safety, honesty, and faimesss
larger a provides stronger token-level guidance, resulting in faster convergence, while a smaller aligns the algo- rithm more closely with conventional DPO behavior. As illustrated in Figure 1, increasing a leads to a more rapid loss reduction compared to lower values of a. Addition- ally, in Table 3, we compare benchmark performances at the respective convergence points for different values of a. Specifically, evaluate checkpoints at step 50 for 2.0, step 60 for 1.0, and epoch I for a 0.5. The results demonstrate comparable performance across all configura- tions, especially for the challenging Arena-Hard benchmark. This desirable property of TGDPO allows for early stopping once the loss converges, significantly reducing computa- tional costs without compromising performance.
References
Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Pieter Abbeel, O., and Zaremba, W. Hindsight experience replay. In Advances in Neural Information Processing Systems, vol- ume 30, 2017.
Bradley, R. A. and Terry, M. E. Rank analysis of incom- plete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
TGDPO is Robust to Variations in Token-Level Rewards ((8,, az). To make TGDPO practical, we propose using token-level rewards derived from pre-trained DPO models as a convenient implementation. A key question arises: how sensitive is TGDPO to the quality of the token-level rewards f(t, (4) defined in Equation (20)? To investigate this, we analyze the behavior of TGDPO using token-level rewards obtained from two DPO models trained with different 3 val- ues: B = 0.1 and / 0.01. The benchmark performances of these models, along with TGDPO's performance using their respective rewards, are presented in Table 4. As ex- pected, DPO with 3 = 0.01 significantly outperforms DPO with 8 0.1. However, when the leven-level rewards from these models are used in TGDPO, the resulting performance is nearly identical. This finding highlights TGDPO's bustness to variations in the quality of token-level rewards, making it less dependent on the specific characteristics of the pre-trained DPO model. Such robustness further en- hances TGDPO's practicality and reliability.
Choshen, L., Fox, L., Aizenbud, Z., and Abend, O. On the weaknesses of reinforcement learning for neural machine translation. In International Conference on Learning Representations, 2020.
Cui, G., Yuan, L., Ding, N., Yao, G., He, B., Zhu, W., Ni, Y., Xie, G., Xie, R., Lin, Y., Liu, Z.,, and Sun, M. Ultrafeedback: Boosting language models with scaled AI feedback. In Proceedings of the 41st International Conference on Machine Learning, pp. 9722 9744, 2024.
Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kiela, D. Model alignment as prospect theoretic optimiza- tion. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp. F. (eds.), Proceedings of the 41st Internationaal Conference on Ma- chine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 12634-1266 PMLR, 21-27 Jul 2024.
6. Conclusion
This paper enhances DPO by incorporating token-level reward guidance, which is achieved by decomposing sequence-level proximal policy optimization into a series of teven-level proximal policy optimization problems. We formulate the problem of token-level proximal policy opti- mization with level reward guidance. The problem
Grattafiori, A., Dubey, A., Jauhri, A., and et al. The Llama 3 herd of models, 2024. URL https: http///xxvvoorr// abs 0/1/2240.22222
Guo, S., Zhang, B., Liu, T., Liu, T., Khalman, M., Llinares, F,, Rame, A., Mesnard, T., Zhao, Y., Piot, B., Fer-

---

TGDPO: Harnessing Token-Level Reward Guidance for Enhancing Direct Preference Optimization
ret, J., and Blondel, M. Direct language model align- ment from online Al feedback, 2024. URL https: //axxiv. org/ abs 04792.
Ouyang, Y., Wang. L., Yang. F., Zhao, P., Huang. C.,, Liu, J., Pang, B., Yang, Y., Zhan, Y., Sun, H., Lin, Q. Ra- jmohan, S., Deng, W., Zhang, D., Sun, F., and Zhang, Q. Token-level proximal policy optimization for query generation, 2024. URL https: /ttr/aa.voorrg.eeeee /abs/ 00722.
Hu, J., Wu, X., Zhu, Z., Xianyu, Wang, W., Zhang, D., and Cao, Y. OpenRLHF: An casy-to-use, scalable and high- performance RLHF framework, 2024. URL https:
//arxiv. org/ aba/2405 2405.. 11143.
Park, R., Rafailov, R., Ermon, S., and Finn, C. Disentan- gling length from quality in direct preference optimiza- tion. In Findings of the Association for Computational Linguisties: ACL. pp. 4998-5017, 2024.
Ivison, H., Wang, Y., Liu, J., Wu, Z., Pyatkin, V., Lambert, N., Smith, N. A., Choi, Y., and Hajishirzi, H. Unpacking DPO and PPO: Disentangling best practices for learning from preference feedback. In The eiihth Annual Conference on Neural Information Processing Systems, 2024.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimiza- tion: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems, volume 36, pp. 53728-53741, 2023.
Jiang, D., Ren, X., and Lin, B. Y. LLM-Blender: Ensem- bling large language models with pairwise ranking and generative fusion. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023.
Rafailov, R., Hejna, J., Park, R., and Finn, C. From r to q": Your language model is secretly q-function. In First Conference on Language Modeling, 2024.
Li, T., Chiang, W.-L.., Frick, E., Dunlap, L., Zhu, B., Gonzalez, J. E., and Stoica, I. From live data to high-quality benchmarks: The Arena-Hard pipeline, April 2024. URL https:// / Imsys org/blloggg 2024-00 04-0041199 aa--aareaaaaaaae
Schulman, J., Wolski, E., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algo- rithms, 2017. URL https: http///xxvvvv././..//.
1707.06347.
Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang, P., and Hashimoto, T. B. AlpacaE- val: An automatic evaluator of instruction-folloiing models. hitps: : htt////gttt////////////////ttttt wttptu/...ootttttuttaaaaaae. alpaca_eval, 2023.
Shao, R., Li, B., Liu, G., Chen, Y., ZhouXiang, Wang, J., Cai, X., and Li, P. Earlier tokens contribute more: Learn- ing direct preference optimization from temporal decay perspective. In The Thirteenth International Conference on Learning Representations, 2025.
Loshchilov, I. and Hutter, F. Decoupled weight decay reg- ularization. In International Conference on Learning Representations, 2019.
Snell, C. V., Kostrikov, I., Su, Y., Yang, S., and Levine, S. Offline RL for natural language generation with im- plicit language learning. In The Eleventh International Conference on Learning Representations, 2023.
Meng, Y., Xia, M., and Chen, D. SimPO: Simple prefer- ence optimization with reference-free reward. In The Tiihth- Annual Conference on Neural Information Processing Systems, 2024.
Team, G., Anil, R., Borgeaud, S., and al. Gemini: A family of highly capable multimodal models, 2024a. URL https: /tt//aaiivv////////v////////aaaa org/
11805.
Miao, Y., Gao, B., Quan, S., Lin, J., Zan, D., Liu, J., Yang, J., Liu, T., and Deng, 7. Aligning CodeL.LMs with direct preference optimization, 2024. URL https::t//// arxivv. org/abs/ 2410. 18585.
Team, G., Riviere, M., Pathak, S., and al. Gemma 2: Im- proving open language models practical size, 2024b. URL https: / arxiv org/ abs / 2408. 00118.
Tian, K., Mitchell, E.,, Yao, H., Manning, C. D., and Finn, Fine-tuming language models for factuality. In The Twelfth International Conference on Learning Representations, 2024.
OpenAI, Achiam, J., Adler, S., and et al. GPT-4 techni- cal report, 2024. URL https://arxiv org/ abs/ 2303. 08774.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C.,
Wang, H., Xiong, W., Xie, T., Zhao, H., and Zhang, T. Inter- pretable preferences via multi-objective reward modeling and mixture-of-experts. In Findings of EMNLP, 2024.
Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, I.., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., and Lowe, R. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pp. 37730-27744 2022.
Wu, Z., Hu, Y., Shi, W., Dziri, N., Suhr, A., Ammanabrolu, P., Smith, N. A., Ostendorf, M., and Hajishirzi, H. Fine- grained human feedback gives better rewards for language
10

---

TGDPO: Harnessing Token-Level Reward Guidance for Enhancing Direct Preference Optimization
model training. In Thirly-seventh Conference on Neural Information Processing Systems, 2023.
Xia, H., Gao, S., Ge, Q.. Xi, Z., Zhang, Q. and Huang, X. Inverse-Q2 Q Token level reinforcement learning for aligning large language models without preference data. In Findings of the Association for Computational Linguis- tics: EMNLP, pp. 8178-81 17888188, 2024.
Yang, S., Zhang, S., Xia, C.,, Feng, Y., Xiong, C., and Zhou, M. Preference- grounded token-level guidance for language model fine-tuning. In Thirty-seventh Conference Neural Information Processing Systems, 2023.
Yang. S., Chen, T., and Zhou, M. dense reward view on aligning text-to-image diffusion with preference. In Pro- ceedings of the 41 st International Conference on Machine Learning, pp. 55998 55998-56032, 2024.
Yin, Y., Yang, S., Xie, Y., Yang. Z., Sun, Y., Awadalla, H., Chen, W., and Zhou, M. Segmenting text and learning their rewards for improved RLHF in language model, 2025. URL attps://arxiv.cr 2501. 02790.
abs/
Zeng, Y., Liu, G., Ma, W., Yang, N., Zhang, H., and Wang, Token-level direct preference optimization. In Proceed- ings of the 41st International Conference Machine Learning, 2024.
Zheng, L.,, Chiang, W.-L., Sheng, Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging LLM-aa-a-judge with MT-Bench and Chatbot Arena. In NeurlPS Datasets and Benchmarks Track, 2023.
Zhong, H., Feng, G., Xiong, W., Cheng, X., Zhao, L.,, He, D., Bian, and Wang, L. DPO meets PPO: Reinforced token optimization for RLHF. In ICML. 2024 Workshop Models of Human Feedback for AI Alignment, 2024.
Ziegler, M., Stiennon, N., Wu, J., Brown, B., Radford, A., Amodei, D.,, Christiano, and Irving, G. Fine-tuning language models from human preferences. 2020. URL https: / /arxivv. org/ abs///bb//aa 1909.99.. 08593.

---

TGDPO: Harnessing Token-Level Reward Guidance for Enhancing Direct Preference Optimization
A. Mathematical Derivations
A.I. Proof of Theorem 4.1
Theorem A.1. The maximum value of the sequence-level proximal policy optimization in Equation (2) is upper bounded by the summation from 0, 0, 0.1....,T of the maximum value of the problem:
Te ((((())))
max De ro ro 04) - BBlog
Ted (1 st)
where St Di denotes that So D and up To(-1sp), p 0,
Proof. According to Section 4.1, ~ II: To (a(ar(s)) in Equation (2) is equivalent to ~ TT(-12)) which is further equivalent to 80 = I D, ap ~ mo(-1sp), = 0, ...... Thus for the sequence-level proximal policy optimization in Equation (2),
Σ Crls ax) - -- Blog To(U)>)
max E
D. Dr IT (aclss)
Tre((y((z)
To ()(1)x)
max E na
ro(se: ar) - -Blog
[x)(()
T-1
B Blog TT(ae(8)) Trer (cels))
max 1tt.a//oo.......r...............
To(se, ae) Σ
(by Equation (6))
ro (0( at) - / Blggg Te(ue(8z)
max Eg Bo (en.D
TTer
To (0(((1))) ro (1 ax) - -Blog
max
B -D
Teell a1 (st)
m=0
To (a(a St) Tee((a (s))
max Egg D. D. T-1
ro(s); -Blog
max Be ro(s), Glog To(ac(8e) Teet (aels))
Fa
where ~ D, denotes that so x D and ap Ta(-1sp), 1,1,...,t 1. This completes the proof.
A.2. Proof of Theorem 4.3
Theorem A.2. The optimal policy for the action a, at time step of the modified token-level proximal policy optimization in Equation (11) is:
ng, (aelse) The((rlsi) exp
ra(siaa))
(f(((8)04)
(22)
Z(xt)
where Z(s:) =(si)=Earrm.di lxc)
is the partition function and D does not depend ne.- Moreover. the token-level reward under the optimal policy is given by
exp
(f(F(s,aa))
ro(St, at) =Blog Te, (atlst) Tre((a)s(s)
+ 3 log Z(s)).
(23)
f(f(s),
Proof. In Equation (11), the modified token-level proximal policy optimization is:
(o(s),))) a:)
Te(a (s) Teat (ael st))
max B.
log
Bf(((st,
12

---

TGDPO: Harnessing Token-Level Reward Guidance for Enhancing Direct Preference Optimization
[er(ac]))) exp
ralsr.a)) (F(F()),a)))
max E g Da
log
X
To To(a (st)
rola).a)) Tre((ac(s)) exp
max E, De
log
log (st)
((s))To(aa
x0
Z(8c) Fee (8)) exp
ralen.e)) (T((F())))))
max B (aattoo((s)
log
+ log ((s))
(t)
Z(x+) Teet (ree(aees (8) exp
3f(F(sr,ar]]
max Eg DiD
E Bai log
+ log ((s))
Xo
T8 (ag)s1)
(5 ac) (f((((t,
max Egg
- Dxx. no((elss)]l
Tee((a)lst) exp
+ log ((st)
(24)
Z(8r)
where the partition function (s;) = Easwima
exp rolscaa)) (F((()))a)))
is independent of mа. Then we can define
Ter(ar)s)) exp
ralec.oi) (F((())))))
Te, (arlst)
Z(st)
which is a valid probability distribution of action at- Furthermore in Equation (24), since ((x)) independent of mo, the optimal policy for the action a, at time step i of the modified token-level proximal policy optimization in Equation (11) can be in the form of Equation (22).
By reorganizing Equation (22), we can get the token-level reward in Equation (23). This completes the proof.
A.3. Proof of Bradley-Terry Model with Token-Level Reward Guidance in Equation (17)
Let
-w Σ
To(vinl<rr ( Tied [z] yo flt
To(xi [z, v/-1)
(PTTT, f,;,,, yw: vi)
ff (f(x, "nl v.)) log
(fi(f([z,)) "I, ())) log
m [x. v1' xtt
1=0
(25)
T--1
T
8(f. f, v1)
Σ
8f (f((z, vo (.v))) log ([2, v0d
Σ
(f((f((,,)) v.)) log 2 (26)
By Equation (15) and the choices of fu and fi in Section 4.3,
(x. Bho)
Tw
(o (11, "e v(
Te
no(u [x, y (
Σ
(f y vle)) log
+ 3fm v v(.))) log ((xx, y
Teo (z, y:(!))
Th
me Te( | )
Te
Σ
(f wos w(u)) log
+
Σ
(f wo 1. log Z(12, ward
Feee( (2r, y
13

---

TGDPO: Harnessing Token-Level Reward Guidance for Enhancing Direct Preference Optimization
Similarly,
(2 m))
Talla, vi (.
To(yi Iz, f/ "1) Teed (vf [r, 1)
T--
(f((f((a, vi "I.t))))))) log
ff vi "I log Z(12, v1
In the above two equations, To and Ti are the lengths of Y/u and respectively. Thus using the notations in Equations (25) and (26) we get
(z yw) (2, w) ( fo f;z, ywu y1) 8(f, 2; yu y1).
Then the Bradley-Terry model with the token-level reward guidance is
Pr(y > 11/2)))
exp (r (2, )w)) Bu ) + exp (ro(x,
(27)
1+ exeep/ (r 91 (e(
a(ro(1, vw) - (1), a (f((TT, f, f yw v1) olf, P3 yw 11))
A.4. Proof of Theorem 4.4
Pr(y 31(z) a (p(TTO: f,f; yres vi) + 8(f,f Bw 31)),
(28)
in which d(f, f; Mwo 31) does not depend on the policy Ta to be optimized, but only on f, x. Uwr and the partition function (ss)) (also does not depend on To. please see Theorem 4.3 in the main text). Since (f) is the sigmoid function which is a strictly increasing function of I, we have:
Theorem A.3. The function in Equation (28) has the same maxima and the same ascent directions as the function (pp(T,, f, f, x, ywr B1)). Moreover, for two policies Ta, and T8.
yer a (pp(To, f, f.f:2, yw y1)) > a (u((Te f. f.f; Ywr v1)
(29)
if and only if
a (p((TT) J,P; Vw 9i) 8(f, 23 Uw 91) (p((T f. Fi yw v)) + 8(f, f: Vwes 91))
(30)
Proof. Note that, ö(J, P3,, ywr y1) is not dependent the policy ma. and for the sigmoid function o(),, it holds that d'(t) > 0 for all t. Then, by the definition, d is an ascent direction of the function (28) if and only if
d' 4 V (p((TO: 1,F:2, Nw: v1) + 8(f. P3 x. Fue 31) > 0.
which is equivalent
o' (v(TT, S. Pi tw: vi) + 8(S, f; Mer vi)) V (p(T 1, Uen mn) > 0
- d" V x S 1, Vw. yi)
- df o' (pp(Ter f.f; yo w)) V (To, f. f; x, Bor v1)
- dt V (yp(TT, f. P; ywr v1) > 0.
14

---

TGDPO: Harnessing Token-Level Reward Guidance for Enhancing Direct Preference Optimization
Hence the function (28) has the same ascent directions as the function a (sp(TT, F3 Mws W1) Similarly,
V To a (poTTo, J. f. Mees y1) + {[, F; n. Bw. uc)
o' (p(TTO, f. F; Mu 31) + 8 (f. P: Me M1) V Vo (TT, yw 31) 0
P fo F; Mw y1 = 0
o' (p((TT, J. 7; Mh: 11) V (TTO, f. 2, Mer wt) =
Vng (vo(TT, f, 2, you u1) = O.
Thus, the function in Equation (28) has the same maxima and the same ascent directions as the function a (o(TT, J, f xr, Vw. ()))
Next, since (l) is strictly increasing, for inequality Equation (29) we have
(p((Te, f. f.f; 8)) > (T( f, f; x, y1)) - (TT) J f yo m1 > ( S, f; w. you yt)
((To, J, f; x, Um, m) + 6 F; Mrrs 11) > (T I f; yon w1) + + Uw 11 Yw v) {f. x, yw u)) > a (sp(TT f. f,A; x, yw vi) 5(f f; yu. W1)
(fI(TT; vw
For easy understanding of Theorem A.3, we simplify in the sequel all notations independent of me to be optimized, then the Bradley-Terry preference model in Equation (28) Pr(yer > 91|2) = a (vp( σo) + 6). and Theorem A.3 is exactly as: Theorem A.A. For the policy no. the function (Lp(T)) + 5) has the same maxima and ascent directions as the function functionnn function.
(v((T))), here (t) is the sigmoid
Proof. Note that the sigmoid function (t) is strictly increasing, meaning that for any real numbers and b, 2 if and only if (aa) 2 (b). Thus, if गह is maximal solution of (T)(T))) + 6), then by the definition, there exists neighborhood N of nz such that Vng E N, ((((T)) + 6) 2 ((p(m))) + 8). So (=rā) + 5 2 (p(mr) + ó, and (P(T)) 2 (p(T)) ). This leads to of ((((T))) 2 (Vp(TT)), meaning that σ is also a maximal solution of ar (sp(TT))) The converse can be proved similarly.
Next, is an ascent direction of the function ((((TT)) + 6) if and only if
df V ( + (v((TT)) + V p(T)) > 0,
which is equivalent to
Vno > 0 o' (p((T)) ))dd V d' o' (((TT)) >
d (((T) 0.
Hence the function ((p(TT)) + 6) has the same ascent directions as the function ((P(TT)) w.r.t. np.
Further, the sigmoid function is strictly increasing, it does not change the order of values.
B. More Experiment Results
B.1. Additional Baseline Comparison
Below supplement the result of TDPO (Zeng et al., 2024) as additional baseline for the experiment in Table of the main paper. The additional result is demonstrated in Table 5. It can be seen that the performance of TDPO is very close to that of DPO. Our TGDPO, on the other hand, outperforms DPO by a large margin. Our TGDPO aims to leverage an existing token-level reward to guide DPO training at the token level. Whereas, TDPO (Zeng et al., 2024) aims to enhance the regulation of KL-divergence by incorporating a forward KL-divergence for each token to the DPO objective. is not guided by a token-level reward.
15

---

TGDPO: Harnessing Token-Level Reward Guidance for Enhancing Direct Preference Optimization
Table Experiment results AlpacaEval 2 eLi al., 2023), Arena-Hard (I. al., 2024), and MT-Bench (Zheng al., 2023) benchmarks.
-lama3-8B-Instrvee PairRM
Llama3-8B-Instee ArmoRM
Method
AlpacaEval
Arena-Hard
MT-Benchh
AlpacaEval 2
Arena-Hard
MT-Bench
Win Rate (%)
Win Rate (%)
Score
Win Rate %))
Win Rate (%)
Win Rate (%)
Score
Win Rate(%)
SFT
30.6
21.4
7.9
27.5
30.6
21.4
27.5
DPO
41.7
30.4
8.0
37.5
40.8
36.2
8.2
46.3
TDPO
40.7
30.2
8.0
39.0
41.3
36.7
8.0
42.5
SimPO
39.8
28.7
7.8
32.5
37.0
28.1
7.8
42.5
TGDPO
43,9
34.3
8.0
41.9
42.5
40.5
7.9
45.0
L ArmoRM
Gemma2-2B-i ArmoRM
Method
AlpacaEval
Arena-Hard
MT-Benchh
AlpacaEval 2
Arena-Hlar
MT-Bench
Win Rate (%)
Win Rate (%)
Score
Win Rate (%
Win Rate (%)
Win Rute (%)
Score
Win Rate (%)
SFT
23.8
17.1
7.0
16.3
32.8
20.1
37.5
DPO
29.6
23.2
7.9
29.4
40.8
26.4
8.0
43.1
TDPO
30.3
23.1
7.8
30.0
41.5
27.0
8.0
40.0
SimPO
26.2
22.6
7.4
15.7
34.8
21.1
40.0
TGDPO
35.8
25,4
8.1
36.9
43.0
30.7
8.1
46.9
Table Experiment results on SFT models AlpacaEval (Li al., 2023), Arena-Hard (Li et al., 2024), MT-Bench (Zheng al.,
2023) benchmarks.
Llama3-SB-SFT-Miitee Ultrafeedback
Method
AlpacaEval
Arena-Hard
MT-Bench
Win Rate (%)
Win Rate (%)
Score
Win Rate( %
SFT
5.0
7.6
16.3
DPO
9.9
10.2
7.8
19.5
TDPO
11.0
11.7
15.7
SimPO
16.4
21.4
7.8
27.5
TGDPO w/ DPO's token reward
12.8
13.8
20.0
TGDPO w/ SimPO's token reward
26.9
25.3
7.6
31.9
B.2. Experiment on SFT Models
In this section, we conduct experiments starting from SFT models. Specifically, we use the open-source SFT model from OpenRLHF (Hu et al., 2024). Laama--B-SFFTMixxur is trained using diverse, high-quality, open-source datasets by SFT and has been trained by RLHF. Following (Meng al., 2024), we conduct preference optimization on the UltraFeedback dataset (Cui al., 2024) using the SFT model the starting point.
The experiment results SFT models on AlpacaEval (Li et al., 2023), Arena-Hard (Li al., 2024), and MT-Bench (Zheng et al., 2023) are shown in Table 6. Our TGDPO can leverage the token-level rewards from DPO SimPO and outperforms them correspondingly. Specifically, TGDPO using SimPO's token-level reward achieves much better performance than all baseline methods. achieves win rate gains of 10.5 on AlpacaEval 2, 4.4 on MT-Bench, and 3.9 on Arena-Hard compared to best-performing baselines.
C. More Discussions on Closely Related Work
Our work proposes framework for incorporating existing token-level rewards explicitly into the loss function of DPO, to guide optimizing policy at fine-grained level. This is challenging task since DPO's reward function is explicitly expressed through the policy being optimized. Especially, a key theoretical challenge in deriving the computable loss function in Equation (20) is the climination the partition functions, which addressed in Theorem 4.4 or Theorem A.3 or Theorem A.4.
16

---

TGDPO: Harnessing Token-Level Reward Guidance for Enhancing Direct Preference Optimization
C.I. Closely Related Work
Table Per-instance losses of closely related direct optimization methods.
Method
Per-Instance Loss
TDPO (Zeng et al., 2024)
ou((,, Uws 31) (xx, the B1)).
where u(x, Uwr w)) = 3 log mo(ymlx) sarluul [2)
Blog sa(p) star (ar)
fx,, Ywn vi)) = f B.Dsog (3r, y05 Tredl (TO) ,3 Droagk (2r, Uui Tred (Ta))
Yang et al. (2024)
CE (}))) [log sola""a"" 1a3) Tuatly (8)
log (a(aloo)) Tarl Tar(a)1o)))
D'PO (Shao et al., 2025)
σ
Y 3 log
To le.
E 7 log Tmar (x.
TGDPO (ours)
Tu-
Bfw (f vi (()))) log
no mo(u 2
Ei= (F((F((T,
(("). log
no [ ( Tuttoil [.
murlis (z,55
Several direct preference optimization methods also perform in a token-level manner. We derive our modified token-level reward beginning from Equation (9), which is similar to those in Zeng et al. (2024) and Yang al. (2024). However, the obtained final per-instance losses different. These per-instance losses are listed in Table 7 for comparisons. From the table, it is obvious that our TGDPO explicitly incorporates existing token-level rewards into the per-instance loss for guiding DPO. While, TDPO (Zeng et al., 2024) constrains each token with forward KL-divergence, and fine-tunes pre-trained LLMs from the token level to enhance the regulation KL-divergence. Additionally, Yang et al. (2024) and DOPO (Shao et al., 2025) focus on earlier tokens of sequential generation for their tasks, by posing temporal decay parameters.
Moreover, the derivation of our TGDPO, the partition function 2()) not dependent the policy be optimized, and can be eliminated from the loss function by using our developed Theorem 4.4 or Theorem A.3 or Theorem A.4, which new and powerful. While, in TDPO (Zeng al., 2024) the partition function is kept their loss function and is changed to the forward KL--ivergence. Yang et al. (2024) managed to eliminate the partition function from their loss function using the lower bounding approach. The method in DO (Shao et al., 2025) does not involve a partition function, since is derived from the KL-constrained RL objective under the maximum entropy RL setting.
C.2. Recovering Several Direct Preference Optimization Methods
In Section 4.2, we mentioned that the loss function (TG)PO(TT) in Equation (20) provides framework of direct preference optimization with token-level reward guidance. With an appropriate choice of f(-), this framework can recover several known direct preference optimization methods. For example, if we take fu = fi = 1, then Equation (20) is the loss function of DPO (Rafailov al., 2023). In the following, we give some other examples. It must be noted that these known preference optimization methods have their respective motivations. We only want to demonstrate that our proposed framework is reasonable by recovering them here.
Note that, our per-instance loss of LTGDPO(TT) is
To Σ
To [x, y =t) ( ff [r, y=(l
T:-1 ff((f((z, (I))) log
no Toliili 31 r1 م [z, wi²²()
STGDPO.(TTT) =0
3f (f(1x, " v..) log
t=0
1. Recovering the per-instance loss of SimPO (Meng al., 2024): By setting for (f(x,, v ((())(()))
1 and
17

---

TGDPO: Harnessing Token-Level Reward Guidance for Enhancing Direct Preference Optimization
[i(((a, "I,,
get
( [r Mu
T:
3
To(vil< 3)r
LTGPPO p(TO)
log
log
Treel (yfe [2 Vie
V1
(erl [x, f (1)
( [r, yan'))
To 11 "1 TTcl [z, yi (
log
log
Tiee
B
To (Tw(Hlll) (z)
B
(31
log
log
Tear (x)
111
Terr (r))
B
B
B 18
B
log (Be())) ))
log To (N (x)
log To (v1(z) +
log Tee (t)
Furthermore, by Theorem 4.4 or Theorem A.3 or Theorem A.4, introducing a constant into the above function does not change where the function is maximized. Hence we get
(1)) (1bull log (o(olz))) - leul
B
LTGDPOL((T))
log (()
which is exactly the per-instance loss of SimPO.
2. Recovering the per-instance loss of R-DPO (Park et al., 2024): By setting fw (f((z, v ve"") ((")) = f((F((a, 1. we get
To To [2t voll) Trer y55
Ti-1
To ao(il(a,, vi Teer [x, (1(
(ICDPO_((TO) 0
Blog
Σ
Blog
1=0
To (v B log To (3112)
Blog
(re(( (x)
Tee xx)
Furthermore, since a alyy.l a yrl does not depend on the policy Ta. by Theorem 4.4 or Theorem A.3 or Theorem A.4, introducing into the above function does not change where the function is maximized. Hence we get
LTGDPO.P(TE) o ( log
Tel (B(Wu (x) Teet ( 2)
3 log T (1)((11
+ (al/ww - -a(en)))
which is exactly the per-instance loss of R-DPO.
3. Recovering the per-instance loss of DPPO (Shao et al., 2025): By setting fm y v(a)) (f((1,, (illa,vv"1 v1 y', we immediately get
To
To I,,, [0, "1 Tee f [r, v15"
To(يi 1Ez, w1( T By log
LTGDPO.P(TO) =
Σ
3y log
Tue Ir, 31 ()=
t=
which is exactly the per-instance loss of D*PO
D. Implementation Details
D.1. Hyperparameter Settings
Following (Meng et al., 2024), we use consistent batch size of 128 and train all methods for 1 epoch in all settings. The AdamW optimizer (Loshchilov & Hutter, 2019) is used. The max sequence length is set to be 2048 and a cosine learning rate schedule with 10% warm-up steps is used. The hyperparameters for each method are grid-searched and are shown in Table 8 for DPO, Table 9 for SimPO, Table 10 for our TGDPO correspondingly. TDPO in Appendix B.1 uses the same hyperparameters as DPO with an additional KL--penalty scale of 0.01. The training is conducted using A A00 GPUs.
18

---

TGDPO: Harnessing Token-Level Reward Guidance for Enhancing Direct Preference Optimization
Table The hyperparameters of DPO for each training setting.
Setting
learning rate
LJama3-8B-Insie Inssrrutttt PairRM
0.01
7e-7
(oama3-8B-Instreee ArmoRM
0.01
7e-7
L I-nttttre-t ArmoRM
0.1
7e-7
Gemma2-2B-H ArmoRM
0.1
Se-7
Llama3-8B-SFT-Mixuue Ultrafeedback
0.1
5e-7
Table 9. The hyperparameters of SimPO for each training setting.
Setting
?
learning rate
L PairRM
1,4
le-6
-lama3-8B-Instruue ArMoRM
10
3.0
le-6
(lamaa.2-3B-Instruu ArmoRM
10
3.0
le-6
Gemma2-2R-it ArmoRM
20
2.0
5c-7
Llamaa--B--PP-MMxtture Ultrafeedback
2.5
0.5
5c-7
Table 10. The hyperparameters of TGDPO for each training setting.
Setting
for f(s,, oz) Y for F(se,
Q
learning
Lluma3-8D Instrutt PairRM
0.01
0.1
0.5
7e-7
-nn-sruuutt ArmoRM
0.01
0.1
0.2
7e-7
Llama3,2-3B- Instruct ArmoRM
0.1
0.1
2.0
7e-7
Gemma2-2B-it ArmoRM
0.1
0.1
0.5
Se-7
Ultrafeedback w/ DPO's token reward
0.1
0.1
0.2
7e-7
Jlama3-8B-SFT--Mixtuu Ultrafeedback w/ SimPO's token reward
2.5
0.5
0.01
1.2
7c-7
D.2. Benehmark Details
Following (Meng et al., 2024), we use a decoding temperature of 0.9 for the Llama models and decoding temperature of 0.5 for the Gemma models for AlpacaEval For Arena-Hard, we use the default greedy decoding for models. We the latest GPT-40-2024-1 1-20 the judge model for AlpacaEval and Arena-Hard. We follow the official default configurations MT-Benchh
19


Reviews:
[{"review_id": "uJIk9vboez", "reviewer": "Reviewer_GJSH", "summary": "The paper presents TGDPO, a formulation of direct preference optimization (DPO) with an implicit token-level reward instead of an implicit sentence-level reward, and shows that this novel method outperforms standard DPO and provides interpretable training dynamics\n\nMore precisely, the paper makes the following contributions and claims:\n1) It derives a token-level RL-finetuning objective that is tractable to optimize (Eq10) and apply DPO to derive a token-level DPO loss.\n2) TGDPO outperforms DPO and other variants in popular alignment benchmarks.\n3) TGDPO provides satisfactory policies when trained until convergence and is robust to the modified token-level reward pre-training phase.\n\n## update after rebuttal\nAll my concerns have been addressed. It also seems that the other reviewer's concerns have either been addressed or are not critical. I maintain my score from the end of the rebuttal.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1 Experiment on SFT model**\n\nThank you for the advice! Below we show the experiment results on UltraFeedback using the open-sourced SFT model OpenRLHF/Llama-3-8b-sft-mixture, which has not been trained by RLHF. Our TGDPO using SimPO's token-level reward achieves much better performance than baselines. Specifically, it achieves win rate gains of 10.5 on AlpacaEval 2 and 3.9 on Arena-Hard compared to best-performing baselines.\n\n|  | Arena-Hard win rate |  AlpacaEval 2 win rate | MT-Bench score |  MT-Bench win rate | \n| - | - | - | - | - |\n| SFT | 6.2 |5.0 |7.6 |16.3 |\n| DPO | 10.2 |9.9 |**7.8** |19.5 |\n| TDPO | 11.7 |11.0 |7.5 |15.7 |\n| SimPO | 21.4 |16.4 |**7.8** |27.5 |\n| TGDPO w/ DPO's token reward | 13.8 |12.8 |7.7 |20.0 |\n| TGDPO w/ SimPO's token reward | **25.3** |**26.9** |7.6 |**31.9** |\n\n**Q2 Setting difference between SimPO (Meng et al., 2024) and ours**\n\nThe key difference in the experiment setting is that we use the latest and more powerful gpt-4o-2024-11-20 as the judge model, while SimPO uses gpt-4-1106-preview (gpt-4 turbo), which was released in Nov 2023.\n\nBelow we compare the Arena-Hard win rate of Llama3-8B-Instruct PairRM using these two judge models. The win rate judged by gpt-4-1106-preview is generally consistent with the SimPO paper, while the result is different from the latest gpt-4o-2024-11-20. This is the reason for SimPO underperforming DPO in Table 1.\n\n|  | gpt-4-1106-preview |  gpt-4o-2024-11-20 | Avg | \n| - | - | - | - |\n| DPO | 32.9 |30.4 |31.7 |\n| SimPO | 33.5 |28.7 |31.1 |\n| TGDPO | 36.9 |34.3 |35.6 |\n\n**Q3 Starting point of TGDPO training**\n\nWe would like to clarify that DPO, SimPO, and our proposed TGDPO have the same starting point for training, which are Instruct or SFT models. As described in Equation (16), TGDPO is designed to leverage any token-level reward models, including pre-trained DPO or SimPO models. We can use off-the-shelf open-sourced token-level reward models. So it is not necessary to train a token-level reward model by ourselves before starting TGDPO. Furthermore, TGDPO can enjoy faster convergence speed with satisfactory performances, as demonstrated in Figure 1 and Table 3 of this paper. Also, naively increasing the training budget for DPO or SimPO usually does not improve the performance. As demonstrated in Table 2, training DPO and SimPO to convergence leads to worse performance. We will clarify the description to avoid misunderstanding.\n\n**Q4 Clarification of Equ. 15**\n\nWe followed the standard way of conducting experiments and TGDPO consistently outperforms baselines. We believe this empirical validation is a key strength of our approach, as it shows that the theoretical insights lead to practical improvements.\n\nFrom theory aspect, now the related assumptions have been completely removed, the approach is novel, and the intuition is outlined below:\n\nLet \n$$h=( f, \\hat{r}; x, y_w, y_l). \n$$\nThe Bradley-Terry model in Equ. (15) is\n$$\n \\Pr(y_w \\succ  y_l | x)  = \\sigma\\left( \\varphi(\\pi_{\\theta}, h) + \\delta(h) \\right).\n$$\nWe find that $\\delta( h)$ does not depend on the policy $\\pi_{\\theta}$ to be optimized,  but only on $f, \\hat{r}, x, y_w, y_l$ and the partition function $Z(s_t)$ (does not depend on $\\pi_{\\theta}$, see Theorem 4.3 in the main text). Since $\\sigma(t)$ is the sigmoid function with $\\sigma'(t)>0$, the above $\\Pr(y_w \\succ  y_l | x)$ has the same maximal solution and the same ascent direction as the function $\\sigma\\left( \\varphi(\\pi_{\\theta},h)\\right)$ with respect to $\\pi_{\\theta}$. \n\nHence, since we focus only on the maximal solution $\\pi_{\\theta}$, we may redefine \n $$\\Pr(y_w \\succ  y_l | x) \\triangleq \\sigma\\left( \\varphi(\\pi_{\\theta}, h)\\right), $$\nand use it for constructing the loss function of our preference optimization. \n\nThe detail can be seen in response to **Q3 of Reviewer F88J**. We will provide the intuition in the final version.\n\n**Q5 Difference with TDPO (Zeng et al., 2024)**\n\nOur work aims to leverage an existing token-level reward to guide DPO training at the token level. Whereas, TDPO aims to enhance the regulation of KL-divergence by constraining each token with forward KL-divergence. It is not guided by a token-level reward.\n\nWe will add more discussions on the differences with TDPO in the final version.\n\n**Q6 Clarification on line 215**\n\nThe paragraph demonstrates that it is possible to obtain a token-level reward, using the approach in [1, 2] or DPO. And, the token-level reward will be adopted for shaping the reward in PPO and subsequently the loss function of DPO. \n\nWe will make the description better.\n\n[1] Preference-grounded token-level guidance for language model fine-tuning\n\n[2] Segmenting text and learning their rewards for improved RLHF in language model", "claims_and_evidence": "Overall, the claims in the paper are backed by enough evidence, but I have concerns about missing information in the experimental setup. \n\n1) The derivation is backed by theoretical results with clear proofs. See the Theoretical claims for details.\n2) Evidence is provided in Table 1, but I have concerns regarding the experimental setup. See the Experimental Designs section,\n3) This is well presented in Figure 1 and Tables 2, 3, and 4.", "methods_and_evaluation_criteria": "I believe that starting from closed-data, fully post-trained models like Llama3-8B-Instruct and Gemma-2-2B-it and then performing further alignment on them gives misleading numbers to the community, as they suggest that the methods presented improve the model further, while what is happening is that they distort their post-training distribution.\nThe authors do not claim to make the models better than what their original authors did, so this is okay in this paper, but I would strongly recommend the authors to consider only SFT models if possible or add a remark about this.\n\nOtherwise, the choice of evaluation benchmarks and datasets is good. The results would be stronger with an additional dataset, which would be more impactful than varying the reward model, as it only serves for preferences.", "theoretical_claims": "I verified the proof of Theorem 4.1. in Appendix A.1.\nThe theorems to derive the optimal policy and the resulting reward are straightforward modifications of the results in the DPO paper. I did not verify their proofs but believe they hold.\nEquation 15 is an important result, but its proof is in the appendix. It would have been nice to provide some intuition in the main paper. I verified the proof. It makes several assumptions, which may or may not hold depending on the data considered. To me, what's important for the paper in this case is that the empirical results still show improvement with this derivation..", "experimental_designs_or_analyses": "It's not clear what the starting point of the TGDPO training is. Is it from the already trained DPO model?\nTGDPO seems to use double the budget and DPO as it has to train a policy before starting. The authors do not discuss this computational consideration.\n\nIt's not clear why SimPO consistently underperforms DPO in Table 1, although the paper uses a similar experimental setup as Meng et al. (2024). This raises concerns regarding the experimental protocol.", "relation_to_broader_scientific_literature": "Sufficient.", "essential_references_not_discussed": "The authors reference the essential prior work and discuss the differences with the work of Zeng et al. (2024), which is critical to clearly identify the contributions of the paper.", "other_strengths_and_weaknesses": "Clarity:\n-The paragraph in lines 215 column two seems grammatically incorrect. It's not clear what point it conveys.\n-I would have appreciated more motivation for the Modified Token-Level PPO problem. Its presentation is a bit abrupt. \n+Nevertheless, the interpretation of the loss derived from it with the practical considerations makes up for this lack of earlier motivation. Perhaps its possible to rewrite some parts to connect these two motivations?", "other_comments_or_suggestions": "Open to reconsidering my score given input from the authors. My main concern is about missing clarifications for the results and experimental protocol.", "questions_for_authors": "No additional questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fJpfYm6Tf4", "reviewer": "Reviewer_uVtj", "summary": "This paper introduces TGDPO, an enhanced version of Direct Preference Optimization (DPO) that incorporates token-level reward guidance to address the limitations of conventional sentence-level DPO. While prior methods like Proximal Policy Optimization (PPO) benefit from fine-grained token-level rewards, DPO, formulated as a sentence-level bandit problem, struggles to leverage such granular signals. To bridge this gap, the authors decompose sentence-level PPO into a sequence of token-level PPO problems, enabling the derivation of a closed-form optimal token-level policy and its corresponding token-level reward. By integrating these token-level rewards with the Bradley-Terry preference model, the proposed TGDPO algorithm introduces a new loss function that guides optimization at the token level.\nExperiments on MT-Bench, AlpacaEval 2, and Arena-Hard demonstrate TGDPO’s superiority over standard DPO. The method also experimentally exhibits robustness to variations in token-level rewards and provides control over convergence speed.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1 TGDPO's dependence on the choice of $\\hat{r}$**\n\nTrained token-level rewards have shown effectiveness for PPO [1, 2]. For DPO, it is interesting to ask if there exists a framework that can incorporate a trained token-level reward explicitly for better performance. Our TGDPO fills this nontrivial gap.\n\n**Q2 Strong assumption for Equ. 15**\n\nThanks for the insightful comment! Now the assumptions have been removed, the approach is novel. The theorem and proof can be seen in Response to **Q3 of Reviewer F88J**. We will make revisions in the final version.\n\n**Q3 How to get the Equation in Lines 311-313 (left)? How to get $\\hat{r}$ in practice?**\n\nThm 1 of [3] shows \n$$r(s_t,a_t) = \\beta \\log \\frac{\\pi_{\\theta}(a_t|s_t)}{\\pi_{\\text{ref}} (a_t|s_t)}$$ \nunder equivalent state-action reward class and invariant re-parameterization. The final loss function in [3] is equivalent to that of DPO [4]. So we adopt the result by setting $s_t= [x, y^{<t}]$ and $a_t= y^t$ and get the Equation. This is a common practice in many works [5].\n\nTo obtain a token-level reward $\\hat{r}$ in practice, we can use off-the-shelf open-sourced models trained by DPO, other methods [1, 2], or run DPO by ourselves.\n\nThanks. We will illustrate more for this equation in the final version.\n\n**Q4 Clarification on win rate and score**\n\nThe win rate is evaluated by calling a judge model (e.g., gpt-4o-2024-11-20) to pairwise compare the responses of the model and a baseline model (e.g., gpt-4-0314) and determine which one is better [6]. The MT-Bench score is evaluated by calling a judge model to directly assign a score to the model's response [7]. Many prior works observed the scores of different methods being similar [6, 8]. This is likely due to the single-instance scoring protocol. \n\n**Q5 Comparison with TDPO**\n\nBelow is the experiment result with TDPO under the Llama3-8B-Instruct PairRM setting.\n\n|  | Arena-Hard win rate |  AlpacaEval 2 win rate | MT-Bench score |  MT-Bench win rate | \n| - | - | - | - | - |\n| SFT | 21.4 |30.6 |7.9 |27.5 |\n| DPO |30.4  | 41.7| **8.0**| 37.5|\n| TDPO | 30.2 | 40.7|**8.0** | 39.0|\n| SimPO |28.7  |39.8 |7.8 | 32.5|\n| TGDPO |  **34.3**| **43.9**| **8.0**|**41.9** |\n\nResult with TDPO using the SFT model OpenRLHF/Llama-3-8b-sft-mixture is in **Q1 of Reviewer GJSH**. The result table with TDPO under other settings is in this link <https://anonymous.4open.science/r/tgdpo_rebuttal/results_with_tdpo.pdf>\n\n**Q6 Will the choice of $f$ affect the results**\n\n$f(u)$ may be adjusted by parameter $\\alpha$ as in Equ. (17). With this, experiment results in Fig. 1 reveal in some degree the effects of different choices of $f(u)$. The outcome benchmark difference presented in Fig. 1 and Tab. 3 is subtle.\n\nOur paper proposes a framework for incorporating existing token-level reward into DPO explicitly, with Equ. (17) as an example. Many choices of $f(u)$ can satisfy Assumption 4.2, which one is the best is left as an open problem for stimulating further research.\n\n\n**Q7 Clarification on convergence and checkpoint selection**\n\nIn Sec. 5.3, we consider loss moving average below a certain threshold (e.g., 0.1) as convergence. In other sections, we train all methods (DPO, SimPO, TGDPO) for 1 epoch and select the checkpoint at the training end.\n\n**Q8 Stability of results**\n\nBelow are the Arena-Hard win rate of Llama3-8B-Instruct PairRM and the 95% confidence interval. From the table we can see all methods have similar levels of stability. \n|  | Arena-Hard win rate |  95% Confidence interval | \n| - | - | - |\n| DPO | 30.4 | (-2.3, 2.2) |\n| SimPO | 28.7 | (-2.0, 2.0) |\n| TDPO | 30.2 | (-2.1, 2.4)|\n| TGDPO | 34.3 |(-2.3, 2.2) |\n\n**Q9 Would error in token-level reward affect results**\n\nBy Assumption 4.2, $|1-f(\\hat{r}(s_t, a_t))|\\le\\varepsilon$ where $\\varepsilon$ is small. Moreover, by the choice of $f(\\hat{r}(s_t, a_t))$ in Equ. (17), the error can be reduced by the parameter $\\alpha$. Thus mild errors in token-level reward may not affect the outcome greatly, as demonstrated in Fig. 1 and Tab. 3. \n\n**Q10 Why does $\\alpha$ affect the convergence rate**\n\n Take win response $y_w$ for an example, the other case is similar. In Equ. (16), for $\\hat{r}([x, y_w^{<t}], y_w^t)>0$ and larger $\\alpha$, the gradient w.r.t. this token is larger, and the convergence would be faster.\n\n**Q11 Code release**\n\nThe code will be released upon acceptance.\n\n[1] Preference-grounded token-level guidance for language model fine-tuning\n\n[2] Segmenting text and learning their rewards for improved RLHF in language model\n\n[3] From r to Q*: Your language model is secretly a Q-function\n\n[4] Direct preference optimization: Your language model is secretly a reward model\n\n[5] Free process rewards without process labels\n\n[6] From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline\n\n[7] Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena\n\n[8] SimPO: Simple Preference Optimization with a Reference-Free Reward", "claims_and_evidence": "Please see the Experiment and Question section.", "methods_and_evaluation_criteria": "The method is the direct result of the theoretical analysis; please see questions in the Theoretical Claims section.", "theoretical_claims": "I have carefully reviewed all the theoretical deductions and proofs, and I have some questions:\n\n1. The modified token-level DPO depends on the choice of $\\hat{r}$, which does not seem to be a well-defined problem.\n\n2. In the proof of the Bradley-Terry Model with Token-Level Reward Guidance in Equation 15, the assumption $T_w \\approx T_l$ and $Z([x, y_w^{<t}) ~= Z([x, y_l^{<t})$ made by the authors in lines 691-692 is very strong. \nHowever, the data $y_w$ and $y_l$ in the offline dataset used to train the DPO may not be generated from the same model. In practice, we might only have the positive data $y_w$, and we generate $t_l$ through negative sampling, or the pairs \n$\\{y_w, y_l\\}$ could be generated using two different models.\n\nHence, the assumption that $T_w \\approx T_l$ and $Z([x, y_w^{<t}) ~= Z([x, y_l^{<t})$ may not hold in this case.\nThis could significantly affect the subsequent deductions of the method.\n\nAdditionally, the assumption should be mentioned in the main text.\n\n3. In Line 311-313 (left), how can we get this equation? and in practice, should we always need to run DPO first to get the $\\hat{r}$.", "experimental_designs_or_analyses": "1. In Tables 1-4, how is the win rate calculated? Why do the scores between the methods appear similar, yet the win rates vary?\n2. In Table 2, the number of baselines is relatively small. It would be beneficial to compare against additional baselines, especially TDPO, which is the most closely related work to this paper.\n3. Will the choice of $f()$ affect the results? The experiment lack the ablation study of the choice of $f$.\n4. In Section 5.3, how is convergence defined? Additionally, how do the authors select checkpoints (no convergence results) in other sections\n5. In Tables 2-4, why is the score not reported\n6. Are the results stable? It would be better to report the standard deviation.", "relation_to_broader_scientific_literature": "The method might benefit other important aspects of LLM alignment, such as safety, honesty, and fairness.", "essential_references_not_discussed": "The literature review on RLHF could benefit from incorporating additional studies on DPO variants.", "other_strengths_and_weaknesses": "Strengths:\n1. This paper is well-written, and the method is well-motivated by rigorous theoretical analysis. The idea is both novel and interesting.\n2. The experiments demonstrate the effectiveness of the method, highlighting additional advantages regarding convergence properties. The authors also provide valuable insights from the experiments.\n\nWeaknesses:\nOverall, after carefully reading the paper, I believe this paper meets the acceptance criteria. However, I have several questions regarding the theoretical and experimental sections, which prevent me from confidently voting for acceptance. Please refer to the other sections for detailed questions and concerns.", "other_comments_or_suggestions": "No.", "questions_for_authors": "1. In lines 221-223 (left), why is \"the token-level reward only used as guidance, and we do not require it to be very accurate\"? Wouldn't an inaccurate token-level reward affect the outcome?\n\n2. Why does $\\alpha$ affect the convergence rate? Are there any theoretical insights into this?\n\n3. Do the authors have any plans to release the code?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R4gZLwbhwJ", "reviewer": "Reviewer_F88J", "summary": "This paper proposes a method that integrates Direct Preference Optimization (DPO) with token-level rewards. The paper first provide an upper bound for rewards in sentence-level LLM generation by decomposing the problem into a series of token-level reward maximization tasks. Building on this foundation, the paper adapts the DPO method to solve these token-level reward-maximization problem. Experiments were conducted on three models and evaluated across three benchmarks. The results demonstrate that the proposed method outperforms baseline approaches.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1 Clarification regarding Equ. 8**\n\nBy [1] (Thanks to  Reviewer  **aioX**), Equ. (8) has connections to an approximation approach common in prior RL works [2, 3, 4]. [1] pointed out this by providing 5 papers including [2, 3, 4], and followed this line to derive the loss function for their DPO, please see Appendix B.2.1 in [4].  Hence, this is a reasonable approach. We will give citations and provide related illustrations.\n\nMoreover, it was pointed out in [5] that \"[6] showed that under the Max-Entropy RL formulation, the token-level log-ratio $\\log \\frac{\\pi_{\\theta}(y|x)}{\\pi_{\\text{ref}}(y|x)}$ can be seen as an implicit token-level reward or advantage function (invariant under reward shaping).\"\n\nNotably, our derivation is mainly from the relaxation approach in optimization. Although we start from the state-action reward, the final derived loss function coincides with the streamline of DPO.  Specially, when the $\\hat{r}(s_t, a_t)$ is not available, we may simply set $\\hat{r}(s_t, a_t)=0$ and by Assumption 4.2 our loss function in Equ. (16) is precisely the loss function of DPO, which demonstrates our approach is reasonable. \n\n**Q2 Explanation for $f$ in Assumption 4.2**\n\nThanks. $f(u)$ is set such that  $f(0)=1$ and  $|f(u) - 1| \\le \\varepsilon$. Concrete $f(u)$ can be chosen by users for their personal usage. Please also check our response to **Q8 of Reviewer aioX** to see benefits of introducing $f$.\n\n**Q3 Strong assumption for Equ. 15**\n\nThank you for the insightful comment! \nWe have resolved this issue based on the finding that $\\delta( f, \\hat{r}; x, y_w, y_l)$ does not depend on the policy $\\pi_{\\theta}$ to be optimized, and the assumptions are removed. The approach is new for canceling partition functions in the BT model. \n\nNow Equ. (15) and its proof have been reorganized as follows:\n$$\n \\Pr(y_w \\succ  y_l | x)  = \\sigma\\left( \\varphi(\\pi_{\\theta}, f, \\hat{r}; x, y_w, y_l) + \\delta( f, \\hat{r}; x, y_w, y_l) \\right),  \\qquad (1)\n$$\nwhere  $\\delta( f, \\hat{r}; x, y_w, y_l)$ does not depend on the policy $\\pi_{\\theta}$ to be optimized, but only on $f, \\hat{r}, x, y_w, y_l$ and the partition function $Z(s_t)$ (does not depend on $\\pi_{\\theta}$, see Theorem 4.3). \nFor briefness, let\n$$h\\triangleq (f, \\hat{r}; x, y_w, y_l) .$$\n\nSince $\\sigma(t)$ is the sigmoid function with $\\sigma'(t)>0$, we have:\n\n**Theorem 1.** The $\\Pr(y_w \\succ  y_l | x)$ in Equ. (1) has the same maximal solution and the same ascent direction as the function $\\sigma\\left( \\varphi(\\pi_{\\theta}, h)\\right)$ with respect to $\\pi_{\\theta}$.  \n\n*Proof.* Note that,  $\\delta( h)$ is not dependent on the policy $\\pi_{\\theta}$ and $\\sigma'(t)>0$. $d$ is an ascent direction of function (1) if \n$$ d^T \\nabla_{\\pi_{\\theta}} \\sigma\\left( \\varphi(\\pi_{\\theta}, h) + \\delta(h) \\right) >0,$$\nwhich is equivalent to\n$$\n\\begin{aligned}\n&  d^T \\sigma'\\left( \\varphi(\\pi_{\\theta}, h)+ \\delta(h)\\right) \\nabla_{\\pi_{\\theta}}  \\varphi(\\pi_{\\theta}, h) >0 \\\\\n& \\Longleftrightarrow  d^T \\sigma'\\left( \\varphi(\\pi_{\\theta}, h)\\right) \\nabla_{\\pi_{\\theta}}  \\varphi(\\pi_{\\theta}, h) >0 \\\\\n& \\Longleftrightarrow d^T \\nabla_{\\pi_{\\theta}}  \\sigma\\left( \\varphi(\\pi_{\\theta}, h)\\right) >0.\n\\end{aligned}\n$$\n\nHence function (1) has the same ascent direction as the function  $\\sigma\\left( \\varphi(\\pi_{\\theta}, h)\\right)$.\nSimilarly, \n$$\n\\begin{aligned}\n& \\nabla_{\\pi_{\\theta}} \\sigma\\left( \\varphi(\\pi_{\\theta}, h) + \\delta( h) \\right) =0  \\\\\n& \\Longleftrightarrow  \\sigma'\\left( \\varphi(\\pi_{\\theta}, h)\\right) \\nabla_{\\pi_{\\theta}}  \\varphi(\\pi_{\\theta}, h) = 0 \\\\\n& \\Longleftrightarrow \\nabla_{\\pi_{\\theta}}  \\sigma\\left( \\varphi(\\pi_{\\theta}, h)\\right)\n= 0.\n\\end{aligned}\n$$\nSo Theorem holds.\n\nThus by Thm 1, since we focus only on optimal $\\pi_{\\theta}$ of function (1), we may redefine \n $$\\Pr(y_w \\succ  y_l | x) \\triangleq \\sigma\\left( \\varphi(\\pi_{\\theta}, f, \\hat{r}; x, y_w, y_l)\\right), $$\nand use it for constructing the loss function in Equ. (16). \n\nWe will update the statement and proof of Equ. (15). Thanks! \n\n**Q4 Why $r(s_t,a_t) = \\beta \\log \\pi_{\\theta} / \\pi_{\\text{ref}}$**\n\nIt was shown in [6] that $r(s_t,a_t) = \\beta \\log \\pi_{\\theta} / \\pi_{\\text{ref}}$ under the definition of equivalent state-action reward class and invariant re-parameterization, which does not require  $V(s_{t}) = V(s_{t+1})$. Please see Theorem 1 there. Moreover, the final loss function in [6] is equivalent to that in [7]. Hence we adopt the equation directly. It is a common practice in many works [5, 8].\n\n[1] A dense reward view on aligning text-to-image diffusion with preference\n\n[2] Approximately optimal approximate reinforcement learning\n\n[3] Relative entropy policy search\n\n[4] Trust Region Policy Optimization\n\n[5] Self-Play Preference Optimization for Language Model Alignment\n\n[6] From r to Q*: Your language model is secretly a Q-function\n\n[7] Direct preference optimization: Your language model is secretly a reward model\n\n[8] Free process rewards without process labels", "claims_and_evidence": "In this paper it is claimed that the proposed TGDPO performs better than baselines like DPO and SimPO. This is adequately supported by experiments in Section 5 of this paper.", "methods_and_evaluation_criteria": "The paper mainly focuses on RLHF problem. The evaluation benchmark involved, namely AlpacaEval, MT-Bench and Arena-Hard, are all widely used benchmarks for alignment evaluation.", "theoretical_claims": "The concerns regarding methodology are listed below\n\n1.While Theorem 4.1 appears to be correct, its intended message is unclear. The objective in equation (8) can certainly serve as an upper bound for equation (2), but there is no guarantee that a policy maximizing (8) will also maximize (2). To be more specific, it looks like the objective in equation (8) is to myopically optimize the reward of the currect step without considering further states. However, in RL, the goal is to maximize the expectation of **total** future rewards (i.e., Q-function).  policy that optimizes one-step rewards might lead the agent into unfavorable states, ultimately hindering its ability to achieve high cumulative rewards.\n\n2. The introduction of function $f$ in Assumption 4.2 appears insufficiently motivated. Could the authors provide further explanation and justification for its inclusion?\n\n3. In equation 15, the $\\approx$ is obtained by assuming $Z(x, y_w^{<t}) \\approx Z(x, y_l^{<t})$.However, this assumption seems questionable. Even if $y_w$ and $y_l$ are both generated on-policy, since there is randomness during sample and the generated response is long, even some slight variation at the beginning might leads to significantly different prefix $y^{<t}$. Therefore it is not appropriate to make such assumption.\n\n4. The equation right above equation (17) gives a way to distribute sentence level reward to token-level reward. However, such distribution is not well-motivated. In fact, [1] gives the exact reward distribution as $r(s_t,a_t) = \\beta \\log \\pi_{\\theta} / \\pi_{\\text{ref}} + V(s_{t}) - V(s_{t+1})$. Therefore, the reward distribution is valid only when $V(s_{t}) = V(s_{t+1})$, which is not established in the paper.\n\n[1] Rafailov, Rafael, et al. \"From $ r $ to $ q^* $: Your language model is secretly a q-function.\" arXiv preprint arXiv:2404.12358 (2024).", "experimental_designs_or_analyses": "Extensive experiments are conducted, and the results looks convincing to me.", "relation_to_broader_scientific_literature": "This paper adapts DPO to token-level reward and the relation with previous works are mostly clearly stated", "essential_references_not_discussed": "I don't see a missing of significant related works.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "See above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4e89omfUrz", "reviewer": "Reviewer_aioX", "summary": "Recent work in RLHF has revealed the benefits of utilizing fine-grained rewards. The combination of token-level guidance with DPO, however, remains to be explored.\nTo address this challenge, this paper decomposes the sequence-level RL formulation in the original DPO derivation as a sequence of token-level RL problem, from which closed-form solution for the optimal token-level policy and reward can be derived.\nFrom this, this paper derives a loss with token-level reward guidance for DPO and proposes a practical reward guidance based on the induced DPO reward.\nThis formulation enables different tokens to exhibit varying degrees of deviation from reference policy.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1 Comparison with existing works**\n\nAll [1], [2] and ours are based on the assumption of dense token-level reward to derive respective DPO loss functions. Eq. (8) in our work seems similar to Eq. (12) in [1], but our Eq. (8) is not solvable since $s_t\\sim\\mathcal{D}\\_t$ is dependent on the policy $\\pi\\_{\\theta}$ to be optimized. This has been demonstrated in line 219 (left) - line 168 (right) of the main text. In the final version we will give citations, discuss connections with [1, 2] and the differences highlighted as follows:\n\n**Motivation Differences:**  We intend to incorporate a learned token-level reward explicitly into DPO. While, they only focus on asigning more weight on earlier tokens.\n  \n**Loss Function Differences:** Our work explicitly incorporate an existing token-level reward in the loss function for guiding DPO. [1] uses the same loss as DPO and modifies the sampling probability for different diffusion timesteps. [2] adds a temporal decay factor for each token in the DPO loss. Their loss functions do not leverage token-level reward.\n\n**Method Differences:** Integrating token-level reward in DPO's loss function explicitly presents distinctive difficulties, especially in the derivation. Thanks to all reviewers' comments, we have proposed a new approach for canceling partition functions in the Bradley-Terry model, please see **Q7**.\n\n[1] A Dense Reward View on Aligning Text-to-Image Diffusion with Preference\n\n[2] Earlier Tokens Contribute More: Learning Direct Preference Optimization From Temporal Decay Perspective\n\n**Q2 Accurate word choice**\n\nWe will change to use \"response-level reward\" and \"KL-regularized policy optimization\" to improve accuracy.\n\n**Q3 Explain \"sentence-level PPO with token-level reward guidance\"**\n\nA sequence of dense token-level reward $r_{\\phi}(s_t, a_t)$ is obtained before PPO, and then it can guide PPO finetuning in a more fine-grained way. \n\n**Q4 What is the corresponding dense token-level reward**\n\nActually, $\\pi_{\\hat\\theta}$ does not necessarily relate to $\\hat{r}$ in Assumption 4.2. $\\pi_{\\hat\\theta}$ is only used for sampling $s_t$ in Equation (10). Now, the first sentence of Assumption 4.2 is modified as: Suppose we have learned a dense token-level reward $\\hat{r}$ using some effective learning approach. In the revision, we will improve the description and notation to reduce misunderstanding.\n\n**Q5 Difference between $\\hat{r}$ and $r_{\\phi}$**\n\n$\\hat{r}$ is an existing dense token-level reward, used for shaping the reward $r_{\\phi}$ in Equ. (10). Following the derivation of DPO, $r_{\\phi}$ is expressed with $\\hat r$, $\\pi_{\\theta}$ and $\\pi_{\\text{ref}}$ by solving Equ. (10). And finally, $\\hat{r}$ will appear in Equ. (16) of our loss function, but $r_{\\phi}$ does not. \n\nIf we optimize $\\pi_\\theta$ against $\\hat r$, then Equ. (10) becomes\n$$\n\\max_{\\pi_{{\\bf\\theta}}} \\mathbb{E}\\_{s_t\\sim\\hat{\\mathcal{D}}\\_t, a_t\\sim \\pi_{\\theta} (\\cdot|s_t) } \\left[{r_{\\phi}(s_t, a_t)} - {\\beta f(\\hat{r}(s_t, a_t))}\\log \\frac{ \\pi_{\\theta} (a_t|s_t)}{\\pi_{\\text{ref}}(a_t|s_t)}\\right]. \n$$\nWe tried and found it is difficult to solve the problem due to the expectation on the product term.\n\n**Q6 Requirement for $f$**\n\nIn Assumption 4.2, the $f$ satisfying $f(0)=1$ and $|f(u)-1|\\le\\varepsilon$ is sufficient for deriving our TGDPO. Concrete $f(u)$ can be chosen by developers personally. As for us, Equ. (17) is adopted in our experiments. If $f(u)\\equiv 1$ then our TGDPO degenerates to DPO, which is not interesting. \n\n **Q7 Strong assumption for Equation 15**\n\nThank you very much for the comment! Now the assumptions have been completely removed, the approach is novel. The theorem and proof can be seen in response to **Q3 of Reviewer F88J**. We will provide revisions in the final version.\n\n**Q8 Benefit of introducing $f(\\hat r)$ in Equ. (10)**\n\nThe benefit of introducing $f(\\hat r)$ in Equ. (10) is in Equ. (16) of the loss function.\nWe adopt Equ. (17) in our experiments. Then take win response $y_w$ for an example, the other is similar. In Equ. (16),\n$$\nf(\\hat{r}([x, y_w^{<t}], y_w^t))   \\log\\frac {\\pi_{\\theta} (y_w^t|[x, y_w^{<t}])}{\\pi_{\\text{ref}}(y_w^t|[x, y_w^{<t}])} = (1 + \\alpha \\hat{r}([x, y_w^{<t}], y_w^t)) \\log\\frac {\\pi_{\\theta} (y_w^t|[x, y_w^{<t}])}{\\pi_{\\text{ref}}(y_w^t|[x, y_w^{<t}])}.\n$$\nFor $\\hat{r}([x, y_w^{<t}], y_w^t)>0$, we emphasize more on this token, otherwise less or keep the weight unchanged as 1.\n\n**Q9 Improper description in L286, L406-408**\n\nThank you very much for pointing out the issues. We modify them as: \n1) L283-286 (right): \"Then this token is optimized with less strength  during the optimization of the loss function  $\\mathcal{L}\\_{\\text{TGDPO}}(\\pi_{\\theta})$, since $f(\\hat{r}([x, y_w^{<t}], y_w^t)) <1$.\"    \n2) L296-299 (right): Similar to point 1.\n3. L406-408 (left): \"distinguish preferred tokens in chosen samples and dispreferred tokens in rejected\nones, TGDPO takes care of them and enables ...\"", "claims_and_evidence": "Yes, the empirical performance validates the proposed method.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes, tested on standard RLHF benchmarks", "relation_to_broader_scientific_literature": "See the Strength section.", "essential_references_not_discussed": "See the Weaknesses section", "other_strengths_and_weaknesses": "## Strengths\n1. Dense/token-level reward is a promising direction for RLHF to improve upon the classical bandit/sequence-level formulation. The combination of this direction with DPO is particularly less cultivated.\n\n2. The proposed method shows competitive performance gain.\n\n\n\n## Weaknesses\n\n1. The idea of \"token-level reward guidance + DPO\"  has been explored, e.g., in [1, 2]. Specially, Equ. (8) in this paper is almost identical to Equ. (12) in [1]. The authors need to properly discuss the connection and novelty compared to these prior works. \n\n2. Some of the basic terms/concepts are confused with others, see the next \"Suggestions\" section.\n\n3. Please check the Questions section for a potential mistakes on the Method section and several unclear parts.\n\n***\n[1] A Dense Reward View on Aligning Text-to-Image Diffusion with Preference. In ICML 2024.\n\n[2] Earlier Tokens Contribute More: Learning Direct Preference Optimization From Temporal Decay Perspective. In ICLR 2025.", "other_comments_or_suggestions": "1. \"sentence-level reward\" is not accurate, what the authors refer to should be termed as bandit/{sequence, trajectory, response}-level reward, since each response may contain multiple sentences, for example, in the summarization task. Ditto all presence of \"sentence\".\n\n2. Most of the appearance of \"proximal policy optimization problem\"/\"PPO problem\" should be changed to \"KL-regularized policy optimization\"/\"KL-regularized RL\"/\"KL-regularized control\".", "questions_for_authors": "1. Could you explain what is \"sentence-level proximal policy optimization with token-level reward guidance\"? In particular, why is PPO sentence-level given that the reward is token-level?\n\n2. In Assumption 4.2: \n  - What is \"the **corresponding** dense token-level reward\"? Why does the reward need to be learnt to correspond to the policy? \n  - What is the different between $\\hat r$ and $r_\\phi$? Why couldn't we optimize $\\pi_\\theta$ against $\\hat r$?\n  - What is the requirement of the function $f$? The current description looks like a constant function of 1 is also valid, which will trivialize Equ. (10). For the proof of Equ. (15), I believe we need the stronger assumption of $\\epsilon \\times M \\times T_w \\approx 0$.\n\n3. What is the benefit of introducing $f(\\hat r)$ in Equ. (10)? See the following question for a problem with the current specification of $f$. \n\n4. L286: \"likely to make this token as a dispreferred one\" --- this is incorrect. The current loss will only make the optimization strength of those token relatively smaller, rather than minimizing their likelihood, since Assumption 4.2 says $|f(u)| \\geq 1 - \\epsilon > 0$ for small $\\epsilon$. Ditto L299. With this, the author may need to revise the claim in L406-408 about \"reduces conflicts\".", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces TGDPO, which incorporates token-level reward into DPO for preference alignment of LLMs. The main idea is to decompose sentence-level DPO problem into a sequence of token-level problems, enabling the use of more fine-grained reward signals.

Two reviewers recommended acceptance, noting the paper's promising idea and strong empirical results. Two other reviewers had concerns mainly about certain theoretical claims (strong assumption for Equ. 15) and lack of related work discussion (using token reward in DPO is not totally novel).

During rebuttal, the authors added experimental results using an SFT model to alleviate concerns about using post-trained models. They also provided a comparison of different judge models. The mentioned related work is promised to be discussed in the subsequent versions. The only remaining issue seems to be that the author provided a new proof which removes the previous strong assumption for Eq. (15) while this is not fully checked.

Given that one reviewer was ultimately satisfied and the authors provided additional experimental results using an SFT model and a comparison of different judge models in their rebuttal, I recommend an accept.  While concerns about the generalizability of the theoretical framework remain, the empirical results suggest that TGDPO has the potential to improve alignment of large language models.
Decision: Accept (poster)