DPO Meets PPO: Reinforced Token Optimization for RLHF
Authors: Han Zhong
Zikang Shan
Guhao Feng
Wei Xiong
Xinle Cheng
Li Zhao
Di He
Jiang Bian
Liwei Wang
Venue: ICML 2025 spotlightposter (2025)
Keywords: RLHF
PPO
sample efficiecncy
Primary area: deep_learning->large_language_models
TL;DR: We formulate RLHF as token-wise MDPs instead of sentence-level bandits, and propose a provable and practical algorithm, RTO, under this framework.

Abstract: In the classical Reinforcement Learning from Human Feedback (RLHF) framework, Proximal Policy Optimization (PPO) is employed to learn from sparse, sentence-level rewards---a challenging scenario in traditional deep reinforcement learning. Despite the great successes of PPO in the alignment of state-of-the-art closed-source large language models (LLMs), its open-source implementation is still largely sub-optimal, as widely reported by numerous research studies. To address these issues, we introduce a framework that models RLHF problems as a Markov decision process (MDP), enabling the capture of fine-grained token-wise information. Furthermore, we provide theoretical insights that demonstrate the superiority of our MDP framework over the previous sentence-level bandit formulation. Under this framework, we introduce an algorithm, dubbed as Reinforced Token Optimization (\texttt{RTO}), which learns the token-wise reward function from preference data and performs policy optimization based on this learned token-wise reward signal. Theoretically, \texttt{RTO} is proven to have the capability of finding the near-optimal policy sample-efficiently. For its practical implementation, \texttt{RTO} innovatively integrates Direct Preference Optimization (DPO) and PPO. DPO, originally derived from sparse sentence rewards, surprisingly provides us with a token-wise characterization of response quality, which is seamlessly incorporated into our subsequent PPO training stage. We conduct extensive experiments to evaluate \texttt{RTO} against PPO and other direct preference learning algorithms. The results highlight the effectiveness of RTO, with the algorithm outperforming PPO by 7.5 points on the AlpacaEval 2 benchmark and by 4.1 points on Arena-Hard.  Our code and models are available at \href{https://github.com/zkshan2002/RTO}{https://github.com/zkshan2002/RTO}.

DPO Meets PPO: Reinforced Token Optimization for RLHF
Han Zhong" 1
Zikang Shan
Guhao Feng"
Wei Xiong Liwei Wang! 12
Xinle Cheng'
Li Zhao
Di He?
Jiang Bian"
Abstract
1. Introduction
In the classical Reinforcement Learning from Human Feedback (RLHF) framework, Proximal Policy Optimization (PPO) is employed to learn from sparse, sentence-level rewards-a challeng- ing scenario in traditional deep reinforcement learning. Despite the great successes of PPO in the alignment of large language models, its open- source implementation still largely sub-optimal. To address these issues, introduce frame- work that models RLHF problems as Markov decision process (MDP), enabling the capture of fine-grained wisen-wisc information. Under this framework, we introduce an algorithm Reinforced Token Optimization (RTO), which learns the token-wise reward function from preference data and performs policy optimization based on this learned token- wise reward signal. Theoretically, RTO is proven to have the capability of finding the near-optimal policy sample-effcienntyy For its practical implementation, RTO innovatively in- tegrates Direct Preference Optimization (DPO) and PPO. DPO, originally derived from sparse sentence rewards, surprisingly provides with token-wise characterization of response quality, which is seamlessly incorporated into our subse- quent PPO training stage. Extensive experiments demonstrate that RTO performs better than PPO and other direct preference learning algorithms. In particular, RTO outperforms PPO by 7.5 points the AlpacaEval 2 benchmark and by 4.1 points irena-Hard Our code and models are avail- able at https//ggthbb.coo//kkaaa2//00//
Reinforcement Learning from Human Feedback (RLHF) has emerged key technique for aligning foundation mod- els with human values and preferences (Christiano et al., 2017: Ziegler et al., 2019). It has been pivotal in enabling Large Language Models (LLMs) to produce more help- ful, harmless, and honest responses (Bai et al., 2022), as demonstrated significant applications such as ChatGPT (OpenAI, 2023), Claude (Anthropic, 2023), and Gemini (Team et al., 2023). The classical RLHF pipeline (Ziegler et al., 2019; Ouyang al., 2022) consists of two steps: (i) Reward training from human feedback, where the learner learns the reward function based on preference data, typi- cally through Maximum Likelihood Estimation (MLE). (ii) Reward- based RL training, where the learner employs the seminal deep RI. algorithm Proximal Policy Optimization (PPO; Schulman et al., 2017) optimize the reward learned in the previous step.
Despite the success of this framework in the aforementioned powerful closed-source LLMs, the training of PPO is known to be unstable and sample-ineffiiient (Choshen et al., 2019).
While researchers have made efforts to propose alternative approaches to the PPO algorithm, with notable examples like rejection sampling fine-tuning (Dong al., 2023; Gul- cehre et al., 2023), direct preference learning algorithms (Rafailov et al., 2023; Zhao et al., 2023; Azar et al., 2023), there is little evidence that these newly proposed approaches alone can make the state-of-the-as LLMss. Therefore, im- proving the performance the PPO algorithm in the con- text of RLHF is still an important research direction that largely under-explored.
After examining the open-source implementation of PPO, we identify that one potential reason for the sub-optimal per- formance of PPO is the mismatch between the formulation of RLHF and the nature of PPO. Specifically, in the existing framework (Ouyang et al., 2022; Bai et al., 2022), RLHF formulated as a bandit, where the entire response sentence is considered be an action, and the reward sentence- level, evaluating only the overall quality of the response. However, PPO is designed for multi-step RL problems mod- eled as Markov decision processes (MDPs), requiring a token-wise reward assignment to each step. In typical im-
Eqqul contribution 'Center for Data Science, Peking University "State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technol- ogy, Peking University Unniversity of Illinois Urbana- Champaign "Microsoft Research Asia. Correspondence to: Han Zhong <hanzhong stu. ptw.euu..duu.cn... Li Zhao <lizo@ microsoft.comm com> Di He < pku.edu.cn>, Liwei Wang <wanglw @cis, www.gltwwcss.pu..dd..m pku.edu.cn..
Proceedings of the nnnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

DPO Meets PPO: Reinforced Token Optimization for RLHF
RLHF as Bandit: Proximal Policy Optimization
RLHF as MDP: Reinforced Token Optimization
SantenseWWise Reward
PPO
LM Pollyy
Token-Wos
Roward
PPO
LM Policy
Figure 1. In the MDP framework of RLHF, RTO uses DPO to derive token-leve -lvel reward function and then applies PPO enhance This approach is significantly different from the traditional RLHF process, which employs PPO to improve sentence-level rewards under the bandit framework of RLHF.
plementations of PPO (e.g.. OpenRLHF and TRL), besides the regularization reward function assigned to each token to ensure the fine-tuned L.M stays close to the supervised fine- tuning (SFT) model, the learned sentence-level reward is only distributed to the last token, while other tokens receive zero learned reward. See (2.3) for the formal mathemati- cal description. Clearly, there is separation in terms of the assignment strategies of the regularization reward and the learned reward. Meanwhile, while it is generally be- lieved that fine-gres unn-ggrined characterization with token-wise feedback can provide more information, in practice, it is also challenging to collect effective token-wise feedback for human conversations and use it in the MLE process. Conse- quently, the construction of (oken-wise reward signals also remains largely under-explored in the literature of RLHF.
PPO and direct preference learning baselines such as DPO (Rafailov et al., 2023), R-DPO (Park et al., 2024), and SimPO (Meng et al., 2024). In particular, RTO achieves a 7.5-point improvement on the AlpacaEval 2 benchmark and a 4. -point improvement on Arena- Addition- ally, RTO exhibits strong data scaling properties compared to PPO - (i) reaching PPO-level performance with only 1/8 of the data and (ii) continuing to improve more data is added, whereas PPO saturates early.
In summary, under the MDP formulation of RLHF, we de- velop new principled RLHF algorith,, RTO, that leverages token-wise reward signals derived from offline preference data using DPO, and subsequently performs PPO training to optimize the token-wise rewards. The pipeline of RTO is visualized in Figure
Our Contributions. In this work, we aim to address the aforementioned issues by developing an RLHF framework with a fine-ge mne-grained token-wing wise reward characterization, es- tablishing the mathematical foundation, and advancing prac- tical algorithmic designs. The key contributions of this work summarized as follows.
1.1. Related Works
We review the works that are mostly related to our project in this subsection. Due to the space constraint, we refer interested readers to the survey (Casper et al., 2023) for a more comprehensive overview of RLHF,
We propose framework that models RLHF as a (KL- regularized) MDP, offering a more precise token-wise characterization of the LLM's generation process. Fur- thermore, we provide theoretical insights into why the tosen-wise MDP formulation is superior to the previous sentence-level bandit formulation of RLHF
RLHF algorithm. As we mentioned in the introduction, tuning the PPO algorithm to its best performance requires extensive efforts and resources are often unavailable to the open-source community. Motivated by this, researchers have made efforts to develop alternative approaches to the PPO algorithm. As a direct extension of the best-of-n infer- ence (Nakano et al., 2021), rejection sampling fine-tuning is proposed by Dong et al. (2023); Gulcehre et al. (2023); Wang et al. (2024), which prompts the LLM to generate n responses per prompt and uses learned reward function to rank the responses and fine-tune the model on those with high rewards. Besides, inspired by the reward-conditioned training in RL literature (Chen et al., 2021), Hu et al. (2023); Yang et al. (2024a) develop conditional SFT to avoid the reward learning. Another line of work aims to skip the re- ward modeling step and may be referred to the direct preference learning approach (Zhao et al., 2023; Rafailov et al., 2023; Azar et al., 2023; Tang et al., 2024). Among them, the direct preference optimization (DPO) algorithm is the most popular one, mostly due to its innovative idea: your
Under the MDP formulation of RLHF, we introduce Rein- forced Token Optimization (RTO), which extracts token- wise reward signals from offline preference data and sub- sequently performs RL training with respect to the learned token-wise rewards. Using MLE as the token-wise re- ward learning oracle, we prove that RTO can learn near- optimal policy in sample-efficien manner.
Moving toward the practical implementation of RTO, we adopt novel token-wise reward extraction approach from direct preference optimization (DPO; Rafailov et al., 2023). By assigning this DPO- token-wise reward function to each token and then optimizing with PPO. RTO demonstrates superior performance compared to both
2

---

DPO Meets PPO: Reinforced Token Optimization for RLHF
language model is secretly reward model. In particular, according the reward benchmark (Lambert al., 2024), the DPO-aligned algorithm often admits a competing rank- ing accuracy as reward function. We will formally discuss the principle of DPO in Appendix C.I, which also partly motivates our methods. After these, there are also many tasks that consider the variants of this direct preference learning approach by increasing the training steps (Xiong et al., 2023; Hoang Tran, 2024) and consider the more gen- eral preference signal sources (Ye et al., 2024; Rosset et al., 2024). Although all these recently proposed algorithms achieve promising results, there is little evidence that these algorithms alone without PPO can make state-of- -o-tth--rr LLMs. Therefore, understanding PPO and improving its performance in the context of foundation model alignment still an important research direction.
the original reward function, the most widely used lear- ing target is Kullback-Leibler (KL.)-egularrize one. In recognition of the above issues, Xiong et al. (2023) first for- mally formulates the RLHF as the reverse-Ki KL constrained contextual bandit in offline, online, and hybrid settings, and proposes sample-efficient algorithms in different settings accordingly. Beyond the reward- wwaard-based framework under the Bradley-Terry model, Azar et al. (2023); Ye al. (2024) consider the RLHF under general preference oracle, and motivate the algorithmic design in a KL-regularized min- imax game between two L.Mss. In particular, Azar et al. (2023) proposes the first sample-efficient planning algo- rithm, and Ye et al. (2024) designs the sample-efficient learning algorithms in offline and online settings. Notably, as these studies of the KL-regularized framework align with the practical applications closely, the theoretical insights nar- urally motivate practically powerful algorithms like GSHF (Xiong et al., 2023), Nash-MD (Azar et al., 2023), and DNO (Rosset et al., 2024). However, we remark that Xiong et al. (2023); Azar et al. (2023); Ye et al. (2024) are still con- fined to the bandit setting, thus differing from the MDP formulation presented in this paper.
Improving PPO in the context of RLHE, Although some works (e.g.. Uesato et al., 2022; Lightman et al., 2023; Yang et al., 2024b) use token-wise or step-wise information to enhance the performance of LLMs, such as their reasoning ability, we will not discuss them in detail here. Instead, we will focus on comparing our work with others that aim to improve the PPO in RLHE, In particular, Li et al. (2023b) and Ahmadian et al. (2024) state that the PPO is not the best fit for RLHF because of the sentence-level reward and deterministic transition, and argue that the reinforce-style (Williams, 1992) algorithms perform better. Wu et al. (2024) proposes to construct several separate reward functions for different goals and use the linear combination of them to guide the PPO training, but the separate models are still confined to the sentence level. Similarly, Jang et al. (2023) extends the PPO to the multi-objective optimization sce- nario, but still uses the sentence-level modeling. Chan et al. (2024) shares similar insights that aim to improve PPO via a dense reward. They still follow the two-staged RLHF framework to model the reward function via MLE of the Bradley-Terry model and assume that the learned reward is based on the transformer (Vaswani et al., 2017). Then, they propose to use the attention value to redistribute the final scalar reward on a token level. In comparison, while sharing similar insights about using a token-wise reward, our techniques to obtain the dense signal and mathematical motivation are fundamentally different.
Concurrent and Subsequent work. We notice several con- current and independent works by Rafailov et al. (2023); Zeng et al. (2024); Meng et al. (2024). Rafailov et al. (2024) also provide token-wise MDP formulation for RLHE Their work shares the same insight as ours, namely that "DPO implicitly optimizes the token-wise reward". Based on this insight, they improve the efficiency of gearch-based algo- rithms. In contrast, we propose new algorithm RTO that leverages the token-wise reward functions to enhance the performance of PPO. In addition, our work provides a theo- retical foundation for the unique advantages of token-wise MDP and its sample-efficient learning. Meng et al. (2024) propose SimPO by modifying the DPO objective, replac- ing the reference model with response length, and adding a margin threshold. Zeng et al. (2024) also consider a token- level reward and leverage this insight to develop token-level DPO, which performs better than the original DPO. These are beyond the scope of our work. In contrast, we utilize the implicit leken-level reward provided by the original DPO as the dense token-level reward for RL training. We in- clude these two algorithms as baselines to demonstrate the superior performance of RTC. Finally, following our work, Cui et al. (2025); Yin et al. (2025) utilizes implicit rewards in RL training to enhance chat and reasoning capabilities, highlighting the broad applicability of our method.
Theoretical study of RLHE. The theoretical study of RLHF may date back to the dueling bandit and dueling RL (c.g., Yue et al., 2012; Saha, 2021; Faury et al., 2020; Bengs et al., 2021; Pacchiano et al., 2021; Chen et al., 2022; Zhu et al., 2023; Wang et al., 2023; Zhan et al., 2023a;b), where the reward maximization problem is considered in the face of preference signals, instead of the absolute reward signals. However, the reward maximization framework admits a greedy and deterministic optimal policy, which deviates from the principle of generative AL. Meanwhile, instead of
1.2. Notation
Given a set X, we denote the collection of distributions over X' by A(X'). We use { to denote the indicator function. For any positive integer ht we use the notation Fıah to denote the sequence {y1, ........h.
yh }.
3

---

DPO Meets PPO: Reinforced Token Optimization for RLHF
2. Preliminaries
To this end, given a prompt x and a response y = y1:H containing H tokens, existing open-source implementations of PPO assign the sentence-level reward TMLE (z, y) to the last token and optimize the following reward:
In this section, we introduce the standard RLHF paradigm. Let X' denote the prompt sampled from distribution P E A(X), and y = (y1, 92...... "............................... Mh..... be the corre- sponding response, which is sequence of tokens generated by LLMs, where yi represents the i-th token. In practice, it is widely assumed (Ziegler et al., 2019; Ouyang et al., 2022) that the preference signal is generated according to the Bradley-Terry (BT) model (Bradley & Terry, 1952):
T Tppo (I,, M1:h))
(2.3)
0 0- B log with Im-Wish 1) if S H -
Jxat (vN x.
TMLE y) 3 log (1) 1.,91 1)_ if h H,
Teer(ya [a.
where I is the current policy to be improved. However, it is well known that sparse rewards can make learning more difficult compared to dense rewards (Andrychowicz et al., 2017). One natural solution is to design dense (oken-wise rewards used for PPO training, but this is beyond the scope of the current bandit formulation for RLHF and motivates us to provide framework with more fine-grained token-wiss -wiee characterization that enables the use of token-wise rewards.
P(y* > y1 ," 8²) = (r(2, v²) rxx,)))). (2.1)
where (2) = (xx((x))(-=)) is the sigmoid function, and r is ground-truth reward function defined at the sentence level. In other words, the reward function r only evaluates the overall performance of the entire response. The classical RLHF pipeline (Ziegler et al., 2019; Ouyang et al., 2022) typically consists of two steps: reward training from human feedback and reward-based RL training. In the first step, the learner is given dataset D = {(z, y", ,/)}, where yd denotes the preferred response over the y². The reward function is learned through Maximal Likelihood Estimation (MLE) on this dataset D:
3. RLHF Formulation: From Bandit to MDP
In this section, we introduce our MDP formulation for RLHE. Section 3.1 describes how to characterize RLHF using token-wise MDPs in the context of LLMs. Section 3.2, we provide the learning objective under this frame- work. Lastly, Section 3.3 demonstrates the advantages of the token-wise MDP formulation compared to the sentence- wise bandit formulation.
TMLE argmax E (1.y"" -D log
V)) (2.2)
In the second step, the learned reward FMLE from the pre- vious step is optimized while ensuring that the updated language model (LLM) does not deviate significantly from the reference model Tref, usually selected as supervised fine-tuned (SFT) LLM. This is because reward optimiza- tion along usually leads to reward hacking (Casper et al., 2023), meaning that the LLM will utilize the imperfection of the reward model and chase for a high reward but with a poor performance at the same time. Formally, the L.LM is optimized with respect to the learned reward FMLE with KL-regularized term:
3.1. MDP Formulation for RLHF
We model the RLHF problem as a Markov decision process (MDP), which is denoted as a tuple M = (S, P, r. H). Here S the state space, A is the action space, P SxAA + (S) is the transition kemnll, denotes the reward function, P signifies the initial state distribution and H is the maximal number of interaction steps. A (Markov) policy in MDPs π : S - (AA) is a mapping from state to a distribution over actions. The interaction between the environment M and the agent can be described as follows. Initially, the starting state $1 is sampled from the initial distribution p. At the h-thh step, the agent observes the state 8i and selects action On based on its policy. The environment then transits to the next state 8h+1, which is sampled from the distribution P(- sh ax). This interaction continues until a certain ending condition is satisfied, which will be triggered within H steps.
(y x)
ã argmax E (x (MLE(E. v) - 8 log Tyot (y
[x]]
where 3 > 0 is an appropriate KL penalty coefficient. This KL-regularized target is widely adopted in practice (Chris- tiano et al., 2017: Ziegler et al., 2019; Ouyang et al., 2022: Bai et al., 2022; Rafailov et al., 2023) to balance reward optimization and the goal of staying close to the reference policy. Another primary technical reason is that this regu- larization ensures that the framework admits stochastic optimal policy, as compared to the deterministic greedy re- ward maximizer. The policy optimization step is typically achieved by PPO (Schulman et al., 2017), a seminal deep RL algorithm for solving multi-step dacision-making prob- lems and its implementation requires a reward signal at each step (corresponding to each token in the context of L.LMs).
In the standard text generation process of large language models (LLMs), each state 8n = (2x, 11-A-1) includes the prompt x and all response tokens produced up to that point. Each action as.- = yh. represents a token from the vocabulary. The transition kernel P is usually known and deterministic, meaning that given tokens 8n = (Ic, V1.h-1) and ax = yh- the environment will transition to $h+1 = (z. V1:1). The policy л maps all the observed tokens so far to a distri-
4

---

DPO Meets PPO: Reinforced Token Optimization for RLHF
bution over the vocabulary. It is important to note that the policy captures the autoregressive nature of LLMs, i.e., (y1sh | 2²) = _ (yk x. 11sh-1) for any h. Due to this, may refer to it as an autoregressive policy to differ- entiate it from policies defined in other ways. Moreover, r S x A + R represents the token-wise reward. The maximum number of tokens that can be generated. H, char- acterizes the length limit for LLM outputs. Each generated text ends with a special end-of-sentence token Eos, which terminates the generation process.
when certain condition is met. In particular, since we as- sume that the maximal length of the generated responses of LLMs is at most H, the summation in (3.2) is taken at most H steps. In the remaining part of this paper, we may use Lh=1 and Σ h=1 interchangeably, as they mostly have the same meaning. The regularized Q-function Q7 of a policy a is related to the regularized value function V3"" as
(7(s, a; r) = ra(ss, a) + Bgr (.0) [V (s(:r)), (3.3) (s:(s) = -E 1 Bog ग [aal 8) + Q3(s, a.),
In our MDP formulation for RLHF, we also model the pref- erence signal using BT model (Bradley& & Terry, 1952), but replace the sentence-level reward function in (2.1) with token-wise reward functions. In specific, for any trajectory pair -| = {($} Mi> and 7² = {(5}, a 1/) the preference is specified by
where we denote rs(ss a) r(s,a) + log Tee 8). Moreover, when it is clear from the context, we may omit the dependency of the ground-truth reward func- tion in Q7(s, a; r), V (s:r) and use the shorthand (s, a), (s). The regularized optimal policy vă is the policy that maximizes the regularized value function defined in (3.2), and its corresponding optimal Q-function and value function are denoted as Q3 and Vz. respectively. By (3.3), it can be shown that
PP > 7+) -
r ab) (Ershh
Σ
Σ rbh oit) ). (3.1)
g s) = ( a) (s))/
(3.4)
Compared to literature that formulates the RLHF problem as a contextual dueling bandit, a subtle difference is that the policy the contextual dueling bandit maps a prompt to a distribution over sentences, which does not capture the autoregressive nature of LLMs. In contrast, our MDP formulation precisely captures this nature. We defer the discussion of these two types of policies in Section C.2. More importantly, the main difference is that the reward function in the MDP formulation is defined on a token level, which contrasts significantly with the sentence-leve reward in the contextual dueling bandit. We discuss the advantages of teken-level rewards in Section 3.3.
Our learning objective is to find a near-optimal policy म, whose performance is measured by the following subopti- mality gap:
SubOpt(f) V3( (00)-13- (p).
(3.5)
where we use the shorthand (p(()) = Ex- V [s) for any policy л. For ease of presentation, we define the state visitation measure d= (s) = Expxp Ex P(st s [|)]]] and the state-action visitation measure d' (s) E P( 8, ag a [[s)]]]- We also use the 2h=
shorthand d' = d to further simplify the notation.
3.2. Learning Objective
3.3. Advantages of Token-Wiss MDP over Sentence- Bandit
Different from classical RL literature, where the sole goal is to maximize the reward function, the objective of RLHF is to maximize the reward function while ensuring that the learned policy does not deviate too much from the reference model (e.g., SFT model) too much. Inspired by this and the formulation of entropy-reeullrizze MDPs (Williams & Peng. 1991; Ziebart, 2010). for any policy æ, we define its corresponding regularized value-function by
Intuitively, the distinction between token-based and trajectory-based rewards reflects the difference between sparse and dense reward settings. In the sparse reward scenario, exploration proves to be more challenging. To illustrate this, we focus on the deterministic MDP with an action set size of A = [A]. We employ an autore- gressive policy x¹ to represent the policy of a powerful LLM, such as GPT-4, Fixing a prompt æ. given responses (v' = vl.H. = vi.h). the evaluation provided by x² is P(y" > y² |21 xr, V1. V2) *"y"' (1)+"=)*"(0 1g= [z) By com- paring this with the BT models of bandit in (2.1) and of our MDP formulation in (3.1), we observe that the sentence- wise reward r, and token-wise as r7 can be speci- fied by rx(xx y) = log n" (y (r) and re re((x, -1. log x" (y) x, 11: Intuitively, the responses that power- ful LLMs tend to choose have higher rewards. It is straight- forward to show that rs y) = -E re((x, y1:h-1), yn). We make the following natural assumption.
T(an, 1x)
()r))) 2+
plans ag) - P. log (ag. [ ma]. (3.2)
where the expectation FE, taken with respect to the ran- domness incurred by the policy æ. Here the summation ends
>'(y'
'In fact, these two trajectories can have different lengths, say 7² [(s), {(sh,,)))) M2 and 7² {(5,, ) with < H1, H2,H2 < H. These trajectories can be extended to length H by assuming that the state ending with Eos is absorbing and yields zero reward. This modification is to simplify the mathematical formulation and does not affect the problem modeling in (3.1). For the sake of clarity, the following theoretical discussion may focus on length-Ft H trajectories.
5

---

DPO Meets PPO: Reinforced Token Optimization for RLHF
Algorithm 1 Reinforced Token Optimization (Theoretical
4.1. Theoretical Version with Sample Complexity Guarantee
Version)
1: Input: Offline dataset D, À > 0, > and problem
We focus on the offline setting and assume the access to offline dataset D {(7") ([>",,/) that contains several trajectory pairs, where T" = {(s% a )/= is preferred over 7' = {(sh ) Each pair of trajectories shares the same initial state/prompt (i.e., sf = s)), but differs in the subsequent tokens. We also assume that the reward function linear, and our following results are ready to be extended to general function approximation (Chen al., 2022; Wang et al., 2023; Zhan et al., 2023a).
dependent coefficient p.
2: Compute OMLE based on D by maximizing the loglike- lihood given in (4.1).
3: Calculate the pessimistic reward F via (4.2).
4: Compute the corresponding optimal policy स with re- spect to F.
5: Output: policy म.
Assumption 4.1 (Linear Reward). We assume r(s, a) ((s, a) 8" for some known feature é S x A + IR and unknown vector 0 E Rd. We also assume that | :) ll2 S I. and | 22 S B.
Assumption 3.1. There exists a response y VI./1 satisfy- ing " x) 2 A for some 0 S 5 < H.
By the pigeon-hole principle, there must be response y such that r²(1) æ) 2 A-" implying that & S H. In prac- tice, is usually much smaller than H because the language model tends to choose the optimal response rather than making a random guess. Now, we define the interaction protocol and the sample complexity. The learner can de- termine response y Y1-H and receive either g2) or (re((1, 11=h-1), 0n )) k==. depending on whether the sentence-level reward or the token-wise reward is used. The sample complexity is defined as the number of responses and corresponding reward signals that need to be gathered to find the optimal response y" Vi.H with length H.
Following the standard reward learning pipeline (Ouyang et al., 2022), we learn the reward function via maximum likelihood estimation (MLE). Specifically, if we parametrize the reward function by 0, then the MLE is given by
MLEE = argmax (D((),
(4.1)
10ll25
)}
where (D( = =E(r (ie log(o E (=x axx
Σ h= ra(8) d.) Inspired by previous literature in of- fline RL (Jin et al., 2021; Rashidinejad et al.. 2021; Xiong et al., 2022; Zhu et al., 2023; Zhan et al., 2023a), given the MLE OLLE, we construct the pessimistic token-wise reward estimation as
Suppose Assumption 3.1 holds. In the setting where only the sentence-wir reward rs is accessible, finding the optimal response v² requires sample complex- ity of A", However, if tokenn-reward signals Te are available, there exists an algorithm that can find the optimal policy with sample complexity (I
Proposition 3.2.
F(s,a) = $(5, a) OMLE e (1(6(s;
(4.2)
where LTD
LE 11 (0(s)) at)
H ( 1 at) o a + Als. À > 0 is a tuning parameter, and o is a problem-dependent coefficient will be specified in Theorem 4.2 and (A.3), Finally, RTO outputs the optimal policy âे with respect to F. i.., के = argmax, V (s) for any s E S. The pseudocode of RTO is given in Algorithm 1.
The proof is deferred to Appendix A. Since & < H typically holds in practice, the gap between A and Amin((+1.11) is deemed large. Hence, Proposition 3.2 re- veals the significant separation of sample complexity be- tween two types of reward signals, providing theoretical insights into the superiority of the token-wise MDP formu- lation over the sentence-wise bandit formulation.
Theorem 4.2. Suppose Assumption 4.1 holds. For 33 > O. A 0, 8 E (0, 1), if we choose e = ((Vd) (see (A.3)), then the output policy āे of Algorithm satisfies
4. Reinforced Token Optimization
SubOpt(7) S B(e.a)dd' (11o(s,))]l;;
We develop the Reinforced Token Optimization (RTC) algo- rithm under the MDP framework introduced in Section 3. At a high level, RTO consists of two main steps: (i) token- wise reward learning, where RTO learns token- wise reward based on the preference data; and (ii) optimizing token-wise reward through RL training methods such as PPO. In Sec- tion 4.1, we provide a theoretically grounded version of RTC with guaranteed sample complexity. Then we present a practical implementation of RTO in Section 4.2.
B E Ea ( [s)
The detailed proof is deferred to Appendix A.2. The first term in Theorem 4.2 measures how well the offline dataset covers the trajectory generated by the policy T Typically, this term decreases at a rate of [D]-1/2 under the mild partial coverage assumption (Jin et al., 2021; Uchara & Sun, 2021; Xiong et al., 2022; Zhu et al., 2023; Zhan et al., 2023a),
6

---

DPO Meets PPO: Reinforced Token Optimization for RLHF
Algorithm 2 Reinforced Token Optimization (Practical Ver- sion)
rewrite Σ
Blog malan (86) Toar(an [8]] as
1: Input: Offline dataset D, parameters 31, 88 > 0, DPO algorithm DPO, and PPO trainer PPO-Update.
Σ
log (an | (88))
2: Compute ddpo < DPO(D) and let ioo Rref the
h=1
H r(sn, an) -V²(s)
reference model.
3: for (. = 1,..,, 7 do
h=1
4: Get a batch of samples Dj from the dataset D but we
only keep the prompts.
+
1 V)(+++1)), (4.3)
5: For each prompt x E De. generate a response y
TT-1( (-)))).
6: Calculate the token-wise reward Tito by (4.7).
7: Te + (at--UUdteeT--1111 Frto (zz, v) b.
where the first equality uses the closed-form of opti- mal policy rg(a|s) = exp( a) - V3 (s))/B} in (3.4), and the second equality follows from the fact that Q7 a) = r a) + -e-P(t- (V Ie
8: end for
9: Output: policy Ty.
in (3.3) with ro(s, a) r(s, a) + $log (a s). We focus on the typ- ical LLM generation scenario where the transition kemel is deterministic. Then we have (+) 0 in (4.3), yielding
where [D] is the size of the offline dataset. The second KL term is always negative, and it arises from the goal of learn- ing a regularized value. We also remark that our algorithm relies on the known transition kernel to compute the exact optimal policy with respect to f. While this is natural in the context of large language models, we provide insights on how to extend our findings to stochastic regularized MDPs and the variant of our RTO algorithm in Appendix B.
that Σx- r(sh, ox)) Σ Glog m.rr [sx) + Vi (s1). Building upon this result and combining it with the def- inition of the BT model in (3.1), for any trajectory pair {Té = {(5) 3 D3=1 satisfying st = =s. we have
=2(e% (&a)
H
H1
PP7 79)
Σ
(shh,
Σ
Em(hhaat)
aR)) (4.4)
h=1
There have also been previous works (Pacchiano et al., 2021; Chen et al., 2022: Wang al., 2023; Li et al., 2023c; Zhan et al., 2023a) studying RLHF under the MDP framework, also known dueling RL and preference-based RL. How- ever, these works do not consider the KL constraint, which is an essential component of RLHF Furthermore, they do not explicitly emphasize the superiority of the MDP framework over the contextual dueling bandit problem in the context of LLMs, and their proposed algorithms does not lead to corre- sponding practical algorithm leveraging dense rewards. In contrast, we will provide a practical implementation of our algorithm, demonstrating the practicality of our approach.
H
3 log T3(a/1 (
H
73 s%)
Σ
3
log
Tref (a) (9))
Tzof (1 (2))
A=1
An interesting observation is that, based on the autoregres- sive nature of policies, (4.4) aligns with the learning objec- tive of DPO proposed by Rafailov et al. (2023), but under the token-level MDP instead of the sentence-level bandit setup. Similar to the bandit setting where the learning ob- jective equivalent to a BT model with sentence- wise re- [)(x))
ward r (z, y) 3 log [.
(Rafailov al., 2023), (4.4) shows that the learning objective in token-wise MDP equiv- alents to BT model with token-wise reward function
4.2. Practical Implementation
r" r''Ss = (zz, Vi:h- 11::--1)..........1... an = =yh)
(an 8p)
log (36 Ir 11 Feef (yh x., 11)
In this subsection, we shift our focus to developing prac- tical version of RTO. The key challenge in implementing RIC in Algorithm 1 lies in learning the token-wie wise reward to be optimized from the offline data. In the most popular frameworks outlined in Instruct-GPTT (Ouyang et al., 2022), Claude (Bai et al., 2022), and LLaMA2 (Touvron et al., 2023) projects replace the last layer of the LLM with a lin- ear layer for a sealar output and maximize the log- likeliooddo as in (2.2). However, this approach gives only sentence- level reward. To bridge the gap in the literature, we present our practical version of RTO in Algorithm 2, which features a novel calculation of laken-wise reward. Our key observa- tion is that, given a trajectory T = {(5,, ( we can
=Blog- Reet (an sp)
(4.5)
where x is the prompt, 91:h-1 is the tokens generated so far, and yh is the token chosen at the current step. In contrast to the previous PPO implementation with sparse reward in (2.3), we will assign the token-wise reward function defined in (4.5) to each step. Formally, for any h, we define
B1 log (3 x. 11:h-1)
Feaf (yh N. W.Ihhhh1) B2 log Tyth x. B1sh-1)
Troof (yn xr, 91:h-1
81 log (d Ir, 11:h-1 B3 log
T Ir, Mish-1)
[yo x, 11:h-1)
Tyer Ix, v1 (4.6)
7

---

DPO Meets PPO: Reinforced Token Optimization for RLHF
Method
Method
Metric
Metric
SFT
DPO
R-DPO
SimPO
TDPO
PPO
RTO
RTO
Semi-RTO
DPPO
RS-PPO
AE (LC)
13:22
17,40
18.34
25.46
20.13
19.47
27.00
AE (LC)
27.00
23.77
21.09
27.52
AE (WR)
8.58
12.25
12.03
20.20
11.97
12.89
22.45
AE (WR)
22.45
19.17
13.06
21.69
AH (SC)
9.2
13.2
14.2
14.5
13.2
16.2
20.3
AH (SC)
20.3
19.0
13.1
19.2
AH (WR)
8.9
13.8
14.1
15.2
12.3
15.6
21.4
AH (WR)
21.4
19.7
12.1
19.9
Table 1. AlpacaEval 2 (AE) and Arena Hard (AH) results.
Table 2. Benchmark performance of ablations studies.
as the token-wist wise reward, where 31 and 32 are tuning pa- rameters, and π is the current policy to be updated. In the last step of (4.6), we use ddpo: the policy learned by DPO, as proxy for the unknown Tg. Finally, we employ PPO to optimize the following token-wiss wwie reward Irto
pacaEval 2 (Li et al., 2023a) and Arena-Hart (Li al., 2024). These benchmarks assess various conversational abil- ities across different types of queries. For AlpacaEval 2, we report both standard win rates (WR) and length-controlled win rates (LC). For Arena-Hard, we present the WR along with its style-controlled (SC) version. Both LCC and SC are specifically designed to mitigate verbosity bias of Ilm judge.
Tito (x, 31.h))
(4.7)
B1 log
(Mp 17,91A 1) Kror (Vh. x-1)
32 log Forllyn 1.))
s(en IZ.91K 1)
if S H 1,
Bi log (dp x.gra-1)
82 log riye (z.gra-1) FurrlUn (=
Fio x-1) +B3 (M,.R(F V1.H) if h H,
Implementation Details of RTO and Baselines. We em- play Llama-3-8B (Dubey et al., 2024) as the base model. For our comparative analysis, we implement several baselines. All subsequent models are initialized with an open- SFT model (Dong et al., 2024) that fine-tunes LLama-3-8B with a diverse mixture of high-quality data. We further train DPO model, which finetunes the SFT model using the pos- itive/negative preference data. Besides these two RL-frre algorithms, we compare three RLHF algorithms relying on RL training. The first one is the standard PPO algorithm, which directly optimizes sentence-level 8B reward in (2.3). Our proposed RTO algorithm leverages both token-wise signals from the DPO model and an additional IB sentence- wise reward MMLE to compute the RTO reward specified in (4.7). The policy is then trained to align with human prefer- ences using PPO updates, as detailed in Algorithm 2. For a comprehensive comparison, we include the following meth- ods as baselines: (i) R-DPO (Park et al., 2024) is a variant of DPO that regularizes response length, (ii) SimPO (Meng et al., 2024) optimizes average likelihood directly and (iii) TDPO (Zeng et al., 2024) reformulates and optimizes DPO at token-level. We include more implementation details in appendix D
where 3/ 2.000 0 is tuning parameter and T'LLE represents a sentence-level reward. This additional sentence-level reward helps prevent responses from becoming either extremely long or extremely short. This aligns with the observation that ensemble rewards can effectively mitigate the overop- timization issues (Coste et al., 2023). We also remark that the sentence-level reward TMLE can be much smaller in mag- nitude compared to both the policy model (actor) and DPO reward model, making the overall computational cost of RTO comparable to the standard RLHF pipeline: The lower cost of using a much smaller critic in PPO compensates for both the small extra cost required by DPO than reward model, and the training and serving of the tiny reward model. For parameter selection, 33 can be set to 1 when TMLE is included. This choice is without loss of generality, as the key factor is the ratio of B3 to 31 and 332 rather than its absolute value. 332 can be chosen similarly standard PPO configurations. The only extra hyperparameter 81 can be set small prevent the DPO reward from dominating, thereby requiring minimum tuning.
5. Experiments
RTO Outperforms PPO and Other Direct Preference Learning Algorithms. As demonstrated in Table 1, while all evaluated algorithms show improvement over the base SFT model, our RTC algorithm achieves superior perfor- mance across all benchmarks. Specifically, RTO outper- forms PPO by achieving a 7.53% higher win rate in the AlpacaEval 2 LC benchmark and a 4.1% higher win rate in the Arena- SC benchmark. These results highlight the effectiveness of incorporating token-wise reward (dense reward) into PPO training. Furthermore, when compared to the leading preference learning baselines, RTO demonstrates improvements of 2 and 4 points in the AlpacaEval 2 LC benchmark and the Arena-Hard SC benchmark, respectively.
5.1. Benchmark Results
We present a thorough comparison of RTO with PPO and other widely used direct preference learning algorithms on popular benchmarks to highlight RTO's strong performance.
Task, Data, and Evaluation. To assess the overall qual- ity of generated text responses across multiple dimensions (c.g., helpfulness, accuracy, and clarity), we employ the dataset UltraFeedback (Cui et al., 2023) that contains com- prehensive human feedback annotations on model outputs. We evaluate models using two established benchmarks: Al-
8

---

DPO Meets PPO: Reinforced Token Optimization for RLHF
1.2
1.2
1.0
1.0
24-
Reward
0.8
0.8
0.6
0.6
0.4
0.4
RIO
0.2
Sumi RTO
0.2
RS-PPO
DPP
DPPO
0.0
0.0
10
20
30
40
50
60
0
10
20
30
40
50
60
Training step
Training step
0.2
Data Fradion
(a) Effect of different reward granularity.
(b) Effect of reward shaping.
(c) Data scaling behavior of PPO and RTO.
Figure 2. (a) and (b) show the obtained reward of PyLE throughout training. (c) shows the AlpacaEval performance of PPO and RTO when trained on fractions of samples.
5.2. In-depth Analysis of RTO Performance
dataset. These experiments evaluated the ability of RTO and PPO to learn an effective policy with limited data. As shown in Figure 2(c), RIC matches PPO's performance using only about 1/8 of the data and ultimately surpasses PPO's final performance. Additionally, RTO exhibits superior data seal- ing behavior compared to PPO - RTO continues to improve with more data, while PPO's 's performance saturates early.
In this subsection, we provide a detailed analysis of RTO. First, we examine the influence of reward granularity by comparing the performance of RL training based on dif- ferent reward types. Next, we demonstrate that the token- wise reward (provided by DPO) primarily serves as reward shaping, which contributes significantly to RTO's success. Finally, we investigate RTO's sample efficiency to support our theoretical findings.
Additional Experiments. To showcase the applicability of RTO, we conducted additional experiments demonstrat- ing: ()) its effectiveness beyond PPO by incorporating the learned token-wise reward function into REINFORCE-type algorithms (Williams, 1992; Hu, 2025), yielding significant improvements (Appendix E), and (ii) its utility for diverse alignment tasks beyond dialogue, such as text summariza- tion (Völske et al., 2017) (Appendix F).
The Influence of Reward Granularity. We analyze three reward granularity settings by redistributing token-level re- wards used in RTO: (i) RTO, where rewards are assigned to each token; (ii) Semi-RTO, where the rewards of all tokens in each sentence are reassigned to their delimiter, and (iii) DPPO, where all rewards are delayed and assigned to the EoS token. Table and Figure 2(a) clearly demonstrates that denser rewards lead to better performance.
6. Conclusion
Reward Shaping via DPO Reward is the Key to RTO's Success. We demonstrate that the superior performance of RTO is not primarily due to replacing the reward model trained with MLE with the implicit reward from DPO. In- stead, its advantage lies in its role as a reward-shaping mech- anism. To illustrate this, we compare RTO and DPPO with another setup, RS-PPO, where the reward matches Trto in (4.7), except for subtracting the DPO implicit reward (ula)) from the last token. This adjustment re- sults in total reward equivalent to TMLE(2, y). effectively employing the DPO wisen-wise implicit reward for reward shaping. From Table and Figure 2(b), we observe that the main contribution the DPO reward to improving RL training lies in reward shaping rather than altering the total reward through its exact value.
In this work, we propose MDP formulation for RLHF that better characterizes token-wise wise information, along with theoretical insights demonstrating its superiority. Build- ing upon this formulation, we introduce a novel algorithm called Reinforced Token Optimization (RTO), which lever- ages token-wise rewards to improve the policy. RTO is shown be both provably sample-efficient and practical. Our practical implementation involves novel token-wise reward learning approach via DPO, followed by optimiza- tion using PPO. This innovative combination of DPO and PPO allows RTO to effectively utilize token-level informa- tion and significantly improve the performance of baselines. Furthermore, our research opens up several intriguing fu- ture directions, such as designing alternative methods for learning token- wise rewards beyond DPO.
31 log =xc((y/>)
Sample Efficiency of RTO. To validate the theoretical claims in Theorem 4.2, which guarantees the provable sam- ple efficiency of RTO, and Proposition 3.2, which demon- strates that RTO is more efficient than PPO due to its use of token-wise wise rewards instead of sentence-wise rewards, we conducted experiments using only a fraction of the full
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.

---

DPO Meets PPO: Reinforced Token Optimization for RLHF
Acknowledgement
Cai, Q., Yang, Z., Jin, C., and Wang. Z. Provably efficient exploration in policy optimization. In International Con- ference Machine Learning, pp. 1283-1294, PMLR, 2020.
LW is supported by National Science and Technology Ma- jor Project (2022ZD01 14902) and National Science Foun- dation of China (NSFC92470123,, NSFC62276005). DH is supported by National Science Foundation of China (NSFC62376007), This work is partially supported by the Shanghai Committee of Science and Technology (Grant No.21DZ1100100).
Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.152111 2023.
References
Cen, S., Cheng, C.,, Chen, Y., Wei, Y., and Chi, Y. Fast global convergence of natural policy gradient methods with entropy regularization. Operations Research, 70(4): 2563-2578, 2022.
Agarwal, A., Kakade, S. M., Lee, J. D., and Mahajan, G. On the theory of policy gradient methods: Optimality. approximation, and distribution shift. The Journal of Machine Learning Research, 22(1):4431-4066 2021.
Chan, A. J., Sun, H., Holt, S., and van der Schaar, M. Dense reward for free in reinforcement learning from human feedback. arXiv preprint arXiv:2402.007788 2024.
Ahmadian, A., Cremer, C., Gallé, M., Fadace, M., Kreutzer, J.. Üstün, A., and Hooker, S. Back to basics: Revisit- ing reinforce style optimization for learning from human feedback in Ilms. arXiv preprint arXiv:2402. 14740, 2024.
Chen, L., Lu, K., Rajeswaran, A.. Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. De- cision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084-15097, 2021.
Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Pieter Abbeel, O., and Zaremba, W. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
Chen, X., Zhong, H., Yang. Z., Wang, Z., and Wang, L. Hluman-in-the-lowe Provably efficient preference- -base reinforcement learning with general function approxima- tion. In International Conference on Machine Learning, pp. 3773-3793, PMLR, 2022.
Anthropic. Introducing claude.
2023. https://wwwt wwt.wwww...rppc..ccomoooo com/ index/
URL
introducing-claudee
Azar, M. G., Rowland, M., Piot, B., Guo, D., Calandriello, D., Valko, M., and Munos, R. A general theoretical paradigm to understand learning from human preferences. arXiv preprint ar ArXiv:2310. 12066, 2023.
Choshen, L., Fox, L., Aizenbud, Z., and Abend, O. On the weaknesses of reinforcement learning for neural machine translation. arXiv preprint arXiv:1907.01755 2019.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training helpful and harmless assistant with rein- forcement learning from human feedback. arXiv preprint 20X1v:2204.05862. 2022.
Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information pro- cessing systems, 30, 2017.
Coste, T, Anwar, U., Kirk, R., and Krueger, D. Reward model ensembles help mitigate overoptimization. arXiv preprint arXiv:23 2023.
Bengs, V., Busa-Fekete, R., El Mesaoudi-Pand A., and Hüllermeier, E. Preference-based online learning with dueling bandits: A survey. The Journal of Machine Learn ing Research, 22(1):278-388 385, 2021.
Cui, G., Yuan, L., Ding, N., Yao, G., Zhu, W., Ni, Y., Xie, G., Liu, Z., and Sun, M. Ultrafeedback: Boosting language models with high-quality feedback, 2023.
Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O' Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., al. Pythia: A suite for ana- lyzing large language models across training and scaling. In International Conference Machine Learning, pp- 2397-2430. PMLR, 2023.
Cui, G., Yuan, L., Wang, Z., Wang, H., Li, W., He, B., Fan, Y., Yu, T., Xu, Q.. Chen, W., Yuan, J., Chen, H., Zhang, K., Lv. X., Wang, S., Yao, Y., Peng. H., Cheng, Y., Liu, Z., Sun, M., Zhou, B., and Ding, N. Process reinforcement through implicit rewards, 2025.
Bradley, R. A. and Terry, M. E. Rank analysis of incom- plete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-34) 3445244444511 1952.
Dong, H., Xiong, W., Goyal, D., Zhang, Y., Chow, W.,
Pan, R., Diao, S., Zhang, J., SHUM, K., and Zhang,
10

---

DPO Meets PPO: Reinforced Token Optimization for RLHI
RAFT: Reward ranked finetuning for generative founda- tion model alignment. Transactions Machine Learning Research, 2023, ISSN 2835-8856.
Kakade, S. and Langford, J. Approximately optimal ap- proximate reinforcement learning. In Proceedings of the Nineteenth International Conference on Machine Learn- ing. pp. 267-274, 2002.
Dong, H., Xiong, W., Pang, B., Wang, H., Zhao, H., Zhou, Y., Jiang, N., Sahoo, D., Xiong, C., and Zhang, T. Rlhf workflow: From reward modeling to online rlhf, 2024.
Kingma, D. P. and Ba, J. Adam: A method for stochastic op- timization, 2017. URL https: http::://axxxvoorgo// /rbbb///
1412. 6980.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Ilama herd of models. arXiv preprint arXiv:2407.21783, 2024.
Lambert, N., Pyatkin, V., Morrison, J., Miranda, L., Lin, B. Y., Chandu, K., Dziri, N., Kumar, S., Zick, T., Choi, Y., et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint MxXiv:2403.13788 2024.
Faury, L., Abeille, M., Calauzènes, C., and Fercoq, O. Im- proved optimistic algorithms for logistic bandits. In- ternational Conference on Machine Learning, pp. 3052- 3060. PMLR. 2020.
Li, T., Chiang, W.-L., Frick, E., Dunlap, L.,, Wu, T., Zhu, B., Gonzalez, E., and Stoica, I. From crowdsourced data high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406. 1939, 2024.
Gulcehre, C., Paine, T. L., Srinivasan, S., Konyushkova, K., Weerts, L., Sharma, A., Siddhant, A., Ahern, A., Wang, M., Gu, C., et al. Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308.0899. 2023.
Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang, P., and Hashimoto, T. B. Alpacae- val: An automatic evaluator of instructionfollloinng models. https: htops://gaa //tphugttu./../////// www.//ttttottaiteeee a alpaca_ _eval 5 2023a.
Hoang Tran, Chris Glaze, B. H. Snorkel-mistral-piin dpo. 2024. URL https://huggingg / hateeeaaassoorreeeMsttaaaaa
Li, Z., Xu, I., Zhang. Y., Yu, Y., Sun, R., and Luo, Z.- Q. Remax: simple, effective, and efficient reinforce- ment learning method for aligning large language models. arXiv e-prints, pp. arXiv-2310, 2023b.
Hu, J. Reinforce++: A simple and efficient approach for aligning large language models. arXiv preprint 2XXi:2501.03266 2025.
Li, Z., Yang. Z., and Wang, M. Reinforcement learning with human feedback: Learning dynamic choices via 1848,, pessimism. arXiv preprint arXiv: /843. 2023c.
Hu, J., Tao, L., Yang, J., and Zhou, C. Aligning language models with offline reinforcement learning from human feedback. arXiv preprint arXiv:2308. 10050, 2023.
Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let's verify step by step. arXiv preprint urXiv:2305.20050, 2023.
Hu, J., Wu, X., Zhu, Z. Xianyu, Wang, W., Zhang, D., and Cao, Y. Openrlhf: An easy-to-use, scalable and high-pue perfrorrffreeeceeeee rihf framework. arXiv preprint 20Xiv:2405.11143, 2024.
Liu, Q., Chung, A., Szepesvári, C., and Jin, C. When is partially observable reinforcement learning not scary? In Conference on Learning Theory, pp. 5175-5220. PMIR, 2022.
Huang. J., Yardim, B., and He, N. On the statistical effi- ciency of mean-field field reinforcement learning with general function approximation. In International Conference Artificial Intelligence and Statistics, pp. 289-297. PMLR, 2024.
Liu, Z., Lu, M., Xiong, W., Zhong, H., Hu, H., Zhang, S., Zheng, S., Yang, Z., and Wang. Z. Maximize explore: One objective function fusing estimation, planning, and exploration. In Thirty- Conference Neural Information Processing Systems, 2023.
Jang. J., Kim, S., Lin, B. Y., Wang, Y., Hessel, J., Zettle- moyer, L., Hajishirzi, H., Choi, Y., and Ammanabrolu, P. Personalized soups: Personalized large language model alignment via host-hoc parameter merging. arXiv preprint arXiv:2310. 15544 2023.
Meng, Y., Xia, M., and Chen, D. Simpo: Simple preference optimization with a reference-free reward. arXiv preprint 2tti::2005.44334 2024.
Nakano, R., Hilton, J., Balaji. S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., et al. with hu-
Jin, Y., Yang, Z., and Wang, Z. Is pessimism provably efficient for offline ri? In International Conference on Machine Learning, pp. 5084-5096 PMLR, 2021.
Webgpt: frowser-assister man feedback. arXiv preprint arXiv:2 2021.

---

DPO Meets PPO: Reinforced Token Optimization for RLHF
OpenAl. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023.
Uehara, M. and Sun, W. Pessimistic model-based offline reinforcement learning under partial coverage. arXiv preprint arXiv:2107.06226 2021.
Ouyang, L., Wu, J.. Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744 2022.
Uesato, J., Kushman, N., Kumar, R., Song, F., Siegel, N., Wang, L.,, Creswell, A., Irving, G., and Higgins, I. Solv- ing math word problems with process-and outcome- feedback. arXiv preprint ar arXiv:2211 2.X::2221..447711 1.1275, 2022.
Pacchiano, A., Saha, A., and Lee, J. Dueling rl: reinforce- ment learning with trajectory preferences. arXiv preprint arXiv:2 /0t:://11..48500 2021.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, I.., Gomez, A. N., Kaiser, t.,, and Polosukhin, I. At- tention is all you need. Advances in neural information processing systems, 30, 2017.
Park, R., Rafailov, R., Ermon, S., and Finn, C. Disentan- gling length from quality in direct preference optimiza- tion. arXiv preprint arXiv:2403. 19159, 2024.
Völske, M., Potthast, M., Syed, S., and Stein, TI; dr: Min- ing reddit learn automatic summarization. In Proceed- ings of the Workshop on New Frontiers in Summarization, pp. 59 63, 2017.
Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning,
C. D., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305. 18290, 2023.
Wang, H., Lin, Y., Xiong, W., Yang, R., Diao, S., Qiu, S., Zhao, H., and Zhang, T. Arithmetic control of Ilms for diverse user preferences: Directional preference alignment with multi-objective rewards. arXiv preprint wrXiv:2402.1857/ 2024.
Rafailov, R., Hejna, J., Park, R., and Finn, C. From to q': Your language model is secretly a q-function. arXiv preprint arXiv:2404. 1020044.233. 12358, 2024.
Rashidinejad, P., Zhu, B., Ma, C., Jiao, J., and Russell, S. Bridging offline reinforcement learning and imita- tion learning: A tale of pessimism. Advances in Neural Information Processing Systems, 14:11702-11716, 2021.
Wang, Y., Liu, Q. and Jin, C. Is rlhf more difficult than standard rI? arXiv preprint arXiv:2306. 2023.
Williams, R. J. Simple statistical gradient- following algo-
rithms for connectionist reinforcement learning. Machine learning, 8:229 -266, 1992.
Rosset, C., Cheng, C.-A., Mitra, A., Santacroce, M., Awadal- lah, A., and Xie, T. Direct nash optimization: Teaching language models to self-improve with general preferences. arXiv preprint arXivv2404.03715, 2024.
Williams, R. and Peng. Function optimization using connectionist reinforcement learning algorithms. Con- nection Science, 3(3):241 1991.
Saha, A. Optimal algorithms for stochastic contextual prefer- ence bandits. Advances in Neural Information Processing Systems, 34:30050-30000 2021.
Wu, T., Yang. Y., Zhong, H., Wang, L., Du, S.,, and Jiao, J. Nearly optimal policy optimization with stable at any time guarantee. In International Conference on Machine Learning, pp.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint /Tl.::/770006444 2017.
PMLR, 2022.
Wu, Z., Hu, Y., Shi, W., Dziri, N., Suhr, A., Ammanabrolu, P., Smith, N. A., Ostendorf, M., and Hajishirzi, H. Fine- grained human feedback gives better rewards for language model training. Advances in Neural Information Process- ing Systems, 36, 2024.
Tang, Y., Guo, Z. D., Zheng. Z., Calandriello, D., Munos, R., Rowland, M., Richemond, P. H., Valko, M., Pires, B. Á., and Piot, B. Generalized preference optimization: A unified approach to offline alignment. arXiv preprint arXiv:2402.05749, 2024.
Xiong, W., Zhong, H., Shi, C., Shen, C., Wang, L., and Zhang, T. Nearly minimax optimal offline reinforcement learning with linear function approximation: Single-agent mdp and markov game. arXiv preprint 15555 2022.
Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M.. Hauth, A.. et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. 1805, 2023.
Xiong, W., Dong. H., Ye, C., Wang. Z., Zhong, H., Ji, H., Jiang, N., and Zhang, T. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Founda- tion Models, 2023.
Touvron, H., Martin, I.., Stone, K., Albert, P., Almahairi, A., Babaci, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288. 2023.
12

---

DPO Meets PPO: Reinforced Token Optimization for RLHI
Yang, R., Pan, X., Luo, F., Qiu, S., Zhong, H., Yu, D., and Multi-objective alignment
Ziebart, B. D. Modeling purposeful adaptive behavior with the principle maximum causal entropy. Carnegie Mel- lon University, 2010.
Chen, J. Rewards-in-contes of foundation models with dynamic preference adjust- ment arXiv preprint 2xXi:2442..1207 2024a.
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning language models from human preferences, arXiv preprint arXiv: 1rX.009909008993, 2019.
Yang, S., Zhang, S., Xia, C.. Feng. Y., Xiong, C.,, and Zhou, M. seeference-groundee token-level guidance for language model fine-tuning. Advances in Neural Infor- mation Processing Systems, 36, 2024b.
Ye, C., Xiong, W., Zhang, Y., Jian, N., and Zhang, T. A the- oretical analysis of nash learning from human feedback under general kl-regularized preference. arXiv preprint arXiv:2402.07.: 00.44002.33344 14, 2024.
Yin, Y., Yang, S., Xie, Y., Yang, Z., Sun, Y., Awadalla, H., Chen, W., and Zhou, M. Segmenting text and learning their rewards for improved rlhf in language model. arXiv preprint wwxiv:2501.02790 2025.
Yue, Y., Broder, J., Kleinberg, R., and Joachims, T. The k-armed dueling bandits problem. Journal of Computer and System Sciences, 78(5):1538-155 1556, 2012.
Zeng, Y., Liu, G., Ma, W., Yang, N., Zhang, H., and Wang, J. Token-level direct preference optimization. arXiv preprint #rXiv:2404111992 2024.
Zhan, W. Uehara, M., Kallus, N., Lee, J. D., and Sun, W. Provable offline reinforeement learning with human feedback. arXiv preprint 2081v22305.14816, 2023a.
Zhan, W. Uchara, M., Sun, W. and Lee, D. How query human feedback efficiently in rl? arXiv preprint arXiv:2305. /8555 2023b.
Zhang, T. Mathematical analysis of machine learning algo- rithms. Cambridge University Press, 2023.
Zhao, Y., Joshi, R., Liu, T., Khalman, M., Saleh, M., and Liu, J. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint 230.::2305..0425. 2023.
Zhong, H. and Zhang, T. A theoretical analysis of optimistic proximal policy optimization in linear markov decision processes. Advances in Neural Information Processing Systems, 36, 2024.
Zhong, H., Xiong, W., Zheng, S., Wang, L., Wang, Z., Yang. Z., and Zhang, T. Gec: A unified framework for interactive decision making in mdp, pomdp, and beyond. arXiv preprint arXiv:221 1.09962, 2022.
Zhu, B., Jiao, J., and Jordan, M. Principled reinforcement learning with human feedback from pairwise or w-wi comparisons. arXiv preprint arXiv:2301. 2023.
13

---

DPO Meets PPO: Reinforced Token Optimization for RLHI
1/8
7/8
7/8
7/8
7/8
3/4
1/8
3/4
1/8
3/4
3/4
1/2
1/4
1/2
1/4
1/2
1/4
1/2
Figure An illustration of our efficient learning algorithm for token-wise reward setting with A = 2, i represent real numbers between and 1/8. We do not specify their exact values they do not influence the optimal path. All nodes in N are colored red, while other nodes are blue, with the optimal leaf node 1/2 emphasized in dark blue. Each node yı.. is labelled with n' (y1sk =)) If non-optimal path (response) is selected, red node in / will be identified, and paths containing this node will be deleted. Here we visualize the pracess of choosing path ending with *, i. and 1/4, respectively. At most m = samples are needed identify the optimal response.
and g = 1. Here * and
A. Detailed Proofs
A.1. Proof of Proposition 3.2
Proof. If only the sentence-level reward is available, the learner must try every possible response and determine the optimal one by ranking the collected sentence-level reward signals, resulting in a sample complexity of A Instead, we consider a binary tree with depth + 1, where each node is indexed by some token sequence Y1:h and has children (yi (h+1))maase All A leaf nodes denote a unique prompt-respone pair (I, 31:H). We define two disjoint node sets:
= (Vish : T (Y1:h x)) < A A (0155""((111111 xx) 2 A $
N (3i: : T (Vi:H x) 2 A
(A.1)
Our key observations are that (i) each path must contain a node in N or M (ii the path containing the node in / is suboptimal; and (iii) [N S A The exploration strategy is to query a new path that does not contain the nodes in N U that have been visited. Since each query of new path (response with length H) can identify new additional node in N U after least A queries, we collect a set paths where each node in N U belongs one of the paths. Finally, ranking all gathered rewards of the node in * identifies the optimal y" yi. Together with the fact that there exists most A nodes, finish the proof of Theorem 3.2. To facilitate understanding, visualize simplified learning process in Figure 3.
A.2. Proof of Theorem 4.2
Recall that the visitation measure of policy is
20
dtd (s) = Ee rup
Σ
P(st = s s1) s
d' (s, = Esjap
Σ
P(sn = s, = aa
(A.2)
Under this notation, we can rewrite the value function in (3.2) as
VT(p) (p) = (o.a) [r(s,a) (L( (1) [re (1s)]]
For simplicity, we will use the shorthand d' = d
Proof of Theorem 4.2. Our proof relies on the following standard MLE analysis.
14

---

DPO Meets PPO: Reinforced Token Optimization for RLHF
Lemma A.I (MLE Analysis). It holds with probability 6 that
d (og((1/))
[[0MLE - -0"11x:0 = C.
+ AB?,
(A.3)
T
where C is an absolute constant and T exp2+exp(-22111 exp(2HLB)).
Proof. See e.g., Faury et al. (2020); Pacchiano al. (2021); Zhu et al. (2023) for detailed proof.
Back to the proof of Theorem 4.2, we first decompose the suboptimality gap defined in (3.5)
SubOpt( V (p:r) - V rp:
(s.a)=d [r(s, a) 3 KL (TT(( ()]l (-1 [[]]]]]]]]]]]]] (B(r,a)ddi (r(s,a) - B KKLKKL((F FTe( (s)]]
(s,a))dd [r(s, a) f(s, a)] + ((s,))= [F(s, - r(s, a)] + V/8 v0 (p: - V (p:
(A.4)
Term(i)
Term(ii)
Term(iii)
Then we analyze these three terms respectively.
Term (i). Recall that the pessimistic reward defined in (4.2) takes the form F(s,a) = (88 a) OMLE - e (11((,
Then we can rewrite Term (i) in (A.4) as
Tirm(i) = (s (0ss, a) T(" - OMLE) + e- (10(s, 2|
< as [lo((s a) a>حا==''' -10 OMEElED te (l((s,))
S 2g E (( a)<2=1)
(A.5)
where the first inequality is obtained by Cauchy-Schware inequality, and the last inequality follows from Lemma A.1.
Term (ii). Similar to the derivation of (A.5), we have
Term (ii) = Es [o,s, a) (OMLE- - 00) - e | a) [bbs
< E (s.a [1( a) (z)) ||0MLE 0'11> e
< 0,
(A.6)
where the first inequality uses Cauchy-Schwarz inequality, and the last inequality is implied by Lemma A.1.
Term (iii). To handle this term, we introduce the following performance difference lemma for MDP with KL constraint.
Lemma A.2 (Performance Different Lemma). For any reward function r and policy pair (æ, n²), holds that
V (p:r) Vz (p:r) = (s.a)~=d= '2 (8, - Vy' (s;r) B log [[|a))]
Proof. See Appendix A.3 for a detailed proof.
When 8 = 0, the regularized MDP becomes the standard MDP, and Lemma A.2 reduces to the standard performance
difference lemma (Kakade & Langford, 2002). Applying Lemma A.2 to Term (iii) in (A.4), we have
Term(iii) = E (s ( R) - V (s; B log (a | s)]
(s,0)~a- [gl (a 5) - 3 log (a)s)
- Es KL (T [- ss)]]
(A.7)
where the second equality follows from the fact that म is the optimal policy with respect to (s; and the expression of optimal policy (a s) = exp((Q](s,,; as v (s: f))/B} /} in (3.4), and the last equality is obtained by the definition of KI. divergence.
15

---

DPO Meets PPO: Reinforced Token Optimization for RLHF
Finishing the Proof.
Plugging (A.5), (A.6), and (A.7) into (A.4), obtain that - 3 Egnd' ( {
SubOpt(f)) (T) S 20 ((aa)=
which finishes the proof of Theorem 4.2.
Remark A.3. If we do not have access to the exact optimal policy मे with respect to f, we can use the policy optimization algorithms to find a opar-optimal optimal policy ã. In such case, Term (iii) in (A.6) becomes Vi3 (p: F) - V (p: =
V (p: F) V (p: V () V (p:7), and need handle the additional error term V (f; V3 (p: This type of error analysis has been established for NPG (Agarwal al., 2021; Cen et al., 2022) and PPO (Cai et al., 2020; Wu et al., 2022; Zhong& & Zhang, 2024).
A.3. Proof of Lemma A.2
Proof of Lemma A.2. Without loss of generality, assume that the initial state is fixed state $1 E S. For simplicity, also omit the dependency of r in the regularized Q- Q-function and value function. First, we have
Vī (s1) Vig (s1) V (s1) B m mn) (rs a1) + I VV3 (s2)]]
(A.8)
+ E a Nal (ooet(l11)))e)) 10) (81, V3 (81).
where uses the equality Q (1) r1(s1,0)) + (1 1V (s2) in (3.3) with ry(s, r(s,a) + 3 log (a s). By (3.3), we further have
Vī (s1) Ea, mn 11)s-1llllggg log (a1 $1) (3(5),a1)
Eay Ea mT( (s) log T(a (1) + (())) + EegeP (n
Plugging this into Term (+) of (A.8), we have
(+) = Eur - [s)) - 3 log π (aa (1) + E. la (s2)]] - Ba en) [sspp (.) (52)l
[33( = -ee ( 133 log ( [8)]] + 4y [V (82) - V (82).:
(A.9)
where we use d7 (8) to denote the visitation measure at the h-th step. Meanwhile, we rewrite (*+) in (A.8) as
(**) Baj .1) 10 (51, VT (s1))-
(A.10)
Plugging (A.9) and (0. into (A.8), have
(s1) - Vs (s1) = Ey 1V (82) - V3 (82)) + E (s 127 (s),ar)) - V3' (s1) - 3 3og ( ss)]
Σ E Ess 127 (sh, an) - -V (sx) 3log T (an ss)]]
1=1
(s 12 (s.a) - vz (s) 3 log T(a s)].
where we use (,) to denote E., Ex, on ( s..) and the definition of dd in (A.2). Therefore, we conclude the proof
of Lemma A.2.
B. Variants of Reinforced Token Optimization
Different from Algorithm I where the learner constructs pessimistic reward estimation and then outputs its corresponding optimal policy. Indeed, we can also perform pessimistic planning with respect to the value function to find the near-optimal policy:
f argmax min (= TO - 3 Es-ar (TT((1)))][ I 8E0
(B.1)
16

---

DPO Meets PPO: Reinforced Token Optimization for RLHF
where O (110112 < B 110 OMeElEE S e} and OMLE is given in (4.1). Here e is the problem-dependent constant
in (A.3) and 20 Σ(- -))) D (B.1), 1), we have the following theoretical guarantee.
a)) - O a [Z= (6(s), o( ) ] + Aaa is the covariance
matrix. For policy
Theorem B.1. Suppose Assumption 4.1 holds. For 3 > 0, A > 0, (0, 1), if we choose Ö(Vd) (see (A.3)), then the
output policy कं of (B.1) satisfies
Sub & 20 | (s,a)dd (0(s, (6(s,,)]ll: a)
Proof of Theorem B.I. For case of presentation, we define
(p) min { (((s,,))vd (o(s, a))) - B- Es-a- ( [-]]] BEB
By Lemma A.I, we know that E with probability This implies that
(p) s ( (((ss 0t - BB. E g [KT (8)+ Tret [s)] (p) (p).
(B.2)
Meanwhile, by (B.1), we have
1 (p) S (o()).
(B.3)
Combining (B.2) and (B.3), we obtain
0 (p) - V (p).
Plugging this into the definition of the suboptimality gap (3.5), we have
SubOpt = (p) V (p) - (p) 1/50 (p)
Now we introduce the notation of 0:
i argmin (s.)))) (A(s, (((s))))) a)) B Eaad' [KL ( Troe [- lee
Under this notation, we further obtain that
SubOpt(*)) S [(( [10) -) T) ((s a)]
0)
(a [10" - (MLE) o(s, a)] ((s.a)=d- [0MLE - T) a)]
< (10MLE - 0* 111x + |10MLE - 011x;) ( [| (d(8,)))lls a)lls>
S 20 ||E (o(s, a)]l
where the second inequality uses we conclude the proof of Theorem B.1.
Schwarz inequality, and the last inequality is obtained by Lemma A.I. Therefore,
Remark B.2 (Extension to Unknown Transitions). In (B.1), we assume that the transition kernel known so that we can compute the state distribution d induced by the policy æ. Although this is natural in LLMs, we briefly sketch the extension the unknown transition setting. Following Zhan et al. (2023a), which is inspired by previous works standard reward-based RI theory (Uehara & Sun, 2021; Liu et al., 2022; Zhong et al., 2022; Liu et al., 2023; Huang et al., 2024), we can also construct a confidence set for the transition kernel
2
2
Pp {p={P: P:
Σ
Σ
log P(T)) 2 max
Σ (r)
Σ
log P(r))
(E
where P(T) the probability of observing the trajectory under the transition P and Ç is tuning parameter. With a proper choice of Ç, also show that P E Op with high probability. Then we can perform the following pessimistic planning
{ ((Essa)-0p(0(s (d(s, a))) To B E.wdy [L[(T( ()][=(e((- [1))]
argmax PEO,
min
where d7 denotes the state distribution induced by policy under the environment P. Combining the analysis of Theorem B.I and previous work on offline RL (Uehara Sun, 2021; Zhan et al., 2023a), we can also establish a similar result to Theorem B. but with an additional estimation error for the transition kernel part. As this part is standard and not the focus of our work, we omit it for simplicity.
17

---

DPO Meets PPO: Reinforced Token Optimization for RLHF
C. Additional Discussions
C.1. Direct Preference Optimization
Direct Preference Optimization (DPO) is a representative algorithm of the direct preference learning algorithm (Rafailov al., 2023; Zhao et al., 2023; Azar et al., 2023; Tang et al., 2024). From high level, these type of algorithms aim to skip the reward modeling and learn directly from the preference data, hence the name direct preference learning. In this section, introduce the mathematical principle of DPO for completeness.
We first recall that in the original two-staged learning paradigm, we aim to optimize the following KL-regularized target:
x x)) Tref (y x)
π = argmax E (x-p.y>x((a))
TMLLE y) - B log
(C.1)
where TMLE is the MLE of the BT model on the offline preference dataset D obtained via
TMLE = argmax
Σ
log o (r(z, y") r(x, v/)).
(C.2)
One notable feature of this KL-constrained optimization problem is that it admits closed-form solution, summarized in the following lemma.
Lemma C.1 (Solution of KL-regularized Optimization (Proposition 7.16 and Theorem 15.3 of Zhang (2023))). Given
loss functional with respect to (-11)). written as
(y [)) = 3- KI. ( x)
(I.(TT(9) T 1²) [0)||= meesl x)) exp (r(c,.u))
Bys
(12))))))))))))))) (x)
rlæ, y) Blog
the minimizer of the loss functional x, (y(yla) (x) x (T (x) exp ( (2r(xc) u) also known as Gibbs distribution.
Therefore, for any fixed reward function r, it leads closed-form policy:
1
Teef (y |x) exp (frl..)))
Tr (y (x)
Z(x)
where Z(x)= = Ey Tret xx) (2x,.//) )) the normalization constant. Then, we can solve the reward as
r(z, y) 8logg ()) Teat (y x) + Bog Z(z).
(C.3)
We can plug (C.3) into (C.2) to get
IT. (y 22) Trof (2) (x)
Blog (x 22) Teat (y² (x)
f
argmax
Σ
log Bog
(C.4)
Clearly, if r is the solution of (C.2), the क. is the solution of (C.4). On the other hand, if ग is optimal for the DPO target in (C.4), then, the induced implicit reward ,3 log x(y |x) is optimal for (C.2).
Interestingly, while the DPO is derived from the sentence- reward function and BT model, the implicit reward naturally gives tosen-wise characterization of the prompt-response pair and can be leveraged as dense reward signal for the PPO training.
C.2. Autoregressive Policy
For the policy defined in contextual dueling bandit setting, it maps from a prompt to a complete sentence. For ease of presentation, we call this type of policy the predetermined policy since it determines the entire sentence regardless of the generation process. In contrast, the Markov policy defined in the MDP formulation generates responses autoregressively: considers not only the prompt but also the tokens generated so far. By definition, the Markov policy is at least as good as the policy that determines the whole sentence based solely on the prompt. deterministic MDPs, the optimal action sequence
18

---

DPO Meets PPO: Reinforced Token Optimization for RLHF
is predetermined given the initial state, which demonstrates the equivalence of these two types of policies. However, for stochastic MDPs, the Markov policy is strictly more expressive than the predetermined policy. The transition can be stochastic for various reasons. For example, if the LLM uses external search engine, the next state $1+1 depends not only on the current tokens (I, V1.A) but also on the text generated by the external search engine x r-( zt B1:1), making it stochastic. Moreover, RLHF may have applications in other scenarios, such as robotics (Christiano et al., 2017), where the transition kernel is stochastic. To clarify, we distinguish these two types of policies in the following proposition.
Proposition C.2. There exists an MDP such that the value of any predetermined policy is at least 0.5 less than that of optimal Markov/autoregresst/ policy.
Proof. We construct an MDP M with state space S = {so, 81 82), action space A {(1, horizon H = 2, fixed initial state so. The reward r and transition kernel P are given by
r(si, aj) = =(} P(si 80: az) P(s2 s0, a;) = 0.5, V(i,)) (0, (0, (1,2)
It is straightforward that the optimal autoregressive policy achieves value of 1. In contrast, any predetermined policy only achieves value of 0.5. This completes the proof.
D. Implementation Details
Training Pipeline Our experiments start with an open-source SFT model OpenRLHF/Llama-3-88 htt.-lllmmm-----------tttttt For baseline PPO, we first train a reward model from this SFT model, then use it as both the reward function and the initialization of critic, following the standard practice. For other baselines, we directly train from this SFT model. For RTO, also train an IB reward model, initialized with Llama-3.2 3.2- I-Innstruc. Again, this tiny reward model functions both part of the RTO reward function, and the initialization of RTO critic. All preference learning uses binarized version of the UltraFeedback dataset, while all reinforcement learning uses a prompt-only version.
Training hyperparameters We use Adam optimizer (Kingma & Ba. 2017) across all experiments with varying learning rates, (0.9, 0.95) betas and no weight decay. We apply a cosine learning rate schedule with 3% warming steps and 10% minimum learning rate, All experiments use a single epoch, since we do not observe much gains from further training, Additionally, we set the max sequence length to 2048. We include all other method -spccffie hyperparameters in the tables below.
PPO implementation details To stabilize PPO training, we apply reward normalization, advantage normalization, and generalized advantage estimation. We use larger learning rate for critic, and use a similar clipped surrogate objective for critic learning. These tricks are implemented by the OpenRLHF (Hu et al., 2024) repo
Reward Model (UltraFeedback)
DPO (UltraFeedback)
Learning Rate
le-6
Learning Rate
5e-7
Batch Size
128
Batch Size
256
Maximum Sequence Length
2048
Maximum Sequence Length
2048
KL Coefficient (8)
0.1
Benchmark decoding hyperparameters For AlpacaEvall 2, we sample with temperature 0.7 and max generation length 4096. For Arena-Hard Hard, we use the default greedy decoding. Both settings apply for all methods.
Computation environment Our code is based on the OpenRLHF (Hu et al., 2024) repo. Our experiments is conducted on 8 80G A100 GPUs We apply techniques like Adam offload, gradient accumulation, and gradient checkpointing to reduce memory footprint and enable full-scale RLHE.
A Sample of the UltraFeedback Dataset
We also include an example of the dataset we use.
19

---

DPO Meets PPO: Reinforced Token Optimization for RLHI
RTO (Ultraeeedback)
PPO (Ulrafeedbackk
Actor Learning Rate
5e-7
Actor Learning Rate
8e-7
Critic Learning Rate
9c-6
Critic Learning Rate
9e-6
Batch Size
128
Batch Size
128
Maximum Prompt Length
1024
Maximum Prompt Length
1024
Maximum Response Length
1024
Maximum Response Length
1024
PPO Update Step
8
PPO Update Step
PPO Clip Coefficient
0.2
PPO Clip Coefficient e
0.2
GAE A
0.95
GAE À
0.95
DPO Reward Rescale (31))
0.05
KL Coefficient (3)
0.01
KL Coefficient (32)
0.01
Reward Rescale (3g)
1
Baselines (UltraFeedbackk
Learning Rate
Se-7
Batch Size
256
Maximum Sequence Length
2048
R-DPO KL Coefficient (8)
0.1
R-DPO Length Penalty (a)
0.001
SimPO Scale (8)
2.5
SimPO Margin (7)
0.5
TDPO KI. Coefficient (3)
0.1
TDPO KL Penalty Rescale (a)
0.2
Prompt:
Which animal has two hands, a hyrax or dog?
Chosen response: Neither hyrax dog has hands. Hyraxes have four legs with feet that adapted for climbing and gripping. Similarly, dogs also have four legs with paws. The term "hands" is usually reserved for primates, such as humans, which have opposable thumbs and fingers.
Rejected response: Thank you for your question. I'm happy to help you with that! However, must point out that the question itself may not be factually coherent. A hyrax is small, terrestrial mammal that belongs to the family Procaviidae, while a dog domesticated carnivorous mammal that belongs to the family Canidae. Neither of these animals have two hands.
Instead, I suggest rephrasing the question to ensure it is factually accurate. For example, "Which ani- mal has four legs, hyrax or dog?" would be a more appropriate and factual question.
Please let me know if there's anything else can assist you with!
E. Additional Experiments on REINFORCE-typ Algorithm
Our proposed dense reward demonstrates broad applicability beyond PPO and functions independently of the specific reinforcement learning algorithm used in RLHF. To demonstrate this versatility, evaluated RTO with alternative REIe algorithm (Williams, 1992), specifically REINFORCE+ (RPP; Hu, 2025). Unlike PPO, RPP does not use a critic network and relies solely on vanilla discounted returns without any value baseline. We include our hyperparameter selections below.
As shown in Table 3, we observe that: (a) the idea using token- wise reward RTO remains highly effective when applied RPP, supporting our claim; and (b) RPP performs worse than PPO, especially complex scenarios, potentially due the
20

---

DPO Meets PPO: Reinforced Token Optimization for RLHF
Method
Metric
SFT
DPO
PPO
RTO (PPO)
RPP
RTO (RPP)
AE (LC)
13.22
17.40
19.47
27.00
18.28
24.71
AE (WR)
8.58
12.23
12.89
22,45
13.91
23.11
AH (SC)
9.2
13.2
16.2
20.3
13.4
18.8
AH (WR)
8.9
13.8
15.6
21.4
15.4
21.8
Table 3. AlpacaFval 2 (AE) and Arena-Hard (AH) results.
critic in PPO capturing fine-grained information that aids learning.
RTO (RPP)
RPP
Actor Learning Rate
5e-7
Actor Learning Rate
5e-7
Batch Size
128
Batch Size
128
Maximum Prompt Length
1024
Maximum Prompt Length
1024
Maximum Response Length
1024
Maximum Response Length
1024
PPO Update Step
8
PPO Update Step
PPO Clip Coefficient
0.2
PPO Clip Coefficient e
0.2
GAE A
0.95
GAE A
0.95
DPO Reward Rescale (31)
0.05
KL Coefficient (3)
0.01
KL Coefficient (3,)
0.01
Reward Rescale (3²)
1
F. Additional Experiments on Summarization Task
F.1 Experimental Setup
Tasks and Data. We consider the Summarization task (Völske et al., 2017), where the model is required to generate a concise summary for a given post from the Reddit forum. Specifically, we fine-tune the foundational model using the Reddit TL:DR summarization dataset (Völske et al., 2017), where each data point comprises a post z and its corresponding summary y. Subsequently,
align the model with human preferences using its preference version, where each data point comprises post and two summaries, with preferences annotated by humans. To facilitate readers, provide examples of the TL:DR datasets in Appendix F.2. We employ the open-sourced Pythia-2.8B model (Biderman al., 2023) as the backbone for this task.
Evaluation. We primarily assess the aligmment performance of various methods using GPT-4. The GPT-4 evaluation harnesses the capabilities of GPT-4 itself and has been shown to align well with human evaluations (Rafailov et al., 2023). For the same prompt, we provide GPT-4 with two responses generated by two different models and ask it to determine which one is superior. We then calculate the win rates, following (Rafailov et al., 2023). The prompts for GPT-4 evaluation are presented Appendix F.4. For each GPT-4 evaluation, we use 100 samples.
Win Rates. Table 4 presents the performance of our method on the TL:DR dataset. We can see that the model trained by RTO outperforms all other baselines. Specifically, we achieve a win rate of 61% over the DPO algorithm evaluated by GPT-4. This illustrates the effectiveness of the RTO algorithm in a real-world text summarization task. All these empirical findings demonstrate the token-wise reward mechanism's advantage in improving model performance.
Ablation Studies on Temperatures. We further compare the model trained by RTO to other baselines on both datasets across different temperatures. In Figure 4, we present the results for these methods as we vary the temperature. We can observe that the RTO model consistently demonstrates superior performance compared to other baselines, highlighting its robustness across different temperatures.
21

---

DPO Meets PPO: Reinforced Token Optimization for RLHI
Win Rate
RTO
DPO
SFT
PPO
DPPO
RTO
0.50
0.61
0.67
0.67
0.67
DPO
0.39
0,50
0.58
0.59
0.50
SFT
0.33
0.42
0.50
0.59
0.49
PPO
0.33
0.41
0.41
0.50
0.40
DPPO
0.33
0.50
0.51
0.60
0.50
Table Win rates between each pair of models evaluated by GPT-4. The value in line column represents the win rate of the model in row against the model in column j.
Summarization v.s.
DPO
PPO
0.45
Win
0.35
0.30
0.25
0.0
0.2
0.4
0.5
0.8
1.0
Figure
Win rates across different sampling temperatures
Optimization Process Curves. To further investigate the benefits of the token-wise reward mechanism in the optimization process, compare the estimated reward during the training period in Figure 5. In this figure, the x-axis represents the training iterations. The y-axis represents the reward given by the implicit reward model derived from the DPO model (the reward model used in training) per batch. As we can see, in one epoch (roughly corresponds to 240 PPO training iterations in Figure 5), the reward the model trained by RTO on TL;DR can achieve about 0.4, while the reward of the model trained by DPPO is roughly -0.2. The results demonstrate that the wise reward mechanism significantly enhances the training process, leading to remarkably higher reward.
22

---

DPO Meets PPO: Reinforced Token Optimization for RLHF
RTO
DPPO
0.4
Reward
0.0
-0.2
50
100
150
200
Iterations
Figure The reward curve of DPPO and RTO during training. The reward given by implicit reward model / log Tapo (x)
Frat s))
optimized by DPO. The x-axis represents the training steps, and the y-axis represents the reward values.
F.2. Examples of Datasets
Prompt:
SUBREDDIT: r/AskReddit
TITLE: Reddit, what event drove you to cry in the bathroom at work?
POST: Yesterday, finally became that girl who goes into the bathroom to cry while at work.
I work a domestic violence shelter, and normally I'm pretty capable of brushing things off. I'm somewhat ashamed to say that was secondhand truama that led me to weep in the bathroom stall like a little girl, but my coworkers. It had been rough day, which pretty normal around here, but it tolerable level of rougness. My patience was wearing thin and I just wanted to go to the support group for advocates and take breather.
Unfortunately, my coworker decided at that time to demand that I clean one of the recently vacated rooms. Not just clean it, but DEEP clean it. I'm not talking weekend-away type clean. mean, she wanted hospital-grade clean. She wanted be able perform surgery on any surface of that room. The checklist she gave me- handwritten of course- had least thirty tasks on it. For dorm-sized room.
I lost it, guys. just completely lost my shit. I told her that would be happy to help clean that room, but she was absolutely off her rocker she thought was going to spend the next four hours cleaning by myself. She was incensed at my apparent refusal, and though tried to reiterate that would do it, but not alone, she started screaming for the lead advocate to put me my place.
Well, the lead advocate just didn't want to deal with the situation and told just do it. was absolutely frustrated, appalled, and overwhelmed. And so...I went into the bathroom and cried. Then I went and cleaned the stupid room.
TL:DR:
Chosen response: was stressed, my lazy coworker demanded clean every speck dust from room alone, I lost my shit, my supervisor sided with my coworker.
Rejected response: Coworker thinks it's okay to ask me to clean room she thinks is dumpster, so cried. Then cleaned it.
23

---

DPO Meets PPO: Reinforced Token Optimization for RLHI
F.3. Training Configurations of TL;DR
We provide the training configuration SFT, DPO, PPO, DPPO, and RTO below. In the table of the training configuration of the standard PPO algorithm, we also present the configuration of training the reward model used in the PPO algorithm.
SFT (TL,DR)
DPO (TL:DR)
Optimizer
AdamW
Optimizer
AdamW
Learning Rate
le-5
Learning Rate
5e-6
Batch Size
32
KL Coefficient (9)
0.1
Epochs 1
Batch Size
32
Epochs
Table 5. Configurations for supervise fine-tuning.
Table Configurations for DPO.
RTO and DPPO (TL;DR)
PPO (TL:DR)
Optimizer
Adam
Optimizer (PPO)
Adam
Learning Rate
3e-6
Optimizer (Reward Model)
A
Training Epochs
Mini Batch Size in PPO
16
Mini Batch Size in PPO
16
Init KI. Coefficient (3)
0.03
DPO KI. Coefficient 81
0.1
Learning Rate (PPO)
3e-6
Init KL Coefficient B2 (RTO)
0.05
Learning Rate (Reward Model)
3e-6
Init KL Coefficient 32 (DPPO)
0.05
Batch Size Per PPO Iteration
256
Batch Size Per PPO Iteration
256
Epochs of PPO Update Per Iteration
2
Maximum Sequence Length
512
Batch Size (Reward Model)
128
Epochs of PPO Update Per Iteration
2
Training Epochs (PPO and Reward Model)
1
Maximum Sequence Length
512
Table 8. Contigurations for RTO and DPPO.
Table Configurations for standard PPO. We also present the configura- tion of training the reward model used in the PPO algorithm in this table.
F4. Evaluation Details
Following the previous work (Rafailov et al., 2023), for evaluations utilizing GPT-4, completions are sampled by top-p sampling method with temperature of 7 = 0.9 and p 0.99 for 100 prompts. To mitigate any positional bias inherent in GPT-4's responses, we ensure that the order completions within each pair is randomized. The version the GPT-4 we used is GPT-4-0613, and the specific prompt utilized for GPT-4 evaluation is detailed as follows.
Prompt for GPT-4 evaluation in summarization task.
Which of the following summaries does a better job of summarizing the most important points the given forum post, without including unimportant or irrelevant details? A good summary is both precise and concise.
Post: <the forum post>
Summary A: <either the test method or baseline>
Summary B: <the other summariza zat ion> FIRST provide a one-sentence comparison of the two summaries, explaining which you prefer and why. SECOND, on new line, state only "A" or "B" to indicate your choice. Your response should use the format:
Comparison:
cone-sencent
comparison and explanation:
Comparison:
Preferred: <"A" or "B">
Table 9, Prompt for GPT-4 evaluation in summarization task.
24


Reviews:
[{"review_id": "P42hPwbiKa", "reviewer": "Reviewer_M1p9", "summary": "The paper presents RTO, a novel reward learning method for tuning LLMs from preference data. Such a process involves two stages: reward modeling and an RL step (typically PPO).  The main novelty of this paper lies in modifying the Bradley-Terry (BT) loss in the reward modeling step to yield a reshaped token-level reward instead of a single per-example reward. This allows the user authors to establish a more direct connection to the second RL step. This results in theoretical results for the regret under the linear reward assumptions. A nice feature of their theoretical approach is that it explicitly considers the KL-regularized version of RL used for LLM fine-tuning, but not so common in general RL.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper. We summarize your questions and address them as follows.\n\n**Q1:** MDP fomulation and Previous Theoretical Work\n\n**A1:**  We agree with the reviewer that the existing PPO is implemented in an *MDP with zero reward for all but the last token*, but we would like to claim that this is *essentially a bandit problem*. This point is explicitly stated in Section 3.5 of the seminal InstructGPT paper [1]. In contrast, a token-level MDP with dense rewards is a real multi-step decision-making problem (see Proposition 3.2 for a theoretical gap).\n\nWe will emphasize the importance of KL constraint in claim (1), to different from existing theoretical work on dueling RL. Moreover, we would like to emphasize that these works are purely theoretical works and does not lead to corresponding paractical algorithm leveraging dense rewards.\n\nFinally, the token-level MDP terminology is mainly used to emphasize the use of dense rewards, in contrast to the bandit formulation with sparse rewards, which aligns with the reviewer's understanding. This token-level MDP was rigorously introduced first in our paper, and it is well recognized by the community [2]. Therefore, we prefer to maintain the terminology of token-level MDPs with KL constraints while highlighting the differences from existing bandit formulations and theoretical dueling RL work.\n\n[1] Training language models to follow instructions with human feedback\n\n[2] Value-incentivized preference optimization: A unified approach to online and offline rlhf\n\n**Q2:** MLE in (4.1) at the trajectory level\n\n**A2:** The MLE objective in (4.1) looks like at the trajectory level because the preference signal is usually given in the trajectory level and the cumulative token-level reward is the trajectory-level reward. However, even the trajectory-based preference reward can naturally induce a cumulative token-level reward. This is more evident for problems with step-wise structure like mathematical reasoning [3].\n\n[3] Entropy-Regularized Process Reward Model\n\n**Q3:** Theorem 4.2 depends on $\\lambda$ and the role of $\\lambda$\n\n**A3:** The parameter $\\lambda$ is just a regularization parameter for theory (ensure $\\Sigma_{\\mathcal{D}}$ has inverse), and our Theorem 4.2 holds for any $\\lambda>0$. You can simply regard $\\lambda = 1$.\n\n**Q4:** Verify Assumption 4.1 by construction\n\n**A4:** If we choose $\\phi(s, a) = 1_{(s, a)}$ as the one-hot vector and $\\theta_{(s, a)}=r(s, a)$,  then the assumption is satisfied with $d = |\\mathcal{S}| \\times |\\mathcal{A}|$. However, the reward function may exhibits some low-dimensional respresentation $\\phi \\in \\mathbb{R}^d$ with $d < |\\mathcal{S}| \\times |\\mathcal{A}|$, and thus we assume Assumption 4.1 with potentially small $d$ and do not make the explicit construction. This is also the main motivation of linear bandit and linear MDP. \n\n**Q5:** Experiments: (1) limited base model and baselines. (2) application to other RL algorithms; (3) hyperparameter $\\beta_3$?\n\n**A5:** (1) Regarding baseline selection, we focused on the most relevant (DPO, PPO), strong (R-DPO, SimPO), and concurrent (SimPO, TDPO) methods. Given the limited performance of numerous other alignment algorithms reported in SimPO, we opted not to include them in our comparisons. Due to time limit, we are still trying to add experiments on other base models. We appreciate your understanding.\n(2) We extend our experiments to REINFORCE++ (RF++) [5], an alternative RL algorithm. We selected RF++ since\n- Unlike reasoning tasks, chat alignment typically does not involve sampling multiple responses per prompt like RLOO and GRPO.\n- RF++ exhibits significant differences from PPO, notably the absence of a critic model.\n\nThe table below shows the power of RTO when applied to RF++.\n\n| Algorithm | AE2 LC | AE2 WR |\n| -------- | -------- | -------- |\n| RF++ | 18.28 | 13.91 |\n| RTO | 24.71 | 23.11 |\n\n[4] REINFORCE++: A Simple and Efficient Approach for Aligning Large Language Models\n\n(3)  Since only relative ratio matters and to minimize tuning, we fix $\\beta_2$ the same value PPO uses, choose $\\beta_3 = 1$, and tune $\\beta_1$ (see discussion below (4.7)). In our internal experiments we found our algorithm are quite robust to the choice of $\\beta_1$. Thus we believe that conversely, RTO is not sensitive to the choice of $\\beta_3$. \n\n**Q6:** Reward shaping\n\n**A6:** Our algorithm can be interpreted as a form of potential-based reward shaping, with $F(s, s' = (s, a)) = \\Phi(s') - \\Phi(s) = \\log \\frac{\\pi(s')}{\\pi_{\\mathrm{ref}}(s')}-\\log \\frac{\\pi(s)}{\\pi_{\\mathrm{ref}}(s)}=\\log\\frac{\\pi(a|s)}{\\pi_{\\mathrm{ref}}(a|s)}$, where $\\Phi$ is the potential function and $F(s, s' = (s, a))$ denotes the token-wise reward function added to each token. Thank you for the insightful question and we will emphasize this in the revision.\n\nRegarding your minor suggestion, we would emphasize the issue of the reward modeling type directly, rather than mentioning the open-source implementation.", "claims_and_evidence": "### Claims\n\nThe authors main claims are that (1) they propose a framework for RLHF as an MDP; (2) demonstrate near optimal sample efficiency under this model; (3) achieve improvements in well-established benchmarks under the new reward model.\n\n\n### Evidence\n\n**Regarding (1)**\n\nI think the contribution should be made more precise and I found it very confusing at first read.\n\n It is true that the reward modeling step assumes a bandit structure, but this step is only for the reward learning stage, and the bandit structure itself is quite irrelevant. The RL (PPO) step in current RLHF is already doing RL over a token-level MDP by assigning a reward per token (with zero reward but the last token). \n\nI also question that the current proposed reward modeling step is framed as an MDP. This is so because the MLE objective in (4.1) is still at the trajectory level. There is simply no way to mathematically define it for a partial trajectory. If it was truly at the transition level, I could apply it over a random sample of transitions. But their framework still requires to randomly sample over trajectoreis. Could the authors comment ton that?\n\nAs another comment, the authors themselves state that:\n\n> ““There have also been previous works (Pacchiano et al., 2021; Chen et al., 2022; Wang et al., 2023; Li et al., 2023c; Zhan et al., 2023a) studying RLHF under the MDP framework, also known as dueling RL and preference-based RL. However, these works do not consider the KL constraint, which is an essential component of RLHF.”\n\nHence, the initial claim in (1) should be made consistent with this statement.\nUltimately, I think this paper should be framed more simply as a a reward modeling strategy for RLHF or a reward shaping strategy.\n\n**Regarding (2)**\n\n* I think this statement is slightly imprecise since the optimality bound of Theorem 4.2 depends on a tuning parameter lambda which is not part of the reward model r.\n\n* I also don’t understand very well the value of Assumption 4.1 (Linear reward) since the authors proposing exactly how to fit the reward model under the proposed modified BT/DPO at token-level. Shouldn’t the assumption be true or false by construction? \n\n**Regarding (3)**\n\nThe experiments look promising, but they are also quite limited since they only apply to baseline Llama 8B and a relative small number of baselines. See my comment on the experiments section.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "I skimmed over the theoretical derivation, but did not check them carefully. \n\nNote that $\\xi$ in Assumption 3.1 is never defined. What is it? Can you define it formally and provide an intuition since it is so import to the claim of sample complexity optimality?", "experimental_designs_or_analyses": "Some additional experiments could improve the practical value of the method:\n\n1.\tExperiment with additional base LLMs. Relying solely on Llama 8b is very limiting. Ideally there would more baseline models and a larger LLM. \n2.\tDoes the performance gains also hold if an alternative RL method for step 2 is applied (e.g. RLOO [1]). PPO is probably not the state of the art.\n3.\tHow sensitive is the performance to the hyperparameter $\\beta_3$?\n\n\n[1] Ahmadian, et al. (2024). Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXiv:2402.14740, 2024.", "relation_to_broader_scientific_literature": "The paper is very timely given the centrality of LLM finetuning in modern AI methods across various fields.", "essential_references_not_discussed": "As far as I am aware, the paper conducts a comprehensive review.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "*Minor suggestion*:  I was a bit thrown off by the phrase “After examining the open-source implementation of PPO” in the introduction. I eventually understood the intention of the authors. But first of all, I don’t know what open source has to do with it. Second, PPO is not an ambiguous algorithm; in fact, the authors are implementing it in their paper without modification. The real issue is the mismatch between the “sentence-level reward modeling stage” and the required token-level reward in PPO, which results in the potentially inefficient reward assignment to the last token only.", "questions_for_authors": "I have already included my questions in the other sections. \n\nAdditional questions:\n\n1.\tHow does the role of $\\lambda$ hyperparameter in the theoretical version carries to the practical version?\n2.\tIf the main value of the framework is the reward shaping, could I obtain a similar performance by applying reward shaping techniques (e.g., potential based) to the sentence-level reward?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lQBGDSdbsg", "reviewer": "Reviewer_wuwo", "summary": "The paper introduces Reinforced Token Optimization (RTO), a framework that integrates Direct Preference Optimization (DPO) and Proximal Policy Optimization (PPO) to improve Reinforcement Learning from Human Feedback (RLHF). The authors argue that existing RLHF implementations using PPO underperform due to a mismatch between sentence-level reward modeling (bandit formulation) and PPO’s requirement for token-wise rewards. RTO addresses this by reformulating RLHF as a Markov Decision Process (MDP) with token-level rewards. Key contributions include:\n\n- MDP formulation\n- RTO algorithm\n- Theoretical guarantees\n- Empirical results", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper. Below are our responses.\n\n**Question Regarding Theory:** Theoretical Proofs: Could you provide a complete proof for the sample efficiency claim in Appendix B?\n\n**Response:** We have provided complete and rigorous proofs for both Proposition 3.2 and Theorem 4.2 in Appendix B. Should you have any specific questions or require further clarification, we are happy to address them. Regarding our extension mentioned in Remark B.3, we have acknowledged that this is a standard result in RL literature (see e.g., Theorem 2 in [1] for a detailed proof). Given that this standard extension is not central to the main claims of our paper, we opted to provide a concrete reference rather than including a detailed proof. We appreciate your understanding.\n\n[1] Fast Global Convergence of Natural Policy Gradient Methods\nwith Entropy Regularization. https://arxiv.org/pdf/2007.06558\n\n\n**Question Regarding Experiments:**(1) DPO’s Role: How does DPO-derived token-wise reward correlate with human preferences? Is there empirical validation beyond benchmark scores? (2) Baseline Details: Were PPO and DPO baselines trained with identical hyperparameters (e.g., KL penalty, reward scaling)? (3) Data efficiency: The claim that RTO achieves PPO-level performance with 1/8 of the data is compelling but requires validation across multiple seeds.\n\n**Response:** Thank you for your question. \n\n(1): We have added the following results to illustrate that DPO-derived token-wise rewards effectively capture human preferences. For the pairwise data, we measure consistency between human choices and DPO reward choices (reward implies an order). We observed a 79.23% consistency in the training data and 72.72% in the test data. Additionally, we want to emphasize that the performance of DPO-derived token-wise rewards used in reinforcement learning training, evaluated by widely recognized benchmarks, is a key standard for assessing its quality and is also the central focus of our work. To further illustrate the generalizability of DPO-derived reward, we've extended our methods to another RL algorithm REINFORCE++ [1]. \n\nThe following table presents the Alpaca Eval 2 benchmark scores, further demonstrating the ability of DPO rewards to effectively capture human preferences and the broader applicability of our RTO method.\n\n| Algorithm | AE2 LC | AE2 WR |\n| -------- | -------- | -------- |\n| REINFORCE++ | 18.28 | 13.91 |\n| RTO (with REINFORCE++ objective) | 24.71 | 23.11 |\n\n[1] REINFORCE++: A Simple and Efficient Approach for Aligning Large Language Models\n\n\n(2): We have included the detailed hyperparameter choices in Appendix E. The DPO and PPO hyperparameters differ, and both sets were derived from well-tuned hyperparameters provided by the OpenRLHF project. For your convenience, we present important hyperparameters here: The KL penalty coefficient ($\\beta$) is $0.01$ for PPO and $0.1$ for DPO. Other PPO hyperparameters are clip coefficient $\\epsilon=0.2$ and GAE $\\lambda=0.95$. We train a reward model for PPO, and another tiny reward model for RTO ($r_\\text{MLE}$) following the standard practice. The reward scaling of PPO is $1$ for reward model and $\\beta=0.01$ for KL reward penalty. For RTO, we use $\\beta_3=1$ for reward model, $\\beta_2=0.01$ for KL reward penalty, and $\\beta_1=0.05$ for DPO reward.\n\n\n(3): Thanks for the suggestions. Our implementation is based on OpenRLHF and our (also the community's) experience (across hundrends of LLM RL training in either preference learning or math/coding task) is that with the well-tuned recipe, the LLM RL is very stable and not greatly affected by the choice of random seed. As a result, we believe our findings are sufficient to demonstrate the sample efficiency of our claim.", "claims_and_evidence": "The claims are supported by theoretical analysis and empirical validation:\n\n- MDP superiority: Theoretically justified via comparisons between bandit and MDP formulations (Section 3.3).\n- RTO’s effectiveness: Experiments on standard benchmarks (AlpacaEval 2, Arena-Hard) and ablation studies (e.g., data scaling) validate performance gains.\n- Token-wise reward extraction: The use of DPO to derive token-level rewards (Appendix D.1) is plausible but requires deeper scrutiny (see Q4/Q5).", "methods_and_evaluation_criteria": "- Methods: RTO’s integration of DPO and PPO is novel and addresses the token-wise reward gap in RLHF. The MDP formulation aligns with LLMs’ autoregressive nature.\n- Evaluation: Benchmarks (AlpacaEval 2, Arena-Hard) are standard for LLM alignment. However, details on baseline implementations (e.g., PPO hyperparameters) and statistical significance are unclear.", "theoretical_claims": "The paper claims RTO’s sample efficiency (Section 4) but lacks formal proofs. The theoretical analysis in Section 3.3 (MDP vs. bandit) is intuitive but not rigorously proven. Appendix B mentions a \"near-optimal policy\" guarantee but does not provide a full proof.", "experimental_designs_or_analyses": "- Baselines: Comparisons with PPO, DPO, R-DPO, and SimPO are reasonable, but implementation details (e.g., reward models, KL penalties) are sparse.\n- Data efficiency: The claim that RTO achieves PPO-level performance with 1/8 of the data is compelling but requires validation across multiple seeds.\n- Ablation studies: Limited analysis of RTO’s components (e.g., DPO’s role in token-wise rewards).", "relation_to_broader_scientific_literature": "The work builds on RLHF (Ziegler et al., 2019), PPO (Schulman et al., 2017), and DPO (Rafailov et al., 2023). It advances the field by addressing the token-level reward sparsity problem, a known limitation of PPO in RLHF.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Strengths: \n\n- Novel integration of DPO and PPO; \n- strong empirical results; \n- addressing a critical RLHF limitation.\n\nWeaknesses: \n- Theoretical gaps; \n- limited ablation studies; \n- insufficient implementation details for reproducibility.", "other_comments_or_suggestions": "None", "questions_for_authors": "1. Theoretical Proofs: Could you provide a complete proof for the sample efficiency claim in Appendix B?\n\n2. DPO’s Role: How does DPO-derived token-wise reward correlate with human preferences? Is there empirical validation beyond benchmark scores?\n\n3. Baseline Details: Were PPO and DPO baselines trained with identical hyperparameters (e.g., KL penalty, reward scaling)?\n\nResponses to these questions could strengthen the paper’s theoretical grounding and empirical rigor.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ySywgKhdci", "reviewer": "Reviewer_J8vS", "summary": "Summary: The authors propose a token-level MDP formulation for LLM. post-training. They use the token-level action probabilities from a Direct Preference Optimization (DPO) trained LLM. Authors argue that the current formulation of LLM post-training is closer to a contextual bandit than it is to a reinforcement learning, and hence does not utilize the full power of the RL machinery, i.e. credit assignment, advantage function etc. Further, authors propose a practical implementation of their method: RTO, where practitioners can directly plug-in DPO token-level probability estimates in a standard PPO implementation.\n\n## update after rebuttal\n\nAll reviewers seem to agree that this is an interesting paper for RL fine-tuning of LLM. My initial assessment of accept has not changed.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and support. Below are our response to your questions.\n\n**Q1:** I have a question on the results. SimPO method outperforms PPO by a large margin (Table 1). Do you have any intuition for this? \n\n**A1:** Our observation is that the RL-free methods are good at fitting the **style** over the deep RL methods, while the human/GPT-4 preference are easily hacked by the preference bias (see Figure 3 of [1] and related discussions). Compared to the DPO, SimPO further removes the KL constraint so it is more easily to achieve a higher in-domain score (AlpacaEval). However, we also notice that SimPO will also often hurt the OOD reasoning performance (see Table 9 of their paper for example). \n\n[1] From Lists to Emojis: How Format Bias Affects Model Alignment\n\n**Q2:** Literature seem to suggest that RL-free offline methods are outperformed by online methods, such as PPO, but the results in this paper seem to suggest differently.\n\n**A2:** Indeed, almost all closed-source state-of-the-art LLMs, including ChatGPT, GPT-4, and the recent DeepSeek R1, are trained using RL-based methods like PPO and GRPO. However, the open-source community has struggled to replicate these RL training techniques effectively. Our work aims to narrow this gap by exploring both dense and sparse reward approaches.", "claims_and_evidence": "The claims made by the authors, i.e. a token-level reward is better than sentence-level reward for LLM post-training is well supported by the experimental claims. Authors compare with both “RL-free” methods like (DPO, SimPO), and the standard online RL-method (PPO).", "methods_and_evaluation_criteria": "The proposed method is a very sensible extension of the existing sentence-level PPO fine-tuning, and the experimental setup is sound.", "theoretical_claims": "I have checked the theoretical claims: Proposition 3.2 (sentence-level post-training has a much higher sample complexity than token-level post-training), Theorem 4.2 (Sub-optimality of token-level rewards), Eq. 4.7 (practical version of token-level rewards), and they all make sense to me.", "experimental_designs_or_analyses": "The experimental design and the analysis of the proposed method RTO, including the sample efficiency, reward granularity are sound.", "relation_to_broader_scientific_literature": "The paper improves the post-training method for LLMs, and given the significance of LLMs recently, the presented method can have a broad impact.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "1. The paper is very well-written, especially the introduction and discussion of preliminary work. In my experience, most papers do not explicitly spell out that the most of the current setup for LLM post-training is in fact a contextual bandit setup, and not a MDP setup, so it's nice to see this assumption explicitly spelled out. \n\n2. In section 3.3., authors prove that the sentence-level MDP formulation is sample inefficient in terms of sample complexity, and a token-level formulation is required. \n\n3. The paper has a nice mix of both solid theoretical analysis, and practical algorithm for implementation. In my experience, some papers (on this topic) are either purely empirical, or purely theoretical. Nothing wrong with it, but it's nice to see a paper with a good mix of both.", "other_comments_or_suggestions": "NA", "questions_for_authors": "1. I have a question on the results. SimPO method outperforms PPO by a large margin (Table 1). Do you have any intuition for this? Literature seem to suggest that RL-free offline methods are outperformed by online methods, such as PPO, but the results in this paper seem to suggest differently. \n\n2. Is Assumption 4.1 valid for only linear reward models, or does it also apply for LLM-based reward model (with \\pi fixed, and \\theta parameter)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VzZDs5sOxh", "reviewer": "Reviewer_E8vv", "summary": "This paper develops an RLHF framework with a fine-grained token-wise reward characterization. Specifically, they model RLHF as an MDP, offering a more precise token-wise characterization of the LLM’s generation process. They introduce RTO algorithm, which extracts token-wise reward signals from offline preference data and subsequently performs RL training with respect to the learned token-wise rewards. In practice, they will introduce a practical implementation of RTO, which uses a token-wise reward extraction approach from DPO.\n\n## update after rebuttal: Thanks for the rebuttal. I do not have any further question. I will keep my original rating.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and support. Below are our response to your questions.\n\n**Q1:** Assumption 3.1 is not a rigorous assumption. The parameters $A$ and \n$\\xi$ have not been defined. What are their possible ranges of values? The current statement is not an assumption without conditions $A$ and $\\xi$. In addition, it is helpful to add more discussions on the intuition of this assumption and why it makes sense in the considered problem.\n\n**A1:** The parameter $A = |\\mathcal{A}|$ is the action set size (Line 220), and $\\xi$ is a constant introduced in Assumption, to clarifiy this, we revise Assumption 3.1 as\n\nAssumption 3.1. There exists a response $y = y_{1:H}$ satisfying $\\pi^*(y|x) \\ge A^{-\\xi}$ for some $0 \\le \\xi \\le H$.\n\nWe have also stated in the paper that \"By the pigeon-hole principle, there must be a response $y$ such that $\\pi^*(y | x) \\ge A^{−H}$, implying that $\\xi ≤ H$ naturally. In practice, $\\xi$ is usually much smaller than $H$ because the language model tends to choose the optimal response rather than making a random guess.\"\n\nMore specifically, there are at most $A^H$ possible responses/trajectories $y = y_{1:H}$ (each step has $A$ actions (tokens) to choose from and the sequence length is at most $H$). Therefore, at least one response is chosen with a probability of at least $A^{-H}$. Moreover, given a prompt, the LLM will only generate several likely tokens with high probability, which makes the final generation probability of the most likely response significantly larger than the random guess probability $A^{-H}$. We denote this as $A^{-\\xi}$ with $\\xi \\ll H$.\n\n\n\n**Q2:** There is a gap between the Algorithm 1 (Theoretical Version) and Algorithm 2 (Practical Version). The theorems were proved for Algorithm 1, whole all experiments were done based on Algorithm 2. If Algorithm 1 is not implementable in practice, why to introduce it? Can you prove the suboptimal gap bound for the Algorithm 2 directly?\n\n**A2:** We introduce the practical version of the theorem primarily to demonstrate that *token-level MDPs with KL constraints can be learned sample-efficiently*. We believe this constitutes a new and important theoretical advancement that deserves emphasis. We also acknowledge that proving the suboptimality gap bound for Algorithm 2 directly is challenging; in fact, even for standard PPO, its theoretical analysis often requires simplifications, such as replacing the GAE estimation with an optimistic estimation. Thank you for your question, and we will highlight our motivation for introducing the theoretical version and discuss its current limitations.\n\n\n**Q3:** A few highly related key references were only mentioned in Section A of the Appendix. For example, Zeng et al. (2024) also considered token-level DPO. Can authors add more discussions on the difference and the technical novelty beyond this paper?\n\n**A3:** Sure. We will include more discussion in the revision. Meng et al. (2024) propose SimPO by modifying the DPO objective, replacing the reference model with response length, and adding a margin threshold. Zeng et al. (2024) also consider a token-level reward and leverage this insight to develop token-level DPO, which performs better than the original DPO. These are beyond the scope of our work. In contrast, we utilize the implicit token-level reward provided by the original DPO as the dense token-level reward for RL training.\n\n**Q4:** The authors used \"DDPO\" and \"DPPO\" in plots and discussions. Are they all different?\n\n**A4:** Thanks for pointing out the typo and we will correct them. Both \"DDPO\" and \"DPPO\" indicate the baseline that uses RTO reward delayed to the last token.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "See my comments in \"Questions For Authors\".", "experimental_designs_or_analyses": "The experiment looks good.", "relation_to_broader_scientific_literature": "See my comments in \"Questions For Authors\".", "essential_references_not_discussed": "See my comments in \"Questions For Authors\".", "other_strengths_and_weaknesses": "See my comments in \"Questions For Authors\".", "other_comments_or_suggestions": "There are typos in Section 5. One of the competitive methods is called token-wise DPO (TDPO). The authors used \"DDPO\" and \"DPPO\" in plots and discussions. Are they all different?", "questions_for_authors": "1. Assumption 3.1 is not a rigorous assumption. The parameters $A$ and $\\xi$ have not been defined. What are their possible ranges of values? The current statement is not an assumption without conditions on $A$ and $\\xi$. In addition, it is helpful to add more discussions on the intuition of this assumption and why it makes sense in the considered problem. \n\n2. There is a gap between the Algorithm 1 (Theoretical Version) and Algorithm 2 (Practical Version). The theorems were proved for Algorithm 1, whole all experiments were done based on Algorithm 2. If Algorithm 1 is not implementable in practice, why to introduce it? Can you prove the suboptimal gap bound for the Algorithm 2 directly?\n\n3. A few highly related key references were only mentioned in Section A of the Appendix. For example, Zeng et al. (2024) also considered token-level DPO. Can authors add more discussions on the difference and the technical novelty beyond this paper?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper aims to address the mismatch between sentence-level reward modeling in RLHF and the use of PPO by reformulating RLHF as an MDP with token-level rewards. The reviewers unanimously appreciated the paper for its clarity, the novelty of the proposed approach, and the significance of its contribution. However, they suggested that the authors clarify the precise contribution of the paper, particularly in distinguishing their method from the PPO step used in current RLHF pipelines and from related work in dueling RL. The authors are also encouraged to incorporate the reviewers’ comments on the experiments in their revised version.
Decision: Accept (spotlight poster)