From $r$ to $Q^*$: Your Language Model is Secretly a Q-Function

Authors: rafael  rafailov
joey  hejna
ryan  park
chelsea  finn
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-22 22:13:13

Abstract: Reinforcement Learning From Human Feedback (RLHF) has been a critical component of the success of the latest generation of generative AI models, including the GPT series. However, this is an involved and complex process and direct alignment algorithms, such as DPO have recently emerged as an alternative approach to the classical RLHF pipeline. Although DPO solves the same objective as the standard RLHF setup, there is a mismatch between the two approaches. Standard RLHF deploys reinforcement learning in a specific token-level MDP, while DPO is derived as a bandit problem in which the whole response of the model is treated as a single arm. In this work we rectify this difference, first we theoretically show that we can derive DPO in the token-level MDP as a general inverse Q-learning algorithm, which satisfies the Bellman equation. Using our theoretical results, we provide three concrete empirical insights. First, we show that because of its token level interpretation, DPO is able to perform some type of credit assignment. Next, we prove that under the token level formulation, classical search-based algorithms, such as MCTS, which have recently been applied to the language generation space, are equivalent to likelihood-based search on a DPO policy and empirically we show that a simple beam search yields meaningful improvement over the base DPO policy. Finally, we show how the choice of SFT policy causes implicit rewards to decline during training. We conclude by discussing applications of our work, including information elicitation in multi-tun dialogue, reasoning, agentic applications and end-to-end training of multi-model systems.

Main paper: 
 introduction: 
 Reinforcement Learning from Human Feedback (RLHF) has become the defacto method for aligning large language models (LLMs) with human intent due to its success in a wide range of applications from summarization (Stiennon et al., 2022) to instruction following (Ouyang et al., 2022) . By learning a reward function from human-labeled comparisons, RLHF is able to capture complex objectives that are in-describedable in practice. Following the success of (Ziegler et al., 2020) , numerous works have considered new algorithms for training and sampling from large models in various domains using techniques from reinforcement learning (RL). In particular direct alignment methods, such as Direct Preference Optimization (DPO) (Rafailov et al., 2023) have gained traction in recent months because of their simplicity (Zhao et al., 2023a; Azar et al., 2023) . Instead of learning a reward function and then using RL, direct alignment methods use the relationship between reward functions and policies in the contextual bandit setting to optimize both simultaneously. Similar ideas have since been applied to vision language (Zhao et al., 2023b) and image generation models (Lee et al., 2023) .While such direct alignment methods are purported to work the same as classical RLHF approaches that use policy gradient algorithms like PPO (Schulman et al., 2017) , fundamental differences remain. For instance, classical RLHF methods optimize token-level value functions with a sparse reward at the terminal state. DPO on the other hand, operates only in a contextual bandits setting, treating the entire response as a single arm. This is despite the fact that tokens are generated one at a time, and dense rewards are commonly known to be beneficial in the RL community. While direct alignment algorithms are interesting, at present it is unclear if they can be applied to sequences in the same way as the underlying RL algorithms used in typical RLHF pipelines.In this work we rectify this difference by deriving DPO within the token-level MDP setting present in large language models using the usual form of binary preference-feedback. We then show that DPO training implicitly learns a token-level reward function, for which the language models logits define the optimal Q function, or expected total future reward. We then demonstrate that DPO is able to flexibly model any possible dense reward function within the token MDP.Empirically, we use our theoretical derivations to justify three practical insights which we believe to be of use to the community. First, we show that despite being derived as a contextual bandit, the implicit rewards of a DPO model have a per-token interpretation. Second, we demonstrate that likelihood search over a DPO model is analogous to searching over a reward function during decoding as done by contemporary works (Liu et al., 2023b; Feng et al., 2024) . Finally, we identify the choice of initial policy and reference distribution as being important in determining the trajectory of implicit rewards during training. 
related work: 
 The problem of aligning policies with human intent using preference feedback has been a long studied problem in reinforcement learning (Akrour et al., 2011; Wilson et al., 2012) . While the primary focus of RLHF was originally in control (Christiano et al., 2017) , following the success of Ziegler et al. (2020) it has recently been broadly adopted by the language modeling (Ouyang et al., 2022; Nakano et al., 2021; Stiennon et al., 2022; Bai et al., 2022a) and even vision communities (Black et al., 2023a; Lee et al., 2023) . Most works in RLHF optimize a learned reward function, used only at the end of generation, with a policy graident stylemethod. Such approaches have been known to be unstable (Engstrom et al., 2020) and hard to scale, while at the same time theoretically existing at an unusual intersection between contextual bandits and RL. In response, several direct alignment methods (Rafailov et al., 2023; Azar et al., 2023; Zhao et al., 2023a) have been developed which simplify the RLHF pipeline by learning a policy from preference data without an intermediate reward function. Such methods however, derived solely as contextual bandits, leave several theoretical and practical questions unanswered which we seek to address.First, though direct alignment methods treat the LLM as a bandit, prior works have demonstrated that it is possible to use dense rewards Zelikman et al. (2022) ; Chan et al. (2024) ; Pan et al. (2023) or even approximate dynamic programming (Snell et al., 2022) . Moreover, using the regret-model of preferences (Knox et al., 2023; 2024) , Contrastive Preference Learning (Hejna et al., 2024) is able to use direct alignment for general MDPs, instead of the specific token MDP used in RLHF. Our work shows how DPO can be interpreted as optimizing a per-token reward function, which in practice is restricted to the family of optimal advantage functions.Second, if DPO does not learn a reward function, can we still use its reward or value? Prior works have considered using best-of-K (Mudgal et al., 2023) or tree search (Liu et al., 2023b) for alignment with a value function Kim et al. (2022) ; Li et al. (2017) or discriminator (Yang & Klein, 2021) . Using the implicit reward, we show that likelihood search results in a similar solution for direct alignment.Our work builds on foundational knowledge in maximum entropy RL (Ziebart, 2010) and inverse RL (Ziebart et al., 2008; Ng et al., 1999; Cao et al., 2021) . In particular, we leverage the mapping between Q-functions and reward functions under a fixed policy as first done in inverse RL by Garg et al. (2022) . Related to our work, Nachum et al. (2017) uses similar derivations for reinforcement learning in control and Watson et al. (2023) does so for inverse RL from demonstration. Hejna & Sadigh (2024) exploit this relationship for RLHF. While related, these works still require an additional loop of reinforcement learning optimization, which we dispose of in our formulation of feedback learning for LLMs. In the LLM domain, Yu et al. (2024) uses pre-trained models as priors for Q-learning, while Cundy & Ermon (2023) considers a similar formulation for imitation learning. In this work instead, we formulate preference-based learning as Q-learning. 
preliminaries: 
 In this section we first define the per-token MDP for large language models, and then describe how it relates to classic RLHF approaches and direct alignment algorithms, specifically DPO. We operate in the typical RLHF setting where we have a dataset D = {(x (i) , y (i) )} N i=1 of language prompts x and target answers y, which can each individually be broken down into a sequence of tokens, for example x = (x 0 , . . . , x m ), from a fixed discrete vocabulary A. Throughout this section we will use the x, y notation for the contextual bandit framing where the entire response y is the action, but will use state s and action a notation from RL literature for describing sequences at the token-level.We define the token level MDP as a tuple M = (S, A, f , r, ρ 0 ), where the state space S consists of all tokens generated so far (i.e. s t = {x 0 , . . . , x m , y 0 , . . . , y t }) and the action space is the vocabulary of tokens A. The dynamics f are the deterministic transition model between tokens f (s, a) = s|a, where | is concatenation. The initial state distribution ρ 0 is a distribution over prompts x, where an initial state s 0 is comprised of the tokens from x. In RLHF, the reward function is learned from human feedback over preferences between responses which we will denote using trajectories τ at the token level. As is typically done (Ziegler et al., 2020; Stiennon et al., 2022) , we assume that preference trajectories start at the same state (initial propmpt) and end in a terminal state (EOS token), from which future rewards are zero. In this token level MDP, the corresponding Bradley-Terry preference model Bradley & Terry (1952); Christiano et al. (2017) isEQUATION)which gives the probability that the "win" trajectory τ w of length N is preferred to the "loss" trajectory τ l of length M. Now that we have defined the token level MDP, we can show how it relates to both classic and direct alignment RLHF methods.Most classical RLHF approaches (Ziegler et al., 2020; Bai et al., 2022b; Ouyang et al., 2022) first learn a reward function from human feedback on prompt and response pairs (x, y w , y l ), then optimize it with a policy gradient-based method like PPO (Schulman et al., 2017) with an entropy-bonus using the following KL-constrained RL objective maxπ θ E a t ∼π θ (•|s t )    T ∑ t=0 (r(s t , a t ) + β log π ref (a t |s t ) KL penalty ) + βH(π θ )|s 0 ∼ ρ(s 0 )    (2)where π ref is a reference policy, often resulting from supervised finetuning, from which the learned policy should not significantly deviate. However, in classic RLHF methods the reward function is learned as a contextual bandit with the preference model p * (y w ⪰ y l ) = exp r(x, y w ) exp r(x, y w ) + exp r(x, y l ) and is thus only applied at the final timestep for the last action where a is EOS. In practice the actual reward used in the token-level PPO isr(s t , a t ) = β log π ref (a t |s t ), if s t+1 is not terminal r(x, y) + β log π ref (a t |s t ), if s t+1 = y is terminal (3)in a maximum entropy formulation. This leads to an interesting contradiction where the reward function r is treated like a bandit, but the actual RL value function and optimization is done per-token in practice.Unlike classical RLHF, DPO, as derived in Rafailov et al. (2023) , stays entirely within the contextual bandits setting entirely and also uses the bandit-based preference model in section 3.2. To circumvent the need for an RL algorithm, DPO uses the well-known closed form solution to the KL-contextual bandit version of the RL problem posed in eq. ( 2) (Ziebart et al., 2008; Levine, 2018) : ,y) where π * is the optimal policy and Z(x) is the partition function that normalizes it. DPO rearranges this equation to solve for reward as r(x, y)π * (y|x) = 1 Z(x) π ref (y|x)e r(x= β log π * (y|x) -β log π ref (y|x) -Z(x).Substituting this relationship into the standard binary cross-entropy loss function used for reward modeling yields the DPO loss equation as the partition function Z(x) cancels from the Bradley Terry model.EQUATIONFor brevity we use σ to denote the logistic function. In the next section, we show how an alternative derivation of DPO can also cast its optimization within the token-level MDP. 
theoretical insights: 
 In this section we explore how DPO can theoretically be cast into the token-level MDP, and explore the consequences of doing so. First, we provide a token level derivation of DPO under the assumptions in section 3.1. Next, we show that even in the token MDP, DPO is able to fit any reward function in the multi-step Bradley Terry preference model eq. ( 1).Ultimately, this shows that DPO can potentially be used for more sequential optimization tasks, like multi-turn interactions or even multi-modal generation.RL in the Token-level MDP. While the original derivation of DPO relies on the fact that Q * (x, y) = r(x, y), this relationship does not hold in the token-level MDP. To resolve this, we need to develop new mathematical results that will allow us to relate the reward function in the Token-level Bradley Terry model eq. ( 1) to the corresponding optimal policy π * . In the general maximum entropy RL setting, the fixed point solution of eq. ( 2) is given by (Ziebart, 2010) asπ * (a t |s t ) = e (Q * (s t ,a t )-V * (s t ))/β(5) where π * (a|s) is the optimal policy and Q * (s, a) is the optimal Q-function which models the total future reward from (s, a) under π * . The optimal value function V * is a function ofQ * , V * (s t ) = β log ∑ a∈A e Q * (s t ,a)/β (6)such that the policy π * integrates to one. Unfortunately unlike in the bandits setting this relationship gives us no specific information about the reward function r at a single state action pair since the optimal policy optimizes for total future returns as estimated by Q.To do so, we will need to consider the relationship between Q * and r.From r to Q * . The relationship between future returns and the current timestep is captured by the belmman equaitons which are satisifed by any valid Q-function. We write this below for the optimal policy π * under the reward r with a KL divergence penalty:Q * (s t , a t ) = r(s t , a t ) + β log π ref (a t |s t ) + V * (s t+1 ), if s t+1 is not terminal r(s t , a t ) + β log π ref (a t |s t ), if s t+1 is terminal (7)We can then rearrange the bellman equation for the optimal Q-function in terms of the reward. This style of relationship was first explored by Garg et al. (2022) in imitation learning and later in Hejna & Sadigh (2024) for preference-based RL. However, these works require the use of a discount factor γ < 1 which is typically not used in RLHF. In the appendix we prove the following Lemma which shows that this relationship is indeed one-to-one in the token MDP as well.Lemma 1. Under mild assumptions, there is a bijection between reward functions r(s t , a t ) and corresponding optimal Q-functions Q * (s t , a t ) in the token MDP.This leads us to a rather interesting conclusion -that an LLM is always the optimal soft Q-functions for some reward function in the token MDP. Consider any LLM which outputs logits l θ and temperature parameter β. As is common practice, we take the sampling policy π to be the softmax over tokens modulated by temperature parameter β -which is precisely eq. ( 5) where Q * = l θ because the value optimal function V * is precisely β log Z(s t ), normalizing the distribution. The corresponding reward function may not be smooth or well-behaved. Notably, the logits have a free parameter due to the softmax. While this free-parameter results in the same optimal policy per later arguments, it means the sequence of values may not be smooth. The question then becomes how to finetune the LLM such that it is the optimal Q-function for a reward function r that aligns with human preferences.To do so, we will complete our derivation of DPO in the token MDP.DPO learns our best estimate of Q * . Now that we have established a bijection between r and Q * , we can derive a token-level version of DPO to align the implicit reward, induced by the Q function represented by the language model, with that of the best estimate of reward, according to Bradley-Terry model in eq. (1). To do so, we need to represent the sum of rewards first in terms of the Q-function Q * , and then in terms of the policy π * . We complete the first step by inverting the Bellman equation in eq. ( 7) and substituting it into the sum of rewards over a trajectory τ = {s 1 , a 1 , . . . , a T-1 , s T }.T-1 ∑ t=0 r(s t , a t ) = T-1 ∑ t=0 (Q * (s t , a t ) -β log π ref (a t |s t ) -V * (s t+1 )) = = Q * (s 0 , a 0 ) -β log π ref (a 0 |s 0 ) + T-1 ∑ t=1 Q * (s t , a t ) -V * (s t ) -β log π ref (a t |s t )The equality follows from V * (s T ) = 0 and re-arranging the sum to isolate t = 0. As V * is written entirely in terms of Q * and β per eq. ( 6), we have expressed the sum of return over the sequence just in terms of Q * . Next, we exchange Q * for π * . We can log-linearize eq. ( 5) asβ log π * (a t |s t ) = Q * (s t , a t ) -V * (s t ). This is equivalent to stating that the language model probabilities are just the softmax over l θ = Q * with temperature β. Continuing from the above, with this substitution we get= Q * (s 0 , a 0 ) -β log π ref (a 0 |s 0 ) + T-1 ∑ t=1 β log π * (a t |s t ) π ref (a t |s t ) = V * (s 0 ) + T-1 ∑ t=0 β log π * (a t |s t ) π ref (a t |s t )where the final step results from adding and subtracting V * (s 0 ) and applying the substitution again. Now, this representation for the sum of rewards in terms of the optimal policy can be directly substituted into the preference model in eq. ( 1), where the V * (s 0 ) term will cancel just as Z(x) did in the original DPO derivation assuming τ w and τ l start at the same state s 0 , giving us the policy-induced preference modelEQUATION)To derive the final DPO loss function, we can take the KL-divergence between the empirical preference model of our dataset p D and the preference model implied by a learned policyp π θ , D KL (p D ||p π θ ). This results in L(π θ , D) = -E (τ w ,τ l )∼D log σ N-1 ∑ t=0 β log π * (a w t |s w t ) π ref (a w t |s w t ) - M-1 ∑ t=0 β log π * (a l t |s l t ) π ref (a l t |s l t ) (9)In the next section we demonstrate that DPO can learn any dense reward function in the token-level MDP.In the previous section we derived DPO using the bijection between reward functions and optimal Q-functions uniquely available in the token-level MDP. An alternative view of DPO casts it as restricting the learned reward function such that it belongs to the class optimal advantage functions A * (s, a) = Q * (s, a) -V * (s) from which an optimal policy is readily obtained per eq. ( 5). Here we show that this restriction does not limit the class of reward functions we can represent. We begin by expanding the definition of equivalency used in Rafailov et al. (2023) to the broader class of potential-based reward shaping functions: Definition 1. Two reward functions r(s t , a t ) and r ′ (s t , a t ) are equivalent if there exists a potential functionΦ(s), such that r ′ (s t , a t ) = r(s t , a t ) + Φ(s t+1 ) -Φ(s t ).In Ng et al. (1999) 's seminal work, the authors proved that two equivalent reward functions defined per definition 1 have the same optimal policy. By log-linearizing the optimal policy fixed point in eq. ( 5) and substituting in the Bellman equation from eq. ( 7) (Nachum et al., 2017; Watson et al., 2023) , we haveEQUATION)This is precisely the optimal advantage function, where V * directly follows the form of a potential shaping function. Watson et al. (2023) first used this derivation to arrive at a "coherent" reward function and follow-ups arrived at the same conclusion by noting that using the advantage as reward preserves the optimal policy (Knox et al., 2024; Hejna et al., 2024) . Unlike prior works, however, we demonstrate that this re-parameterization also leads to the same exact preference distribution as r.Theorem 1. Given a reference policy π ref and a parameter β > 0 all reward classes consistent with the Plackett-Luce (and Bradley-Terry) models in eq. ( 1) can be represented with the a reparameterization of the formr(s, a) = β log π(a|s) -β log π ref (a|s) (11)within the token MDP where V * (s t ) = 0 for all terminal states.Proof. Above we derived the invariance of the optimal policy under the re-parameterization.The preference model can be shown to be invariant by substituting and following the same steps used to arrive at eq. ( 8) in the last section, or by following Definition 1 from Watson et al. (2023) .Interestingly, in practice, the potential function Φ(s t ) represents the free parameter in the logits of the language model. An equal shift along all logits yields the same policy, but different Q-functions and corresponding rewards. The above Theorem proves that all of these are in the same equivalence class and induce the same set of preferences.Figure 1 : Credit assignment in DPO based on answer-level feedback. We provide two summaries to a Reddit post about a job interview. The left is the base response and on the right we have introduced errors in the salary range and the position level. Each token is colored corresponding to the DPO implicit reward as expressed in Eq. 11 (darker is higher), using the trained model. We see that the model correctly highlights the erroneous statements, without much change to the value of the other tokens, which indicates the ability to do credit assignment.Moreover, this Theorem implies that we can use DPO to learn the optimal policy for any per-token reward function, provided preference queries start at the same state and end at a terminal state. In addition, DPO always fits an optimal advantage function for some reward which is responsible for credit assignment. Thus, the training data determines how close the learned advantage corresponds to that of the true reward. This is in contrast to methods that estimate the reward function and then additionally employ some policy improvement mechanism. Which algorithm performs better remains largely an open or empirical question.The above derivations cast a language model as a Q function in the discrete token-level MDP. While this interpretation does not generally hold in continuous spaces, we can extend many of our results to other specially structured MDPs, like those present in diffusion. See Appendix B for more thorough treatment. 
practical insights: 
 In this section we discuss the empirical implications of our theoretical analysis. First, we qualitatively show that DPO can learn per-token credit assignment. Next, we use the derivations of the prior section to connect guided decoding and search-based algorithms, such as MCTS, to likelihood-based search on the DPO policy and empirically validate these results. Finally, (for the first time), we mathematically explain the phenomenon of decreasing likelihoods during DPO training, observed in the research and industry community.For all empirical evaluations we use the Pythia 2.8B model Biderman et al. (2023) and the Reddit TL;DR summarization dataset Stiennon et al. (2022) . We use the default hyperparameters from the original public DPO implementation, unless otherwise stated.In the previous section we outlined how the trained DPO policy represents an optimal Q-function for some reward that optimizes the preference equation. In this section, we evaluate qualitatively if the DPO-trained model is able to learn credit assignment from trajectory feedback. We begin with a generic set of Reddit posts for the TL;DR test dataset, which we provide in Appendix C with additional examples. In our representative example the user discusses an employment negotiations situation. Two answers are shown in Figure 1 . The base summary, which is correct is provided on the left. On the right we modify the summary by introducing a higher-level position and a corresponding higher salary.For each token in both answers we compute the DPO reward (equivalently the advantage function or "coherent" reward (Watson et al., 2023) ), r(s, a) = β log π θ (s|a)β log π ref (s|a), where π θ as outlined in Theorem 1 (here π θ is our DPO-trained model and π ref is the SFT model). In Figure 1 each token is colored proportionally to this reward. We see that the model successfully identifies the tokens corresponding to the erroneous statements, while still maintaining comparable values for the rest, which is indicates that it can do credit assignment. Moreover, we see that within the context of the first error ("250K" salary) the model still allocates reasonable values to the rest of the tokens and specifically identifies the second error "management position". This is a promising sign of the ability to do "stitching" Levine et al. (2020) i.e. a form of combinatorial generalization from offline data. If this is the case, our findings could be significant for the use of reinforcement learning and RLHF in LLMs, particularly for compositional tasks, such as code and reasoning. At the same time, in the recently introduced RewardBench Lambert et al. ( 2024), DPO models have demonstrated strong performance as classifiers on reasoning tasks. We believe these are encouraging results, which warrant further large-scale study beyond our qualitative observations. max a 0 ,...,a K r(s 0 , a 0 ) + β log π ref (s 0 , a 0 ) + . . . + r(s t , a t ) + β log π ref (s K , a K ) + V * (s K+1 ) (12)where V * is the optimal corresponding value function. Now, if we directly substitute the reward representation from Eq. 10 into the above and considering a telescoping sum, with some standard algebra, we obtain that the above objective is equivalent to maxEQUATIONwhere π * is the corresponding optimal policy. Now, since the starting state is fixed (it's given by the prompt) we have that a search algorithm based on the conservative reward function of the RLHF objective and the corresponding optimal value policy is equivalent to likelihood search on the corresponding optimal policy. We empirically verify this property in Fig. 2 , which shows the win rate of DPO models trained with three different β values against the preferred summary in the test dataset. We see that a 5-beam search improves win-rates by 10-15% over the base policy (1-beam), which is comparable to the value-function guided search improvements reported in Mudgal et al. (2024) . Interestingly, we see performance degrade with higher number of beams and that answers with exploding length are produced, which is a sign of reward over-optimization Gao et This can be seen as a form of likelihood based search as in Eq. 12, however instead of directly searching on the DPO model, it uses the STaR model as a proposal distribution. We hypothesize this is beneficial in preventing reward hacking, which is potentially an issue with deeper search as shown in Fig. 2 .Several recent works Under our considerations from the prior chapter, then this becomes equivalent to EQUATIONA surface level interpretation of DPO would lead one to believe it increases the likelihood of chosen responses, while decreasing the likelihood of rejected responses. This however, does not account for a well observed phenomena in which the likelihood of the chosen responses actually decrease over time (Pal et al., 2024) . This is illustrated on the left half of fig. 3 , which we show that when performing SFT before DPO, the implicit rewards of both the chosen and rejected response decline, though the margin between them increases. However, given a MaxEnt RL framing, this phenomena may be expected.Consider the expected log ratio (or implicit / "coherent" reward) of a policy under the reference model, which is often measured during training. Algebraic manipulation yields the following relationship:EQUATIONAt the beginning of training when π = π ref , the implicit rewards are trivially zero. However at the end of training, assuming π ref ̸ = π * , the KL-divergence is necessarily positive, indicating that the implicit rewards must decrease in expectation to converge. This means that the average implicit rewards should go down when starting from the SFT model. In fact, on the left side of fig. 3 we show that when one does not SFT before DPO, there is little discernible trend in the average implicit reward and the implicit rewards of the chosen responses remain above zero. In fact, this trend also holds for CPL Hejna et al. (2024) for the general MDP, where the implicit rewards actually increase if SFT is not used.One might realize that the previous analysis does not necessitate that the implicit rewards of the chosen must decrease, just that the implicit rewards must decrease on average. However, in practice it is common place (and recommended by Rafailov et al. (2023) ) to SFT on only the chosen responses to form π ref .For this section only we will call this choice of reference π w ref .Substituting π w ref into eq. ( 16), we can see that when SFTing on the positive answers the implicit rewards of the chosen responses must go down because at convergence as ref is done in practice (Rafailov et al., 2023) , it does mean that DPO will decrease the likelihood of all data in favor of extrapolated responses, which could cause over-fitting. Moreover, now that we have provided a derivation of DPO in the token-level MDP, one might expect it to exhibit characteristics like an RL algorithm -namely that the implied Q-function monotonically increases over time. However, this is not necesarily the case. Note that per analysis in Section section 3.1, DPO can be viewed as adjusting the reward (or advantage) from which the optimal policy is deterministically mapped within the token-level MDP. DPO does not train a policy to maximize reward, and thus we do not argue about whether its implied value functions should increase or decrease over time.E π w ref [β log π * -β log π w ref ] = -βD KL (π w ref ||π * ). 
discussion: 
 In this work we formulated the DPO optimization algorithm as learning an optimal Qfunction, which is represented by an LLM. This formulation and our results provide theoretical justification for empirically observed DPO training phenomena, which are not explained by the original bandit formulation. We further link and unify a family of proposed new LLM search algorithms by likelihood search under DPO and show comparable empirical gains by a simple 1-line code change to using beam search. Most importantly, we show qualitative early signs that DPO is able to learn credit assignment directly from feedback data. While larger-scale empirical exploration is necessary, we believe this an encouraging early sign. Our results indicate a number of promising future directions to explore: Teaching language models to be an interactive conversationalists has been difficult, as RLHF is optimized as a single-turn bandit formulation. Moreover, classical methods, such as PPO, are not applicable in this setting. Recent work by Andukuri et al. (2024) has shown success in this domain using STaR and extending DPO to multi-turn conversational trees is a promising direction.Agentic LLMs: LLM agents, such as WebGPT (Nakano et al., 2022) are equipped to take autonomous actions, such as browsing the Web and collecting information before providing an answer. The user then provides feedback based on the final output. Our derivations indicate that DPO training (on the full model trajectories) could learn optimal exploration behaviour. Recent works Song et al. (2024) ; Xi et al. (2024) shows promise in that direction.Modern image generation systems, such as Dalle 3 Betker et al. (2023) use an LLM to produce high quality conditioning before calling a diffusion generation model. Also, recent long-form video generation models Hu et al. (2023) ; Gupta et al. (2023) combine transformer-based auto-regressive generations with a diffusion-based decoder. Such systems could potentially be optimized end-to-end with ahybrid version of DPO. We expand on these points in the Appendix.We believe these are promising directions for future work.Conditional diffusion image generation models, such as Stable Diffusion 3 Esser et al. (2024) have also used a form of the DPO algorithm as outlined in Wallace et al. (2023) . Our analysis can no longer be directly applied in that setting, since the generations are continuous. However, we could translate many of our results to that setting, if we consider a certain diffusion MDP. We outline our results bellow.We borrow the denoising MDP formulation from Black et al. (2023b) ; Fan et al. (2023) . We again have the tuple (S, A, f , r, ρ 0 ), with the same formulation as in Section 3.1. At the same time consider a diffusion process with time index t and T total steps, conditioned on context c and image denoted by x t . Then we can map the diffusion generation process to an MDP in the following ways t = (c, T) if t = 0 (c, x T-t , T -t) otherwiseThat is the initial state consists of the prompt c and afterwards each state consists of the current denoised image x t and time step Tt. Notice that the time-steps in the MDP are inverse to the direction of the diffusion process (i.e. we start at noise and end at the final image). The action is just the next image iteration, from where the dynamics is also straightforward:a t ≜ x T-t+1 f (s t = (c, x T-t , T -t), a t ) = (c, a t , T -t -1)Notice that in this case the policy is stochastic, but the dynamics of the MDP is still deterministic. Finally, the initial distribution, is just the distribution of prompts:ρ(s 0 ) ≜ (p(c), 0)Given the above formulation, we can also prove that Lemma 1 also holds in the diffusion MDP. Lemma 2. Under mild assumptions, there is a bijection between reward functions r(s t , a t ) and corresponding optimal Q-functions Q * (s t , a t ) in the diffusion MDP.Proof. Since the MDP still has deterministic dynamics, we have that Eq. 5-7 still hold. Now, given a reference policy π ref , parameter β and a critic Q, we can trivially recover the unique reward function by inverting Eq. 7. We will prove that given a reward function r(s t , a t ), we can recover a unique critic Q. We work inductively in the diffusion MDP starting with t = T, where we have V * (s T ) = 0 for all terminal states. We then have thatQ * (s t-1 , a t-1 ) = Q * (s t = (c, x T-t+1 , T -t + 1), a = x T-t ) = r(s t = (c, x T-t+1 , T -t + 1), a = x T-t ) + β log p re f (x T-t |c, x T-t+1 , T -t + 1)+ β log A e Q * (s t =(c,x T-t ,T-t),x T-t-1 )/β dx T-t-1where π re f is the reference backwards diffusion process. In this case even though the state space is deterministic, our approach to the proof of Lemma 1 still holds by using backwards induction on the diffusion step t. Notice, that from V(s T = (c, x 0 , 0)) = 0 we can uniquely determine the critic values for all states at time step T -1. Proceeding inductively backwards through time in the MDP/denoising process (forward in the diffusion process), we obtain the desired result.Given the proof of this Lemma, we can then directly apply the results of Section 4.2, including Theorem 1. Our results, also give us insights into the formulation of Wallace et al. (2023) . In particular, by changing the sampling scheme of the intermediate diffusion the authors obtain two alternative formulations (Appendix S2 in Wallace et al. (2023) ). Both of these schemes are suggested as empirical approximations in the formulation of the Diffusion-DPO algorithm, however in the view of Q-learning both of these are valid approaches to generating off-policy data. In fact, our interpretation of DPO allows for general off-policy data sampling and aggregation methods, which could yield a whole family of DPO algorithms in this domain. We leave the exploration of this direction for further work.SUBREDDIT: r/running TITLE: Tips on getting back into running after 4 years of not doing so & shin splints POST: Hey everyone, I was hoping to gather some tips from people who left running and had to start over. A semi-lengthy background on myself to help you understand where I am coming from. In high school I was a very good cross country runner, running from 35-50 miles a week and never slower than 8-9 minute miles. At the end of senior year, I planned on taking a break from running and then try to race half or full marathons in the spring. I ended up not running at all after xc. 4 years later, I was noticing how much I miss the sport (especially after seeing the success of xc friends) so I decided to join a running group to get back into it. But the only group at my university that I could find was a triathlon club. I joined them, but only did the running workouts. After about 4 weeks, I developed shin splints. This is because I haven't ran in 4 years but thought 6 miles was ok after 4 weeks. Also, being 25 pounds heavier didnt help. After taking 3 months off and training on the bike and in the pool, I finally was back to running in february. but my shinsplints was still around. I finished my first sprint triathlon last week, and have been trying to get miles back under my feet again. I havent felt shin splints severely since the beginning of March, but I can feel it looming around. After a half year of it, I am getting really really frustrated. I cant run more than 4 miles still and my fastest mile is 8 minutes. I know I will probably never run like I did when I was 17, but its difficult because of remembering what I used to be capable of running.DPO successfully identifies the wrong tokens ("busted knee") in the right summary and correctly classifies this pair. SUBREDDIT: r/AskReddit TITLE: What is a sub-$800 camera that can shoot high quality video ideal for music videolike appearances?POST: [This is a video of what we're trying to achieve.]( My school currently has a Sony HVR-HD1000u, and compared to that, our videos are nowhere near as good. I understand that things like lighting and color correction play a pretty big role, but even then I feel like our videos are never that clean. I usually can't get 720p clips out of our camera and the slow motion that they have is something we can't even come close to.One possible *problem* is that for some reason we can't use firewire to connect the camera to the computer so we have to play the tape on this thing that basically plays it and then we capture the tape playing. I feel like this is probably a huge problem because it's like trying to show a friend a movie by screen-capping from Skype.SO, should we scrap the HVR-HD1000u and get a Canon T2i (a cheaper DSLR which from the samples I've seen on YouTube and clips from that video, seems pretty high quality), or continue trying to use the Sony? DPO successfully identifies the wrong tokens ("iPhone") in the right summary and correctly classifies this pair. SUBREDDIT: r/personalfinance TITLE: When asked about salary expectations during my interview I said 38k to 45k. Was just offered the position with 38k. Should I try and negotiate? POST: So I interviewed for a position last week, and before the interview I saw online that the industry average for this position was $41,000. During the interview, they asked me my salary expectations, I said between $38,000 and $45,000 hoping it'd land somewhere in the middle. I received my offer today, and it was for $38,000. I can't help but wonder if I had just said $41,000 they probably would've offered it... Anyways, so what I know is they are hiring 3 other people for this same position... I either got lucky and guessed exactly what salary they were planning on paying all of us to begin with, or we're all getting paid differently. As for the job, it is the ideal entry level position for me right now, and is a great company with benefits etc so I actually wouldn't mind working there for the 38k salary.But it would be nice to get an even 40 at least, so my question is, is it common practice to negotiate salary after receiving an offer already? I also must say that I don't have any leverage as this is entry level and I would have probably still accepted had the offer been even as low as 30k. As such, I'm very afraid the offer may be retracted if I do try and negotiate, if that sort of thing happens? DPO successfully identifies the wrong tokens ("250k" and "management position") in the right summary and correctly classifies this pair.Figure 4 : Example of an end-to-end generative AI workflow. The user request an image of a Minotaur in the streets of New York. However, we see that the rejected image does not actually represent a Minotaur, but a bull. The prompt refiner generates a valid description and specifically includes the phrasing "the body of a man and the head of a bull", but the image generation model fails to follow that prompt. At the same time in the case of the chosen image (which does reflect the prompt) the prompt refiner generates a more descriptive text, which the image generator is able to follow more closely. While it is possible to train each component separately, joint training can in theory, optimize the refiner to generate prompts that the image generator can more closely realize, while also training the generator to directly produce more aligned images.End-to-end system: Here, we present an outline for end-to-end training of generative AI systems from human feedback using DPO in multi-step MDP. We assume two models -a prompt refiner π θ (z|x), which is a language model, generating discrete tokens in an autoregressive way, which, given a prompt x produces a refined prompt z. This prompt is then fed into an image-generation diffusion model π ϕ (y|z), (which we parameterize as a denoising model ϵ ϕ ), which generates the final image. A real case example of that system is shown in Figure 4 .During the feedback gathering stage, a user provides a query x and two refined prompts are sampled from z 1 , z 2 ∼ π θ (x|z), which the user does not directly evaluate. Then, the image generation model generates two images y i ∼ π ϕ (y|z i ) for i = 1, 2. The user then provides a preference over the y i to yield the preference pair {y w , z w ≻ y l , z l |x} Optimization: Optimization is carried in a hybrid MDP, where the initial state is the prompt x and has the same form as the token MDP, as outlined in 3.1. When the EOS token is encountered in that MDP, at which point the transition dynamics switches to the diffusion denoising MDP introduced in Black et al. (2023a) . Notice that this is still a valid MDP and all the conclusions of our derivations in the main section of our paper hold. We could then optimize this system end-to-end using a hybrid DPO objective, combining our results, with those presented in Wallace et al. (2023) we have:r θ,ϕ (x w , y w ) = β |z w | ∑ i=0 log π θ (z w i |x, z w <i ) π ref (z w i |x, z w <i ) prompt refiner MDP + γTω(λ t )E t∼U (0,T) (∥ϵ w -ϵ ref (y w t , z w , t)∥ 2 2 -∥ϵ w -ϵ ϕ (y w t , z w , t)∥ 2 2 )diffusion MDP objective (17) where β and γ are two separate discounting factors for each modality. Here the diffusion objective follows directly from the derivation of Eq. 9 and the sampling scheme proposed in Wallace et al. (2023) (Eq. 12-14 in that paper). Notice here that the image generation model is conditioned on the corresponding refined prompt z w . We can define r(x l , y l ) similarly and optimize the DPO objective:L DPO θ,ϕ = -E (x,y w ,y l )∼D log σ r θ,ϕ (x w , y w )r θ,ϕ (x l , y l )We demonstrate a particular real use of such a system in Figure 4 . The user request an image of a Minotaur in the streets of New York. However, we see that the rejected image does not actually represent a Minotaur, but a bull. The prompt refiner generates a valid description and specifically includes the phrasing "the body of a man and the head of a bull", but the image generation model fails to follow that prompt. At the same time in the case of the chosen image (which does reflect the prompt) the prompt refiner generates a more descriptive text, which the image generator is able to follow more closely. While it is possible to train each component separately, joint training can in theory, optimize the refiner to generate prompts that the image generator can more closely realize, while also training the generator to directly produce more aligned images.A recent line of work on long-form video generation Hu et al. (2023) ; Gupta et al. (2023) by combining auto-regressive transformer generation with a diffusion model decoding or uspcaling of the actual video frames to obtain temporally consistent and high-fidelity generations. We could deploy similar RLHF pipelines to the video-generation problem as well. It is also straightforward to extend the DPO joint optimization framework, presented in the previous section in Eq. 18 to this stetting as well. Instead of textual prompt refiner tokens, the variables z would represent the latent token generations of the autoregressive component π θ , which would be decoded into actual video frames y via the diffusion decoder π ϕ . We believe this is an exciting direction to pursue for the emerging video generation technologies. We consider whether the beam search trends in Figure 2 hold for PPO, since PPO is known to exhibit reward over-optimization as well (Gao et al. (2023) ). We use a PPO-tuned GPT-J-6B model released by CarperAI1 , fine-tuned on the same TL;DR dataset. We use the same sampling parameters as with the DPO experiments (τ = 1.0, k = 50) and generate 256 samples from test set prompts, with GPT-4 as the evaluator. We only report results for 1, 2, 5, and 7 beams; higher number of beams were tried but exhausted memory on an NVIDIA A40 GPU.In Figure 5 , we observe a similar over-optimization phenomenon as with DPO, with 2 beams both increasing winrate and decreasing sample length (even under the lengthbiased evaluator GPT-4). However, more than 2 beams leads to a decline in downstream performance and large uptick in sample length, similar behavior to DPO in the overoptimization regime. We find that the benefits of increasing the beam size are more limited when using this model.All samples generated from the β = 0.1 checkpoint.SUBREDDIT: r/AskReddit TITLE: Question regarding learning/teaching.POST: I apologize in advance if this is not the right sub reddit for this kind of question.for the past 2 years, i've been slowly but surely developing my illustration skills. I am 18, recently out of high school, and will be going to college for a bachelors in illustration in likely 1-2 years.Despite teaching my self to be better, by using resources from the internet/library ext, but improvement has been slow. It can be seen, but I still feel i am behind where i should be at my age, and for how long i have been drawing/painting. I've met many artists who are self taught, and never went to college at all for what they do, yet i have it in my mind that college will help me greatly, since i learn faster when taught by another, rather then myself.My question is, is this actually possible? do some people just have a harder time of teaching theirself? I have the dedication and interest to pursue education, but i respond so much better to being instructed, rather than instructing myself. Is this normal?Num beams = 1. TL;DR: i want to continue my art education, but still be able to teach myself as much as possible, is it possible to do both at the same time? I appreciate all inputs.Num beams = 5. TL;DR: 18, recently out of high school, and will be going to college for a bachelors in illustration in likely 1-2 years. Slow but steady improvement in my drawing/painting skills, but i respond better to being taught rather then instructing myself. Is this normal?, Num beams = 20. TL;DR: 18 year old self taught myself to be better at drawing/painting over the past 2 years, but still feel i am behind where i should be at my age, and for how long i have been drawing/painting. Do some people just have a harder time of teaching themselves? Is this normal?SUBREDDIT: r/legaladvice TITLE: Mom was hurt at work, I think it's something that is a little past worker's comp. Any advice?POST: My mom works at an old navy for something upwards of almost ten years. Over the span of her working their they have been lingering in and out of being abusive employers. Not that this isn't something I wouldn't expect out of a big corporation, something happened recently that I'm close to positive should be lawsuit worthy or something in my mom's favor.The other day on the clock my mom was cleaning one of the bathrooms and while taking the garbage out there was a faced up syringe under some paper towels! She found out after jabbing her hand with the syringe and noticed that there was preexisting blood on this syringe!!! She was then brought to the hospital (obviously) to be screened for all sorts of diseases and had the option of starting intense medications to prevent the microscopic chance of that needle being contaminated with HIV though the medications would make her very ill. All old navy did about the situation was lock the bathrooms so the public can't use them.Is there anything about this situation that my mom can use in her favor? She didn't turn the syringe over to Old Navy because they were being very nonchalant and dismissive about the situation.Num beams = 1. TL;DR: mom is a working at old navy, had a bloody syringe found under some paper towels on the garbage disposal, likely contaminated with HIV/AIDS. Any help in what she can use in her favor in future lawsuits or otherwise?Num beams = 5. TL;DR: Mom found syringe with preexisting blood under paper towels at work. Old navy was dismissive and locked the bathrooms so the public couldn't use them.Is there anything my mom can use in her favor? She didn't turn the syringe over to Old Navy because they were being very nonchalant and dismissive about the situation.Num beams = 20. TL;DR: Mom found a syringe in the bathroom at work that had preexisting blood on it. Old navy was dismissive about the situation and locked the bathrooms so the public couldn't use them. Is there anything my mom can use in her favor? She didn't turn the syringe over to Old Navy because they were being very nonchalant and dismissive about the situation. A: My mom works at an old navy for something upwards of almost ten years. Over the span of her working their they have been lingering in and out of being abusive employers. Not that this isn't something I wouldn't expect out of a big corporation, something happened recently that I'm close to positive should be lawsuit worthy or something in my mom's favor. The other day on the clock my mom was cleaning one of the bathrooms and while taking the garbage out there was a faced up syringe under some paper towels! She found out after jabbing her hand with the syringe and noticed that there was preexisting blood on this syringe!!! She was then brought to the hospital (obviously) to be screened for all sorts of diseases and had the option of starting intense medications SUBREDDIT: r/AskReddit TITLE: [Serious] How can I tell my mother politely, that she is dealing with our dog in an inappropriate way?POST: My mother watched Cesar Millan a lot before we got our dog, and she acted like she knew everything there was to know about training and raising a dog.Its been about 2 years with our dog, and he has a problem with barking at the door when people go to open it (doesn't seem like that much of a problem to me). My mother's reaction is to yell at him, or if he does it sequentially, she will strike him in the thigh/rear end.Whenever I step in and try and take control of the situation, she will get mad, and try and tell me how to handle this kind of thing, because she thinks she is a dog expert. (This upsets me because it feels like I am being talked down to)The last strand was crossed today, when I was in the basement watching tv, I heard my dog bark at the door, and then my mother get up and yell at him. He ran into our sun room, which is above the basement, and I could hear her hit him, and his paws skid on the floor. I went up to check on him (he was fine, no damage, happy little guy when he's with me) and my mother yelled at me to put him back in the room where he is to stay for a long long time because he tried to bite her when she was disciplining ḧim. https://huggingface.co/CarperAI/openai summarize tldr ppo 
 
 Appendix:  


Decision: This theoretical paper sheds new lights into connections with the status quo algorithm of RLHF (PPO) and a popular alternative introduced not so long ago (DPO). In particular it provides an alternative derivation of Direct Preference Optimization that shows that it does produce token-level rewards (instead of single final action at the end of the sequence).

This alternative derivation then allows the authors to leverage beam search for decoding and also to explain the unsatisfactory phenomena where the rewards of both completions (better & worse) decline all while the difference between them increases.

Most reviewers loved this paper, despite some experimental conclusions which are not of the same level than the theoretical insights. It would certainly be a valuable addition to COLM's programme and trigger insightful discussions.

Reviews: review: paper_summary: paper_summary: This paper compares the classic pipeline adopted in the GPT series with DPO, a more recent method proposed by the research to overcome the complexities and intricacies of the RLHF pipeline with PPO. The main issue with DPO is that it is formulated as a contextual bandit problem where the entire response is considered as an arm ignoring the fine-grained decision-making process underlying language modelling and its relationship with credit-assignment. 

As a first contribution, the authors demonstrate that it is possible to reframe DPO in the token-level MDP which gives the ability to shed some light on the credit assignment ability of this method. Additionally, because it is now possible to define a Q-function in terms of single tokens, it is possible to formulate search-based algorithms (such as Monte-Carlo Tree Search) via likelihood-based search implemented as beam-search. Finally, thanks to this theoretical framework, the authors study a well-known problem in DPO training which causes implicit rewards to decline during training and conclude that this is a result of the DPO training process (i.e., this is a feature of DPO and not a bug).
strength_weakness: strength_weakness: Overall, I think this paper represents a great contribution to the community and in general can be considered as a great work for many to learn from. However, I was a bit disappointed by the level of detail of the case studies reported in Section 5 which I will list below. Additionally, I will report more details and suggestions for authors in the "Questions for Authors" box.  

1. [Section 5.1 - training details] Section 5.1 reports a case study on text summarization to study the ability of DPO to learn credit assignment but it somehow lacks some details regarding the concrete steps involved in training this model.
2. [Section 5.1 - credit assignment comparison] Again connected with the previous point, the evaluation feels very anecdotical. I believe this credit-assignment evaluation is valuable but it should be executed in a more principled way.
3. Overall, all the sections involving some training should report more details to facilitate other researchers to reproduce your experiments. I understand that the space is limited and authors decided to prioritise their theoretical contribution but I think it's important to report these details in the appendix.
questions: questions: ## Suggestions regarding weaknesses of the case studies

1. [Section 5.1 - training details] The authors specify that they used Pythia as a base model but it's not clear how they completed the training for the Reddit TL;DR dataset. Was this model instruction-tuned first using SFT or not? Did they try completing this step for other open-source models? I believe it would be good to comment on this to provide more information regarding how their work will influence the overall LLM training pipeline.
2. [Section 5.1 - credit assignment comparison] I believe that the authors should execute this evaluation in a more systematic way. They only report very specific examples for the DPO model only. Does a model trained only with SFT behave in a similar fashion? Is there any difference between the two?
3. I would recommend the authors use the appendix to report more details regarding the training steps required to reproduce their case studies. I think it's important to at least include hyperparameters and other steps required to reproduce the training pipeline that they have used.

## Minor edits

1. Typo at Page 4: belmman equaitons
2. I would suggest the authors consider moving their related work at the end of the paper considering that the "Preliminaries" section provides a very good overview of all the technical background required to understand their theoretical contribution.
3. Authors should consider citing older work on using RL for image captioning such as [1]

## References
[1] Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi, and Rita Cucchiara. 2020. Meshed-memory transformer for image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10578–10587.
paper_summary: paper_summary: This work offers an alternative derivation of Direct Preference Optimization using Markov Decision Processes (MDPs) that enables to define token-level rewards. This has theoretical and practical applications. Fundamentally that DPO can learn token level rewards and assign different credit to tokens, a relation between guided decoding and search-based algorithms and an explanation of the phenomenon of decreasing likelihoods during DPO training.

The core of the paper is a derivation that follows closely that of DPO but from an MDP perspective i.e. [S4.1]

1. Determining the optimal policy for the KL-controled reward maximization loss. Compared to DPO here they consider a more general case of the loss that uses a MDP characterization with states and actions as a function of time. The optimal policy for this case is a know result, as it happened with DPO.

2. Deriving a the reward from the this optimal policy. Compared to DPO, here the use of the  Bellman Equations is needed to obtain the token level relations. The value function cancels in the same way the partition function cancels in DPO. I may have missed details on this part, but it seems solid. Result of this is Eq (9) a derivation of DPO as an MDP.

3. In addition to this they generalize the definition of equivalency of DPO for the MDP case. This exploits very similar properties as in the case of DPO (S4.2).

In the following sections they use these derivations to deduce certain properties of DPO and empirically confirm these i.e.

4. The MDP derivation of DPO, indicates that it can learn token level rewards. 

5. Guided decoding that maximizes a reward, also based in the MDP formulation, is the same as maximum likelihood DPO optimization.

6. Explain the known phenomena that "the implicit rewards of both the chosen and rejected response decline, though the margin between them increases", which once is seen from the perspective of the MDP it is explainable.
strength_weakness: strength_weakness: Not really reasons to reject, but more clarity on some RL terms NLP people may not be fully familiar with. Thanks to RLHF's impact in LLMs, the average reader on this conference will be familiar with a lot of the material (I am one of those), but maybe some concepts may remain unfamiliar, such as, bandits, MDP and Q function. 

I have to say I did not fully understand some of the claims and some aspects seem self-evident, I put these on the question section. Probably after discussion this should all be ok.
questions: questions: ### S 4.1
>While the original derivation of DPO relies on the fact that Q∗(x, y) = r(x, y)

*Question 1:* Can you clarify this assertion? The original derivation in (Rafailov et al 2023) does not use MDPs or Q. It basically consists on two elements

-  derivation of the form of the optimal policy for the KL-controlled reward maximization objective (appendix A.1). This just needs algebraic manipulation of the original objective and checking the solution satisfies this and is unique.

- derivation of the DPO objective, which follows again by algebraic manipulation of the optimal distribution derived above and replacing this in the Bradly-Terry model.

I am aware that there are earlier derivations for the optimal policy, but this one seems simple and not need of the assumption above. 

### S4.1
>This style of relationship was first explored by Garg et al. (2022) in imitation learning and later in Hejna & Sadigh (2024) for preference-based RL. However, these works require the use of a discount factor γ < 1 which is typically not used in RLHF.

*Question 2:* Could you expand on this difference? It seems to be and important aspect to judge the novelty. Maybe add some summary of the derivation used in those works in the Appendix A.  

### S.1
>Second, we demonstrate that likelihood search over a DPO model is analogous to searchign over a reward function during decoding as done by contemporary works (). 

*Question 3:* can you provide this missing reference? 

### S 4.2

>Interestingly, in practice, the potential function Φ(st) represents the free parameter in the logits of the language model. An equal shift along all logits yields the same policy, but different Q-functions and corresponding rewards. The above Theorem proves that all of these are in the same equivalence class and induce the same set of preferences.

I fail to see why this is not self-evident. Scaling the logits by f(x) does not affect the probability distribution. Could you please clarify this?
paper_summary: paper_summary: This paper formulates DPO in the token-level MDP and provides practical insights into DPO policy based on the derivation. The derivation is carefully step-by-step, and the paper is well organized. The paper provides interesting insights into the token-level behavior of DPO.
strength_weakness: strength_weakness: * I understand that the main contribution is the token-level formulation of DPO, and I find the practical insights based on this formulation interesting. However, the empirical results are too weak to support the practical insights the paper insists on.
  - Section 5.1 shows only several qualitative results (including the appendix) to argue that DPO learns token-level rewards. 
  - Section 5.2 attempts to make a connection between guided decoding and DPO by showing that DPO improves performance with moderate beam size. However, the relationship between the argument and the results is unclear. In text generation, it is well-known that beam search with a moderate size of beam improves performance on (e.g., [1]). Why should the cause of that improvement be attributed to DPO?
* The derivation provides a new interpretation of DPO but does not accompany improvement on DPO. This is a minor point and not the focus of this paper, but the current introduction makes such an expectation (e.g., "we rectify this difference").

[1] If beam search is the answer, what was the question? (EMNLP 2020)
questions: questions: * I do not clearly see the equivalence between Eq (2) and the formulation of RLHF in the previous studies. Are they exactly the same?
* The first line in Section 4.1 says, "While the original derivation of DPO relies on the fact that $Q^*(x, y) = r(x, y)$," but I could not find this statement before this line. Where is it explained?
* Isn't M and N opposite in Eq. (9)?
* Section 5.3 provides interesting findings, but what is its connection with the token-level formulation of DPO?
* Typos
  - Abstract: multi-tun -> multi-turn
  - Section 1, paragraph 4: searchgn -> searching; missing citation in the next line.
  - Section 3.1, after Eq (1): "loss" -> "lose"?
  - Section 3.3, paragraph 1: redundant "entirely"
  - Setion 4.1, paragraph 1: $\pi*$ -> $\pi^*$

Comments: title: Response
comment: **The concern about this Section 5.2 is not addressed…**

We are working on evaluating a PPO-trained model under beam search for the same dataset and will report results as soon as they are available. As stated by the reviewer, a-priori we also expect performance gains in PPO policies with beam search. The goal of Section 5.2 is not to explain the performance gains of BS, but rather to show the equivalence of value-guided search in LLMs (which has driven a lot of research interest recently) to likelihood-based search (at least in the RLHF domain), of which BS is an example. We empirically show similar performance gains with simple BS as reported in prior works with value-guided decoding, without actually training a separate value function. 

**The connection between Section 5.3…**

The expectation in Eq. 14 is implicitly taken over the sequence of tokens induced by the reference model, but we believe the reviewer is right and this result also holds in a bandit interpretation where the corresponding expectation is over the prompt distribution.
I appreciate the authors' response.
I am already positive about the paper, but I have some concerns that the response does not sufficiently address.
* The concern about this Section 5.2 is not addressed in this response. In general, appropriately increasing the width of the beam search will result in better performance than the greedy search or excessively large beam size, and the fact that this phenomenon has been reconfirmed for DPO policy does not support that DPO is implicitly learning a value function. If one wants to make such a claim, it is necessary to conduct similar experiments and compare, for example, RLHF policy in the absence of value function guidance.
* The connection between Section 5.3 and the main claim, token-level formulation of DPO, remains unclear.
Thank you! We will be sure to report the points raised as limitations. 

1. We will report full details on implementation. All of our hyper-parameters were taken from the official DPO repository. For sake of completeness, we include this here. We use Pythia 2.8B for all experiments. We first SFT on the preferred responses using batch size 64 with two gradient accumulation steps and learning rate of 5e-7. Prompt-responses are truncated to 512 tokens and we train for one epoch using the RMSProp optimizer. Gradient norms are clipped to 10. We then run DPO using batch size 32 (each example is positive and negative) again with 2 gradient accumulation steps. We use the same learning rate and truncation for DPO. For DPO we use beta = 0.1.

2. We will qualify the credit assignment portions of the draft to further indicate that we are using qualitative (not quantitative) analysis and that the existing evidence is anecdotal in Section 5.1.

We hope this addresses your suggestions!
thanks for the refrences
I thank the authors for your honest answers. I would appreciate it if they could report the points that I raised above as limitations of this work which, nevertheless, I still believe represents a very good contribution to this conference.
Hi! Sorry about that oversight. The references are:

Liu, Jiacheng, et al. "Making ppo even better: Value-guided monte-carlo tree search decoding." arXiv preprint arXiv:2309.15028 (2023).

Feng, Xidong, et al. "Alphazero-like tree-search can guide large language model decoding and training." arXiv preprint arXiv:2309.17179 (2023).

Please let us know if you have any other questions!
I thank the authors for their answers. Unless I missed something, this question remains unanswered

>Question 3: can you provide this missing reference?