From $r$ to $Q^*$: Your Language Model is Secretly a Q-Function
Authors: Rafael Rafailov
Joey Hejna
Ryan Park
Chelsea Finn
Venue: COLM (2024)
Keywords: RLHF, Alignment, Direct Preference Optimization
TL;DR: We prove that direct alignment algorithms, such as DPO can be interpreted in a Q learning framework. This yields novel insight, empirical results and new potential applications.

Abstract: Reinforcement Learning From Human Feedback (RLHF) has been a critical component of the success of the latest generation of generative AI models, including the GPT series. However, this is an involved and complex process and direct alignment algorithms, such as DPO have recently emerged as an alternative approach to the classical RLHF pipeline. Although DPO solves the same objective as the standard RLHF setup, there is a mismatch between the two approaches. Standard RLHF deploys reinforcement learning in a specific token-level MDP, while DPO is derived as a bandit problem in which the whole response of the model is treated as a single arm. In this work we rectify this difference, first we theoretically show that we can derive DPO in the token-level MDP as a general inverse Q-learning algorithm, which satisfies the Bellman equation. Using our theoretical results, we provide three concrete empirical insights. First, we show that because of its token level interpretation, DPO is able to perform some type of credit assignment. Next, we prove that under the token level formulation, classical search-based algorithms, such as MCTS, which have recently been applied to the language generation space, are equivalent to likelihood-based search on a DPO policy and empirically we show that a simple beam search yields meaningful improvement over the base DPO policy. Finally, we show how the choice of SFT policy causes implicit rewards to decline during training. We conclude by discussing applications of our work, including information elicitation in multi-tun dialogue, reasoning, agentic applications and end-to-end training of multi-model systems.

Published as a conference paper at COLM 2024
From r to Q": Your Language Model is Secretly a Q-Function
Rafael Rafailov*
Joey Hejna* Stanford University jhejna@stanford. edu
Ryan Park Stanford University
Stanford University
rafailov@stanford..o edu
ryparkestanford..
Chelsea Finn Stanford University cbfinn@stanfon edu
Abstract
Reinforcement Learning From Human Feedback (RLHF) has been critical the success the latest generation of generative Al models. In response to the complex nature of the classical RLHF pipeline, direct alignment al- gorithms such as Direct Preference Optimization (DPO) have emerged as alternative approach. Although DPO solves the same objective as the standard RLHF setup, there is a mismatch between the two approaches. Standard RLHF deploys reinforcement learning in specific token-level MDP, while DPO derived as a bandit problem in which the whole re- sponse of the model is treated as a single arm. In this work we rectify this difference. We theoretically show that we can derive DPO in the token-level MDP general inverse Q-learning algorithm, which satisfies the Bellman equation. Using our theoretical results, we provide three concrete empirical insights. First, we show that because of its token level interpretation, DPO is able to perform some type of credit assignment. Next, we prove that under the token level formulation, classical search-based algorithms, such MCTS, which have recently been applied to the language generation space, are equivalent to likelihood-based search on DPO policy. Empiri- cally we show that simple beam search yields meaningful improvement over the base DPO policy. Finally, we show how the choice of reference policy causes implicit rewards to decline during training. We conclude by discussing applications of our work, including information elicitation in multi-turn dialogue, reasoning, agentic applications and end-to-end training of multi-model systems.
Introduction
Reinforcement Learning from Human Feedback (RLHF) has become the defacto method for aligning large language models (LLMs) with human intent due to its success in wide range of applications from summarization (Stiennon et al., 2022) instruction following (Ouyang et al., 2022). By learning reward function from human-labeled comparisons, RLHF is able to capture complex objectives that are in-describedable in practice. Following the success of (Ziegler et al., 2020), numerous works have considered new algorithms for training and sam- pling from large models in various domains using techniques from reinforcement learning (RL). In particular direct alignment methods, such as Direct Preference Optimization (DPO) (Rafailov et al., 2023) have gained traction in recent months because of their simplicity (Zhao et al., 2023a; Azar al., 2023). Instead of learning reward function and then using RL, direct alignment methods use the relationship between reward functions and policies in the contextual bandit setting optimize both simultaneously. Similar ideas have since been applied to vision language (Zhao et al., 2023b) and image generation models (Lee et al., 2023).
* Denotes equal contribution

---

Published as a conference paper at COLM 2024
While such direct alignment methods are purported to work the same classical RLHF approaches that use policy gradient algorithms like PPO (Schulman et al., 2017), fundamen- tal differences remain. For instance, classical RLHF methods optimize token-level value functions with a sparse reward at the terminal state. DPO on the other hand, operates only in contextual bandits setting, treating the entire response the fact that tokens are generated one at time, and dense rewards are commonly known to be beneficial in the RL community. While direct alignment algorithms are interesting, at present it is unclear if they can be applied to sequences in the same way as the underlying RL algorithms used in typical RLHF pipelines.
single arm. This is despite
In this work we rectify this difference by deriving DPO within the token-level MDP setting present in large language models using the usual form of binary refferenceffedbaee We then show that DPO training implicitly learns a token-level reward function, for which the language models logits define the optimal Q function, or expected total future reward. We then demonstrate that DPO is able to flexibly model any possible dense reward function within the token MDP.
Empirically, we use our theoretical derivations to justify three practical insights which we believe to be of use to the community. First, we show that despite being derived as a contextual bandit, the implicit rewards of DPO model have per-token interpretation. Second, we demonstrate that likelihood search over a DPO model is analogous to searching over reward function during decoding as done by contemporary works (Liu et al., 2023b; Feng et al., 2024). Finally, we identify the choice of initial policy and reference distribution as being important in determining the trajectory of implicit rewards during training.
2 Related Work
The problem of aligning policies with human intent using preference feedback has been a long studied problem in reinforcement learning (Akrour et al., 2011; Wilson al., 2012). While the primary focus of RLHF was originally in control (Christiano et al., 2017), following the success of Ziegler et al. (2020) it has recently been broadly adopted by the language modeling (Ouyang et al., 2022, Nakano et al., 2021; Stiennon et al., 2022; Bai et al., 2022a) and even vision communities (Black et al., 2023a; Lee et al., 2023). Most works in RLHF optimize learned reward function, used only at the end of generation, with a policy graident style- method. Such approaches have been known to be unstable (Engstrom et al., 2020) and hard to scale, while at the same time theoretically existing at an unusual intersection between contextual bandits and RL. In response, several direct alignment methods (Rafailov et al., 2023; Azar et al., 2023; Zhao et al., 2023a) have been developed which simplify the RLHF pipeline by learning a policy from preference data without an intermediate reward function. Such methods however, derived solely as contextual bandits, leave several theoretical and practical questions unanswered which we seek to address.
First, though direct alignment methods treat the LLM as bandit, prior works have demonstrated that it is possible to use dense rewards Zelikman et al. (2022); Chan et al. (2024); Pan et al. (2023) or even approximate dynamic programming (Snell et al., 2022). Moreover, using the regret-model of preferences (Knox et al., 2023; 2024), Contrastive Preference Learning (Hejna et al., 2024) is able use direct alignment for general MDPs, instead of the specific token MDP used in RLHF. Our work shows how DPO can be interpreted as optimizing a per-token reward function, which in practice is restricted to the family of optimal advantage functions.
Second, if DPO does not learn a reward function, can we still use its reward or value? Prior works have considered using best-of-K (Mudgal et al., 2023) or tree search (Liu et al., 2023b) for alignment with a value function Kim et al. (2022); Li et al. (2017) or discriminator (Yang & Klein, 2021). Using the implicit reward, we show that likelihood search results in similar solution for direct alignment.
Our work builds on foundational knowledge in maximum entropy RL (Ziebart, 2010) and inverse RL (Ziebart et al., 2008; Ng et al., 1999; Cao et al., 2021). In particular, we leverage the mapping between Q-functions and reward functions under fixed policy first done
2

---

Published as a conference paper at COLM 2024
in inverse RL by Garg et al. (2022). Related to work, Nachum et al. (2017) uses similar derivations for reinforcement learning in control and Watson et al. (2023) does so for inverse RL from demonstration. Hejna & Sadigh (2024) exploit this relationship for RLHF. While related, these works still require an additional loop of reinforcement learning optimization, which we dispose of in our formulation of feedback learning for LLMs. In the LLM domain, Yu et al. (2024) uses pre-trained models as priors for Q-learning, while Cundy & Ermon (2023) considers a similar formulation for imitation learning. In this work instead, we formulate preference-based learning as Q-learning.
3 Preliminaries
In this section we first define the per-token MDP for large language models, and then de- scribe how it relates to classic RLHF approaches and direct alignment algorithms, specifically DPO. We operate in the typical RLHF setting where we have dataset D = {(xli) y()) )
of language prompts and target answers y, which can each individually be broken down into a sequence of tokens, for example x = (xo....,
from a fixed discrete vocabulary
A. Throughout this section we will use the y notation for the contextual bandit framing where the entire response y is the action, but will use state and action notation from RL literature for describing sequences at the token-level.
3.1 The Token-level MDP for Large Language Models
We define the token level MDP as a tuple M = (S,, where the state space S consists of all tokens generated far (i.e. S; {xo...., Xm, yo. yo..... and the action space is the vocabulary of tokens A. The dynamics f are the deterministic transition model between tokens f(, s|a, where concatenation. The initial state distribution po is distribution over prompts where an initial state 5p is comprised of the tokens from In RLHF, the reward function is learned from human feedback over preferences between responses which we will denote using trajectories at the token level. As is typically done (Ziegler al., 2020; Stiennon et al., 2022), we assume that preference trajectories start at the same state (initial propmpt) and end in terminal state (EOS token), from which future rewards are zero. In this token level MDP, the corresponding Bradley-Terry preference model Bradley & Terry (1952); Christiano et al. (2017) is
ENN r(s)")
exp
pº (7
T')
exp r(""" a!r) + exp r(s,
which gives the probability that the "win" trajectory "H of length N is preferred to the "loss" trajectory rl of length M. Now that we have defined the token level MDP, we can show how it relates to both classic and direct alignment RLHF methods.
3.2 The Classical RLHF Methods
Most classical RLHF approaches (Ziegler et al., 2020; Bai et al., 2022b; Ouyang et al., 2022) first learn reward function from human feedback on prompt and response pairs (x, yd y'), then optimize it with policy gradient-based method like PPO (Schulman et al., 2017) with an entropy-bonus using the following KL-constrained RL objective
max E (s))s) W0
Σ (r) Blog (a:(s)))) + (To ~ p(so)
(2)
KL penalty
where nee is a reference policy, often resulting from supervised finetuning, from which the learned policy should not significantly deviate. However, in classic RLHF methods the reward function learned as contextual bandit with the preference model
rx,
pd ("(y" z "xxrxxx"")+exxp(ss"""
3

---

Published as a conference paper at COLM 2024
and is thus only applied at the final timestep for the last action where is EOS. In practice the actual reward used in the token-level PPO is
log Tee (ref((r)s))) r(st, at) yrxx,) + Blog (a:(s))
if 8t 1 is not terminal
if 5 = y terminal
in a maximum entropy formulation. This leads to an interesting contradiction where the reward function is treated like bandit, but the actual RL value function and optimization is done per-token in practice.
3.3 Direct Preference Optimization
Unlike classical RLHE, DPO, as derived in Rafailov et al. (2023), stays entirely within the contextual bandits setting entirely and also uses the bandit-based preference model in section 3.2. To circumvent the need for an RL algorithm, DPO uses the well-known closed form solution to the KL-contextual bandit version of the RL problem posed in eq. (2)
(Ziebart et al., 2008; Levine, 2018):
n [ylx)=
Z(x) TTe (yleeyy)
1
where π² is the optimal policy and [(x) the partition function that normalizes it. DPO re- arranges this equation to solve for reward as r(x.y) f log 7 (ylx) B log [y/) (x).. Substituting this relationship into the standard binary cross-entropy loss function used for reward modeling yields the DPO loss equation as the partition function Z(x) cancels from the Bradley Terry model.
( Reet) = - BB (x.x"."))) >DD
logo
Blog
Ty Tref Tye
x) x)
B Blog
Ty x)
]
ne (y' x)
For brevity we use o to denote the logistic function. In the next section, we show how an alternative derivation of DPO can also cast its optimization within the token-level MDP.
4 Theoretical Insights
In this section we explore how DPO can theoretically be cast into the token-level MDP, and explore the consequences of doing so. First, we provide a token level derivation of DPO under the assumptions in section 3.1. Next, show that even in the token MDP, DPO is able to fit any reward function in the multi-step Bradley Terry preference model eq. (1). Ultimately, this shows that DPO can potentially be used for more sequential optimization tasks, like multi-turn interactions or even multi-modal generation.
4.1 DPO as Q-function in the Token Level MDP
RL in the Token-level MDP. While the original derivation of DPO relies on the fact that Q* yx. rxx, y), this relationship does not hold in the token-level MDP. To resolve this, we need to develop new mathematical results that will allow us to relate the reward function in the Token-level Bradley Terry model eq. (1) the corresponding optimal policy by (Ziebart, 2010)
πt. In the general maximum entropy RL setting, the fixed point solution of eq. (2) is given
7 (" ss)) = ((' (s..a.)
(5)
where πz (a|s) is the optimal policy and Q* (s, a) is the optimal Q-function which models the total future reward from as) under π². The optimal value function V is a function of Q',
(*() log Σ e ('uaa)
(6)
a
such that the policy 7t integrates to one. Unfortunately unlike in the bandits setting this relationship gives no specific information about the reward function at single state
4

---

Published as a conference paper at COLM 2024
action pair since the optimal policy optimizes for total future returns estimated by Q- To do so, we will need to consider the relationship between Q* and r.
From to Q". The relationship between future returns and the current timestep is captured by the belmman equaitons which are satisifed by any valid Q-function. We write this below for the optimal policy πt" under the reward with KL divergence penalty:
[r(s;, at) ) + log Ter (ar St) + V (s(41),, s is not terminal (((s a:) fog (Ter(ar)5)
Q* a:)
if 6 is terminal
We can then rearrange the bellman equation for the optimal Q-function in terms of the reward. This style of relationship was first explored by Garg et al. (2022) in imitation learning and later in Hejna & Sadigh (2024) for preference-baeed RI. However, these works require the use of a discount factor y < which is typically not used in RLHF. the appendix we prove the following Lemma which shows that this relationship is indeed one-to-one in the token MDP as well.
Lemma 1. Under mild assumptions, there is a bijection between reward functions r(s), at) and corresponding optimal Q-functions Q* (st, at) in the token MDP.
This leads us to rather interesting conclusion that an LLM is always the optimal soft Q-functions for some reward function in the token MDP. Consider any LLM which outputs logits lo and temperature parameter B. As is common practice policy π to be the softmax over tokens modulated by temperature parameter & which is precisely eq. (5) where Q* because the value optimal function V is precisely & log Z (s:), normalizing the distribution. The corresponding reward function may not be smooth or well-behaved. Notably, the logits have a free parameter due the softmax. While this free-parameter results in the same optimal policy per later arguments, it means the sequence of values may not be smooth. The question then becomes how to finetune the LLM such that it is the optimal Q-function for reward function r that aligns with human preferences. To do so, we will complete our derivation of DPO in the token MDP.
take the sampling
DPO learns our best estimate of Q" Now that we have established bijection between r and Q*, we can derive a token-level version of DPO to align the implicit reward, induced by the Q function represented by the language model, with that of the best estimate of reward, according to Bradley-Terry model in eq. (1). To do so, we need to represent the sum of rewards first in terms of the Q-function Q', and then in terms of the policy πτ'. We complete the first step by inverting the Bellman equation in eq. (7) and substituting it into the sum of rewards over trajectory
[1,,a,,.....
1. ST).
T-1 Σ (Q" (s), - 0
Σ r(si, ar)
1---Blog Boog (arls)) ($++1)) =
=Q* ap) B logg nh (20(0)))))) +
Q* V*(s) B log (arr(a)(s))
The equality follows from V*(ST) = 0 and re-arranging the sum to isolate f = O. As V is written entirely in terms of Q' and B pereq. over the sequence just in terms of Q" Next, we exchange Q* for πτ. We can log-linearize eq. (5) as B log πt* (s) (*(s;, a:) (s;). This is equivalent to stating that the language model probabilities are just the softmax over lo = Q* with temperature f. Continuing from the above, with this substitution we get
eq. (6), we have expressed the sum of return
π (ai(s)) Blog Ne (a/
π (ails:) ne (a: (s:)
= Qsn,, an) - B log (an)s))
= V*( + Σ $log
where the final step results from adding and subtracting V*(sp) and applying the substitution again. Now, this representation for the sum of rewards terms of the optimal policy can be directly substituted into the preference model in eq. (1), where the V*( term will cancel just as 7(x) did in the original DPO derivation assuming Th and r/ start
5

---

Published as a conference paper at COLM 2024
at the same state so, giving us the policy-induced preference model
N-1 Σ
π (a spr) Tet a (
M
Blog 'I' (a/(ss))
Pxr (T 2 r))
Blog
(8)
Ttee (a|ss
To derive the final DPO loss function, can take the KL-divergence between the empirical preference model of our dataset pD and the preference model implied by a learned policy Prg DKL (polpp))))g) This results
(T,, D) = -E(rr)))D logo
(Z
(N-1 Σ
Blog Ht' (af"ss))) Net (a.) s
M-1
Blog ''' (a/ls/)
))
Σ
The (/(sss)
(9)
In the next section we demonstrate that DPO can learn any dense reward function in the token-level MDP.
4.2 Token-Level DPO Can Parameterize Any Dense Reward Function.
In the previous section we derived DPO using the bijection between reward functions and optimal Q-functions uniquely available in the token-level MDP. An alternative view of DPO casts as restricting the learned reward function such that belongs to the class optimal advantage functions A'
(" a) (*(s) from which optimal policy is readily obtained per eq. (5). Here we show that this restriction does not limit the class of reward functions we represent. We begin by expanding the definition of equivalency used in Rafailov et al. (2023) to the broader class of potential-based reward shaping functions:
Definition Two reward functions r(s,, at) and r' (st, at) are equivalent if there exists potential function (s), such that r' (s; a:) = r(s a:) + (((++1) (((r).
In Ng et al. (1999)'s seminal work, the authors proved that two equivalent reward functions defined per definition have the same optimal policy. By log-linearizing the optimal policy fixed point in eq. (5) and substituting in the Bellman equation from eq. (7) (Nachum et al., 2017; Watson et al., 2023), we have
nO (at St) Blog (rrr(a:)s:)
= r(s., at) + V (s:+1) - (s:).
(10)
This is precisely the optimal advantage function, where V* directly follows the form of potential shaping function. Watson et al. (2023) first used this derivation to arrive at "coherent" reward function and follow-ups arrived at the same conclusion by noting that using the advantage reward preserves the optimal policy (Knox et al., 2024; Hejna et al., 2024). Unlike prior works, however, we demonstrate that this re-parameterization also leads to the same exact preference distribution a.
Theorem 1. Given reference policy The and parameter & O all reward classes consistent with the Plackett-Luce (and Bradley-Terry models in eq. (1) be represented with the re- parameterization of the form
r(s,a) = Blog 7 (a]s) - Boog (ay()
(11)
within the token MDP where (*()s) for all terminal states.
Proof. Above we derived the invariance of the optimal policy under the re- prraaamtterrzation. The preference model can be shown to be invariant by substituting and following the same steps used to arrive at eq. (8) in the last section, by following Definition 1 from Watson et al. (2023).
Interestingly, in practice, the potential function (s;) represents the free parameter the logits of the language model. An equal shift along all logits yields the same policy, but different Q-functions and corresponding rewards. The above Theorem proves that all of these are in the same equivalence class and induce the same set of preferences.

---

Published as a conference paper at COLM 2024
5 aid 38 to 45 salary expectation
5 aid 38 to 45 salary expectation
during Interview was oftered 38 Have
during Interview was offered 250 leverage and overall its great management
Have
no
leverage and overall its a great and company Should negotiate 7
entry level position
position and company reguliate
Should
Figure 1: Credit assignment in DPO based on answer-level feedback. We provide two summaries to Reddit post about job interview. The left is the base response and the right we have introduced errors in the salary range and the position level. Each token is colored corresponding to the DPO implicit reward as expressed in Eq. 11 (darker is higher), using the trained model. We see that the model correctly highlights the erroneous statements, without much change the value of the other tokens, which indicates the ability to do credit assignment.
Moreover, this Theorem implies that we can use DPO to learn the optimal policy for any per-token reward function, provided preference queries start at the same state and end at terminal state. In addition, DPO always fits an optimal advantage function for some reward which is responsible for credit assignment. Thus, the training data determines how close the learned advantage corresponds to that of the true reward. This is in contrast to methods that estimate the reward function and then additionally employ some policy improvement mechanism. Which algorithm performs better remains largely an open or empirical question.
The above derivations cast a language model as a Q function in the discrete token-level MDP. While this interpretation does not generally hold in continuous spaces, we can extend many of our results other specially structured MDPs, like those present in diffusion. See Appendix B for more thorough treatment.
Practical Insights
In this section we discuss the empirical implications of our theoretical analysis. First, we qualitatively show that DPO can learn per-token credit assignment. Next, we use the derivations of the prior section to connect guided decoding and search-based algorithms, such as MCTS, likelihood-basee results. Finally, (for the first time), we mathematically explain the phenomenon of decreasing likelihoods during DPO training, observed in the research and industry community.
search on the DPO policy and empirically validate these
For all empirical evaluations we use the Pythia 2.8B model Biderman et al. (2023) and the Reddit TL;DR summarization dataset Stiennon et al. (2022). We use the default hyper- parameters from the original public DPO implementation, unless otherwise stated.
5.1 Does DPO Learn Credit Assignment?
In the previous section we outlined how the trained DPO policy represents an optimal Q-function for some reward that optimizes the preference equation. In this section, we evaluate qualitatively if the DPO-trained model is able to learn credit assignment from trajectory feedback. We begin with generic set of Reddit posts for the TL;DR test dataset, which we provide in Appendix C with additional examples. In our representative example the user discusses an employment negotiations situation. Two answers are shown in Figure 1. The base summary, which is correct is provided on the left. On the right we modify the summary by introducing higher-level position and a corresponding higher salary. For each token in both answers we compute the DPO reward (equivalently the advantage function or "coherent" reward (Watson et al., 2023)), r(s, a) Blog 7y (sla) B Te (s)a), where Ti as outlined in Theorem (here Ti is our DPO-trained model and Ne is the SFT model). Figure 1 each token colored proportionally to this reward. We see that the model successfully identifies the tokens corresponding to the erroneous statements, while still maintaining comparable values for the rest, which is indicates that it can do credit assignment. Moreover, model still allocates reasonable values to the rest the tokens and specifically identifies the second error "management position". This is promising sign of the ability to do "stitching"
that within the context of the first error ("250K" salary) the
7

---

Published as a conference paper at COLM 2024
Levine et al. (2020) i.e. form of combinatorial generalization from offline data. If this is the case, our findings could be significant for the use of reinforcement learning and RLHF in LLMs, particularly for compositional tasks, such code and reasoning. At the same time, in the recently introduced RewardBench Lambert et al. (2024), DPO models have demonstrated strong performance classifiers on reasoning tasks. We believe these are encouraging results, which warrant further large-scale study beyond our qualitative observations.
5.2 Connecting Guided Decoding and Search to Bikelihood-Based DPO Optimization
Recently Large Language Models have been combined with search algorithms during the inference stage Mudgal et al. (2024); Feng et al. (2024); Huang et al. (2024); Liu et al. (2023a), which have found to improve the quality of responses over standard next token decoding. Following the standard literature, these methods rely
(usually sparse) reward signal or model rg (st ar) which they use to train a separate value function Vg(s;). During inference time they deploy graph-search algorithm in the token MDP outlined in Section 3.1 to maximize the sum of rewards. Let us consider the search problem outlined in Eq. 2 with partial expansion of length K:
max r(50, ap) + Bogg (So, ao) +.... r(s),ar) + log (ee (k) + V (sx (12) 40...aa
where V* is the optimal corresponding value function. Now, if we directly substitute the reward representation from Eq. 10 into the above and considering telescoping sum, with some standard algebra, we obtain that the above objective is equivalent to
max ag....ag
V (5o) + foog xt" (2o|sn) +...+ + - Blog π (ax|sk)
(13)
where π² is the corresponding optimal policy. Now, since the starting state is fixed (it's given by the prompt) have that search algorithm based on the conservative reward function of the RLHF objective and the corresponding optimal value policy is equivalent to likelihood search on the corresponding optimal policy. We empirically verify this property in Fig. 2, which shows the win rate of DPO models trained with three different & values against the preferred summary in the test dataset. We see that 5-beam search improves win-rates by 10-15% over the base policy (1-beam), which is comparable to the value-function guided search improvements reported in Mudgal et al. (2024). Interestingly, we see performance de- grade with higher number of beams and that answers with exploding length are produced, which is a sign of reward over-optimization Gao et al. (2023); Park et al. (2024); Rafailov et al. (2024) and would explain the degradation in performance. These observations are consistent with our formulation of beam search as a search over a learned reward function.
These findings are consistent with the result of the recently proposed V-STaR algorithm Hos- seini et al. (2024), which combines the approach of STaR Zelikman et al. (2022) with a DPO trained verifer. At inference time, the STaR model produces several candidate reasoning chains (plans) which are ranked by the DPO verifier likelihood. This can be seen as form of likelihood based search as in Eq. 12, however instead of directly searching on the DPO model, it uses the STaR model as a proposal distribution. We hypothesize this is beneficial in pre- venting reward hacking, which is potentially an issue with deeper search as shown in Fig. 2.
5.3 Connections Between Proxy Tuning and Reinforcement Learning
Several recent works Mitchell et al. (2023); Liu et al. (2024a;b) have proposed an approach of inference-time model alignment through a proxy guidance model. These approaches start with (unaligned) base model Thase and proxy model 7proxy and a target distribution reference model The The inference time re-alignment of the base model is carried by re-weighting the conditional probabilities of each token:
Tprrxy (a|s:) (
([aas:) os Trase (als:) (als)(
(14)
(Tee(aass))
Under our considerations from the prior chapter, then this becomes equivalent to
TT ss)) (alse(a)s:) exp($(Q" (st, a) V (s:)))
(15)
8

---

Published as a conference paper at COLM 2024
Number beams vs. winrete on TLDR
Number of bcams vs. longth on TLDR
(8-0 05)
DRO (8=0 1)
120
DPO 1p:0
0.6
Mean cample lenggn
100
0.-
GPTA
03:0.03)
DPO (3-0.5)
Number beerss
Number bearss
Figure 2: Model performance using beam search. Left: Win rate of the model generated summaries over the preferred summary on 256 held-out test prompts from the Reddit TL;DR dataset, as evaluated by GPT 4. Right: The average answer length based on number of beams. We see exploding verbosity with more than beams, which also leads to lower model win rates, despite GPT4's well-know preference length bias.
where ((Q* a) (s;) the optimal implicit advantage from the proxy tuning model. That is our theoretical results allows us to tie the realignment approaches of Mitchell et al. (2023); Liu al. (2024a;b) to recent works which explicitly train critic models Mudgal al. (2024) for token-level decoding.
5.4 Likelihoods should decrease when using DPO.
A surface level interpretation of DPO would lead one to believe it increases the likelihood of chosen responses, while decreasing the likelihood of rejected responses. This however, does not account for a well observed phenomena in which the likelihood of the chosen responses actually decrease over time (Pal et al., 2024). This illustrated on the left half of fig. 3, which we show that when performing SFT before DPO, the implicit rewards of both the chosen and rejected response decline, though the margin between them increases. However, given MaxEnt RL framing, this phenomena may be expected.
Consider the expected log ratio (or implicit / "coherent" reward) of a policy under the reference model, which is often measured during training. Algebraic manipulation yields the following relationship:
Taass) Tro [arr(a|s)_
E Ea (ss)
Blog
KIDKI. ( (+s)(()))
(16)
At the beginning of training when π = Tref the implicit rewards are trivially zero. However at the end of training, assuming 7re + π², the KL-ddieerennce is necessarily positive, indicating that the implicit rewards must decrease in expectation to converge. This means that the average implicit rewards should go down when starting from the SFT model. In fact, on the left side of fig. 3 we show that when one does not SFT before DPO, there is little discernible trend in the average implicit reward and the implicit rewards of the chosen responses remain above zero. In fact, this trend also holds for CPL Hejna et al. (2024) for the general MDP, where the implicit rewards actually increase if SFT is not used.
One might realize that the previous analysis does not necessitate that the implicit rewards of the chosen must decrease, just that the implicit rewards must decrease on average. However, in practice it is common place (and recommended by Rafailov et al. (2023)) to SFT only the chosen responses to form Tref- For this section only we will call this choice of reference re Substituting n into eq. (16), we can see that when SFTing on the positive answers the implicit rewards of the chosen responses must go down because at convergence as [Blog n log !e
Based on this derivation and choice of nhe the likelihood of the chosen response should decrease in the process of DPO training.
9

---

Published as a conference paper at COLM 2024
TLD0 [B0.1]] mptici renard evolction SFT SFT
-0.23
-0:40
-0.75
-1.00
Chaan wARDs,
-1.25
Chaan vnart,
Ave 10 Peleceed dox 5m
Figure 3: The evolution of implicit rewards for DPO on TLDR (left) and CPL on the bin-picking dataset (right) during training. We see that when we start with SFT, reward values decrease, whereas starting without SFT causes implicit rewards to be positive for DPO and increase for CPL.
While choosing Tee = T is done in practice (Rafailov et al., 2023), does mean that DPO rec will decrease the likelihood of all data in favor of extrapolated responses, which could cause over-fitting. Moreover, now that we have provided a derivation of DPO in the token-level MDP, one might expect it to exhibit characteristics like an RL algorithm namely that the implied Q-function monotonically increases over time. However, this is not necesarily the case. Note that per analysis in Section section 3.1, DPO can be viewed adjusting the reward (or advantage) from which the optimal policy is deterministically mapped within the token-level MDP. DPO does not train policy to maximize reward, and thus do not argue about whether its implied value. functions should increase or decrease over time.
6 Discussion
In this work we formulated the DPO optimization algorithm as learning an optimal Q- function, which is represented by an LLM. This formulation and our results provide theoreti- cal justification for empirically observed DPO training phenomena, which are not explained by the original bandit formulation. We further link and unify a family of proposed new LLM search algorithms by likelihood search under DPO and show comparable empirical gains by a simple 1-line code change to using beam search. Most importantly, we show qualitative early signs that DPO is able to learn credit assignment directly from feedback data. While larger-scale empirical exploration is necessary, we believe this an encouraging early sign. Our results indicate a number of promising future directions explore:
Learning intermediate reasoning from outcome feedback: Recent works have shown promising results on that front Pang et al. (2024); Hwang et al. (2024).
Teaching language models to be an interactive conversationalists has been difficult, as RLHF is optimized as a single-turn bandit formulation. Moreover, classical methods, such as PPO, are not applicable in this setting. Recent work by Andukuri et al. (2024) has shown success in this domain using STaR and extending DPO multi-turn conversational trees is promising direction.
Multi-turn conversations:
LLM agents, such as WebGPT (Nakano et al., 2022) are equipped to take autonomous actions, such as browsing the Web and collecting information before providing an answer. The user then provides feedback based on the final output. Our derivations indicate that DPO training (on the full model trajectories) could learn optimal exploration behaviour. Recent works Song et al. (2024); Xi et al. (2024) shows promise in that direction.
Agentic LLMs:
End-to-end training of generative AI systems: Modern image generation systems, such as Dalle Betker et al. (2023) use an LLM to produce high quality conditioning before calling diffusion generation model. Also, recent long-form video generation models Hu et al. (2023); Gupta et al. (2023) combine transformer-basee auto-regressive generations with a diffusion-based decoder. Such systems could potentially be optimized end-to-end with ahybrid version of DPO. We expand on these points in the Appendix.
We believe these are promising directions for future work.
10

---

Published as a conference paper at COLM 2024
Acknowledgements
Chelsea Finn is a CIFAR Fellow in the Learning in Machines and Brains program. JH is supported by NDSEG Fellowship. This work was also supported by ONR grant N00014-22-1-2621 and the Volkswagen Group.
References
Riad Akrour, Marc Schoenauer, and Michele Sebag. Preference-based policy learning. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, 2011.
Chinmaya Andukuri, Jan-Philipp Fränken, Tobias Gerstenberg, and Noah D. Goodman. Star-gate: Teaching language models to ask clarifying questions, 2024.
2024.
Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. A general theoretical paradigm to understand learning from human preferences, 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield- Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training helpful and harmless assistant with reinforcement learning from human feedback, 2022a.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Delleen-Lawton Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac HatfeeldDoddss, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional ai: Harmlessness from ai feedback, 2022b.
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, Wesam Manassra, Prafulla Dhariwal, Casey Chu, Yunxin Jiao, and Aditya Ramesh. Improving image generation with better captions, 2023. pdf.
URL https://cdn openai http:/pppp/p/paaaaaacl.a
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling, 2023.
Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. ar Xiv preprint ar arXio:2305.13301, 2023a.
Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. ar Xiv preprint arXiv:2305.13301, 2023b.
Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: the method of paired comparisons. Biometrika, 39(3/4):324-455, 1952. doi: https:
/ //doi.org/// htfgg/10.200 /t////10.00////// 00007/2334099
Haoyang Cao, Samuel Cohen, and Lukasz Szpruch. Identifiability in inverse reinforcement learning. Advances in Neural Information Processing Systems, 34:12362-12373, 2021.
Alex J Chan, Hao Sun, Samuel Holt, and Mihaela van der Schaar. Dense reward for free in reinforcement learning from human feedback. arXiv preprint 2700022402.00782, 2024.
11

---

Published as a conference paper at COLM 2024
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran As- sociates, Inc., 2017. URL https: https://proceeddings hwwp:eceeednnnnnnniisrr......m htpppppppccppp....pp///p https//c/pppprfilll/pappppeeeeee fitess/apperr//11/// 2tle/d552ccaddd503cc11 (we00ff9ff2222 (tt.aa/ddeeccofooooaoddeedoddddd Fo pdf.
Chris Cundy and Stefano Ermon. Sequencematch:
Imitation learning for autoregressive sequence modelling with backtracking. arXiv preprint Xii:2306.005266 2023.
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep policy gradients: Á case study on ppo and trpo. arviv preprint Xrxx::2005.12729 2020.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis, 2024.
Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Dpok: Re- inforcement learning for fine-tuning text-to-image diffusion models. arXiv preprint 1Xit:2305.16388 2023.
Xidong Feng, Ziyu Wan, Muning Wen, Stephen Marcus McAleer, Ying Wen, Weinan Zhang, and Jun Wang. Alphazero-like like tree-search can guide large language model decoding and training, 2024.
Leo Gao,John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. International Conference on machine Learning, 2023.
Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, Matthieu Geist, and Stefano Ermon. Iq-learn: Inverse soft-q learning for imitation, 2022.
Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Li Fei-Fei, Irfan Essa, Lu Jiang, and José Lezama. Photorealistic video generation with diffusion models, 2023.
Joey Hejna and Dorsa Sadigh. Inverse preference learning: Preference-based rl without reward function. Advances in Neural Information Processing Systems, 36, 2024.
Joey Hejna, Rafael Rafailov, Harshit Sikchi, Chelsea Finn, Scott Niekum, W. Bradley Knox, and Dorsa Sadigh. Contrastive preference learning: Learning from human feedback without reinforcement learning. In The Twelfth International Conference on Learning Repre- sentations, 2024. URL httptttsss//ooeeeiiiieeee
Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishabh Agarwal. V-star: Training verifiers for self-taught reasoners, 2024.
Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, world Jamie Shotton, and Gianluca Corrado. Gaia-l: A generative world model for autonomous driving, 2023.
James Y. Huang, Sailik Sengupta, Daniele Bonadiman, Yi an Lai, Arshit Gupta, Nikolaos Pappas, Saab Mansour, Katrin Kirchhoff, and Dan Roth. Deal: Decoding-ting time alignment for large language models, 2024.
Hyeonbin Hwang, Doyoung Kim, Seungone Kim, Seonghyeon Ye, and Minjoon Seo. Self- explore avoid the pit: Improving the reasoning capabilities of language models with fine-grained rewards, 2024. URL
240/ 10346.
Minbeom Kim, Hwanhee Lee, Kang Min Yoo, Joonsuk Park, Hwaran Lee, and Kyomin Jung. Critic-guided decoding for controlled text generation. arXio preprint arXiv:2212.10933, 2022.
12

---

Published as a conference paper at COLM 2024
W Bradley Knox, Stephane Hatgis- Kessell, Serena Booth, Scott Niekum, Peter Stone, and Alessandro G Allievi. Models of human preference for learning reward functions. Trans- actions on Machine Learning Research, 2023.
W Bradley Knox, Stephane Hatgis-Kessell Sigurdur Orn Adalgeirsson, Serena Booth, Anca Dragan, Peter Stone, and Scott Niekum. Learning optimal advantage from preferences and mistaking it for reward. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 10066-10073, 2024.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khy- athi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. Rewardbench: Evaluating reward models for language modeling, 2024.
Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-t--image models using human feedback. ar e-prints, pp. arXiv-2302, 2023.
Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review, 2018
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems, 2020.
Jiwei Li, Will Monroe, and Dan Jurafsky. Learning to decode for future success. arXiv preprint ar (ix:::111106666 70106449, 2017.
Alisa Liu, Xiaochuang Han, Yizhong Wang, Yulia Tsvetkov, Yejin Choi, and Noah A. Smith. Tuning language models by proxy, 2024a. URL https://arxiv.ort www.:/s//aaa////eeeeeooe 08565.
Jiacheng Liu, Andrew Cohen, Ramakanth Pasunuru, Yejin Choi, Hannaneh Hajishirzi, and Asli Celikyilmaz. Don't throw away your value model! making ppo even better via value-guided monte-carlo tree search decoding, 2023a.
Jiacheng Liu, Andrew Cohen, Ramakanth Pasunuru, Yejin Choi, Hannaneh Hajishirzi, and Asli Celikyilmaz. Making ppo even better: Value-guided monte-carlo tree search decoding. Xiu preprint ar Xivv22309.15023 2023b.
Tianlin Liu, Shangmin Guo, Leonardo Bianco, Daniele Calandriello, Quentin Berthet, Felipe Llinares, Jessica Hoffmann, Lucas Dixon, Michal Valko, and Mathieu Blondel. Decoding- time realignment of language models, 2024b. URL https:
/40ss//222..00 02992.
Eric Mitchell, Rafael Rafailov, Archit Sharma, Chelsea Finn, and Christopher D. Manning. An emulator for fine-tuning large language models using small language models, 2023.
Sidharth Mudgal, Jong Lee, Harish Ganapathy, YaGuang Li, Tao Wang, Yanping Huang, Zhifeng Chen, Heng-Tze Cheng, Michael Collins, Trevor Strohman, et al. Controlled decoding from language models. arviv preprint ar XrX::2331011702, 2023.
Sidharth Mudgal, Jong Lee, Harish Ganapathy, YaGuang Li, Tao Wang, Yanping Huang, Zhifeng Chen, Heng-Tze Cheng, Michael Collins, Trevor Strohman, Jilin Chen, Alex Beutel, and Ahmad Beirami. Controlled decoding from language models, 2024.
Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the gap between value and policy based reinforcement learning, 2017.
Reichhro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser- assisted question-answering with human feedback. arxiv preprint ar Xit:21122.0933 2021.
Reichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Brosser-assisted question-answering with human feedback, 2022.
13

---

Published as a conference paper at COLM 2024
Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward trans- formations: Theory and application to reward shaping. In loml, volume 99, pp. 278-287, 1999.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Chris- tiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 27730-27744. Cur- ran Associates, Inc., 2022. URL httpss:/pproceeedii neurips. cc/ wwtppes.cepppp..ppppeerr.eeee ooaa pdf.
Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with dpo-positive ar Xiv preprint ar Xit::2402.13228 2024.
Sarah Pan, Vladislav Lialin, Sherin Muckatira, and Anna Rumshisky. Let's reinforce step by step. Xrx preprint 1XXi::3111.0822 2023.
Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization, 2024. URL https://araa
19t/
Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization, 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023, URL 18290.
http:://arxxw.org/bbbss..
Rafael Rafailov, Yaswanth Chittepu, Ryan Park, Harshit Sikchi, Joey Hejna, Bradley Knox, Chelsea Finn, and Scott Niekum. Scaling laws for reward model overoptimization in direct alignment algorithms, 2024. URL https://arxi: org/ org./bs//////// /b26/ 02900.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.
Charlie Victor Snell, llya Kostrikov, Yi Su, Sherry Yang, and Sergey Levine. Offline rl for natural language generation with implicit language q llerring. In The Eleventh International Conference on Learning Representations, 2022.
Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin. Trial and error: Exploration-based trajectory optimization for Ilm agents, 2024.
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback, 2022.
Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purush- walkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization, 2023.
Joe Watson, Sandy Huang, and Nicolas Heess. Coherent soft imitation learning. In Thirty- seventh Conference on Neural Information Processing Systems, 2023. URL https:// het/Fforum?ickkcCdddaauu.
Aaron Wilson, Alan Fern, and Prasad Tadepalli. A bayesian approach for policy learning from trajectory preference queries. In Advances in Neural Information Processing Systems, 2012.
14

---

Published as a conference paper at COLM 2024
Zhiheng Xi, Yiwen Ding, Wenxiang Chen, Boyang Hong, Honglin Guo, Junzhe Wang, Dingwen Yang, Chenyang Liao, Xin Guo, Wei He, Songyang Gao, Lu Chen, Rui Zheng, Yicheng Zou, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu, and Yu- Gang Jiang. Agentgym: Evolving large language model-based agents across diverse environments, 2024. URL pettps://ardiie org/ abs/2406. 04151.
Kevin Yang and Dan Klein. Fudge: Controlled text generation with future discriminators. arviv preprint arXiv:2104.05218, 2021.
Zishun Yu, Yunzhe Tao, Liyu Chen, Tao Sun, and Hongxia Yang. B-coder: Value-based deep reinforcement learning for program synthesis, 2024.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476- 2022.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter Liu. Slic- hf: Sequence likelihood calibration with human feedback. ar preprint ar arXiv:2305.10425, 2023a.
Zhiyua Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, and Conghui He.
Beyond hallucinations: Enhancing ivlms through hallucination--wware direct preference optimization. arv preprint (rXiv:2311.16839, 2023b.
Brian Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Carnegie Mellon University, 2010.
Brian D Ziebart, Andrew L Maas, Andrew Bagnell, Anind K Dey, al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pp. 1433-1438. Chicago, IL, USA, 2008.
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human prefer- ences, 2020.
15

---

Published as a conference paper at COLM 2024
A Proof of Lemma
Lemma 1. For a fixed policy π, there is a bijection between reward functions and corresponding optimal Q-functions (Q*) in the deterministic tree-structured LLM MDP.
Proof. Let Q denote the optimal Q-function for reward We prove the statement directly, starting with the injective case.
Assume there exists a reward function r' + r such that Q Qr Then, there must exist state action pair such that r' (s; a+) + r(s;, ay). In fact, proceeding backwards from leaf node (terminal state), there must be a first state action pair (s;, where r' ar) # r at). The Q functions this location are
º (st, a:) ('s,,, ar) (s-1), (s;, = r(s;, + V, (s1+1)
By the fact that this was the first location where the reward functions differed starting from leaf node, we must have that (5(+1) V, (S++1). This is because we can recursively solve for the optimal policy, value, and Q-function using eq. (5) eq. (7), and eq- (6) from Ziebart et (2008). The rewards in all possible future states from s, a are equal by virtue of this being the location of the first difference and thus the dynamic programming solution up to this point the same. Thus, we can see that Q (St, ar) # Q" (st, a;), completing this direction. Note that this proof does not hold in general MDPs, only the token MDP where is impossible to return to the same state after taking any number of actions.
The surjective direction is easier. For all Q', Q" (s;, a,) - V* (5;+)) under deterministic dynamics. Thus, we can see that the mapping is surjective.
compute reward function r(s), ar)
16

---

Published as a conference paper at COLM 2024
B Treatment of Diffusion Models
Conditional diffusion image generation models, such as Stable Diffusion 3 Esser et al. (2024) have also used form of the DPO algorithm outlined in Wallace al. (2023). Our analysis can no longer be directly applied in that setting, since the generations are continuous. However, we could translate many of our results to that setting, if we consider certain diffusion MDP. We outline our results bellow.
B.1 Diffusion MDP
We borrow the denoising MDP formulation from Black et al. (2023b); Fan et al. (2023). We again have the tuple (S, frr.f.r,poll with the same formulation in Section 3.1. At the same time consider a diffusion process with time index and T total steps, conditioned on context and image denoted by x,. Then we can map the diffusion generation process an MDP in the following way
(c, (C, otherwise
if
That is the initial state consists of the prompt c and afterwards each state consists of the current denoised image and time step T - Notice that the time-steps in the MDP are inverse to the direction of the diffusion process (i.e. we start noise and end the final image). The action is just the next image iteration, from where the dynamics is also straightforward:
a; - XT-1+1
f(s: = (c, T 1)) (c. at, a 1-1
Notice that in this case the policy is stochastic, but the dynamics of the MDP is still deter- ministic. Finally, the initial distribution, is just the distribution of prompts:
((s) A (p(c).0)
B.2 Theoretical Results for the Diffusion MDP
Given the above formulation, we also prove that Lemma 1 also holds in the diffusion MDP.
Lemma Under mild assumptions, there is a bijection between reward functions r(s;, at) and corresponding optimal Q-functions Q* (st, at) in the diffusion MDP.
Proof. Since the MDP still has deterministic dynamics, we have that Eq. 5-7 still hold. Now, given reference policy Tee parameter and a critic Q we can trivially recover the unique reward function by inverting Eq. 7. We will prove that given a reward function r(s,, ar), can recover unique critic Q. We work inductively in the diffusion MDP starting with t = T, where we have V (ST) = for all terminal states. We then have that
(1--11)))-1) Q* (c, (c, f+ 1), xT-1)
r(s: (c, (c, 1), x)-1) foo Pref (XT 1+ T-
$log /
where Are is the reference backwards diffusion process. In this case even though the state space is deterministic, our approach to the proof of Lemma 1 still holds by using backwards induction on the diffusion step f. Notice, that from V(sT = (c, 0)) = 0 we can uniquely determine the critic values for all states at time step T- 1. Proceeding inductively backwards through time in the MDP/ denoising process (forward in the diffusion process), obtain the desired result.
17

---

Published as a conference paper at COLM 2024
Given the proof of this Lemma, can then directly apply the results of Section 4.2, inchuding Theorem 1. Our results, also give us insights into the formulation of Wallace et al. (2023). In particular, by changing the sampling scheme of the intermediate diffusion the authors obtain two alternative formulations (Appendix S2 in Wallace et al. (2023)). Both of these schemes suggested empirical approximations in the formulation of the Diffusion-DPO algorithm, however in the view of Q-learning both of these are valid approaches to generating off-policy data. In fact, our interpretation of DPO allows for general off-policy data sampling and aggregation methods, which could yield whole family of DPO algorithms in this domain. We leave the exploration of this direction for further work.
18

---

Published as a conference paper at COLM 2024
C Reddit TL;DR Posts
SUBREDDIT: r/running
TITLE: Tips on getting back into running after years of not doing so & shin splints
POST: Hey everyone, was hoping to gather some tips from people who left running and had to start over. A semi-lengthy background on myself help you understand where am coming from. In high school I was very good cross country runner, running from 35-50 miles week and never slower than 8-9 minute miles. At the end of senior year, planned on taking break from running and then try to race half or full marathons in the spring. I ended up not running all after xc. 4 years later, I was noticing how much miss the sport (especially after seeing the success of xc friends) back into it. But the only group at my university that could find was a triathlon club. joined them, but only did the running workouts. After about 4 weeks, developed shin splints. This is because haven't ran in years but thought miles was ok after weeks. Also, being 25 pounds heavier didnt help. After taking 3 months off and training on the bike and in the pool, finally was back to running in february. but my shinsplints was still around. I finished my first sprint triathlon last week, and have been trying to get miles back under my feet again. I havent felt shin splints severely since the beginning of March, but I can feel it looming around. After half year of it, I am getting really really frustrated. I cant run more than 4 miles still and my fastest mile is 8 minutes. know will probably never run like did when I was 17, but its difficult because of remembering what used to be capable of running.
decided to join running group to get
Was very good runner in high school stopped running for 4 years started spi
Was very good runner in high school stopped running for years started again recently but still knee frustrated trying to get back
again recently but still have frustrated trying to get back to where used to be
usted
to where used to be
DPO successfully identifies the wrong tokens ("busted knee") in the right summary and correctly classifies this pair.
19

---

Published as a conference paper at COLM 2024
SUBREDDIT.r/ssRRee
TITLE: What is sub-$800 camera that can shoot high quality video ideal for music video- like appearances?
POST:
[This is a video of what we're trying to achieve.]
My school currently has Sony HVR-HD1000u, and compared to that, our videos are nowhere near as good. understand that things like lighting and color correction play pretty big role, but even then feel like our videos are never that clean. usually can't get 720p clips out of our camera and the slow motion that they have is something we can't even come close to.
One possible "problem" is that for some reason we can't use firewire to connect the camera to the computer so have to play the tape this thing that basically plays it and then capture the tape playing. feel like this is probably a huge problem because it's like trying to show friend movie by screen-capping from Skype.
SO, should we scrap the HVR-HD1000u and get a Canon T2i (a cheaper DSLR which from the samples I've seen on YouTube and clips from that video, seems pretty high quality), or continue trying to use the Sony?
To get music video quality would
TO get music video quality would
it be better to use our Sony H VR
it be better to use our Sony H VR
HD 1000 u buy Canon 2
HD 1000 buy an iPhone Or entirely
something else entirely (< $
something
DPO successfully identifies the wrong tokens ("iPhone") in the right summary and correctly classifies this pair.
20

---

Published as a conference paper at COLM 2024
SUBREDDIT: p/rrsonaliinn
TITLE: When asked about salary expectations during my interview I said 38k to 45k. Was just offered the position with 38k. Should try and negotiate?
POST: So interviewed for a position last week, and before the interview saw online that the industry average for this position was $41,000. During the interview, they asked me my salary expectations, said between $38,000 and $45,000 hoping it'd land somewhere in the middle. received my offer today, and it was for $38,000. I can't help but wonder if had just said $41,000 they probably would' 've offered it...
Anyways, what I know they hiring 3 other people for this same position.... either got lucky and guessed exactly what salary they were planning on paying all of us to begin with, or we're all getting paid differently. As for the job, is the ideal entry level position for me right now, and is great company with benefits etc so actually wouldn's 't mind working there for the 38k salary.
But it would be nice to get an even 40 at least, so my question is, is it common practice to negotiate salary after receiving an offer already? I also must say that don't have any leverage as this is entry level and would have probably still accepted had the offer been even as low as 30k. As such, I'm very afraid the offer may be retracted I do try and negotiate, if that sort of thing happens?
38
salary expectation
5 aid 38 to 45 salary expectation during Interview was offered 250 Have no leverage and overall its great
Have no leverage and overall Its a great
during interview was offered
entry level position and company
management position and company Should
Should negotiate
?
regutiate
DPO successfully identifies the wrong tokens ("250k" and "management position") in the right summary and correctly classifies this pair.
21

---

Published as a conference paper at COLM 2024
User Feedback
Internal Generation
phononddiisto image of mi minooour walling down Fifh Jvenue in New York City It with
dene upper oody
powerttll lege of bud. This minctturis cressed modern canual cisthng asoroprate icone New no City landmarks, including yellow cabs, skylcnpprrs and bustling dewallk,, capturig esvence city
User Prompt
Generate photorealistic picture of a minotaur
DALL-E
New York City!
phottoralistic image a -iease mythical cresture with the body a mar and Plea York through bustingg caim an collected amadet the bu human crount dkporrppes rking high the mnnotae's strenigh paree the puople around it under a clear blue say.
Figure 4: Example of an end-to-end generative AI workflow. The user request an image of Minotaur in the streets of New York. However, we see that the rejected image does not actually represent Minotaur, but bull. The prompt refiner generates a valid description and specifically includes the phrasing "the body of man and the head of bull", but the image generation model fails to follow that prompt. At the same time in the case of the chosen image (which does reflect the prompt) the prompt refiner generates a more descriptive text, which the image generator is able to follow more closely. While it is possible to train each component separately, joint training can in theory, optimize the refiner to generate prompts that the image generator can more closely realize, while also training the generator to directly produce more aligned images.
D End-to-End Training of Generative AI Systems
End-to-end system: Here, we present outline for end-to-end training of generative AI systems from human feedback using DPO in multi-step MDP. We assume two models a prompt refiner (z)z))), which is a language model, generating discrete tokens in an autoregressive way, which, given a prompt produces a refined prompt z. This prompt is then fed into an image- diffusion model πt my(ylz), (which we parameterize as denoising model EØ), which generates the final image. A real case example of that system is shown in Figure
Feedback generation: During the feedback gathering stage, a user provides a query x and two refined prompts are sampled from z z² ग.p (x(z,, which the user does not directly evaluate. Then, the image generation model generates two images y' ()l)zz)) for i 1,2. The user then provides a preference over the y' to yield the preference pair 1y" > |x}
Optimization: Optimization is carried in a hybrid MDP, where the initial state is the prompt x and has the same form as the token MDP, as outlined in 3.1. When the EOS token is encountered in that MDP, at which point the transition dynamics switches to the diffusion denoising MDP introduced in Black et al. (2023a). Notice that this is still a valid MDP and all the conclusions of our derivations in the main section of our paper hold. We could then optimize this system end-to-end using a hybrid DPO objective, combining our results, with those presented in Wallace al. (2023) we have:
Ie x, 20 si
re y B
log
not 2 X, zz
prompt refiner MDP
TTu(A:) (1 Cref YY" 2 2z1,) - - 21111
(17)
diffusion MDP objective
22

---

Published as a conference paper at COLM 2024
where & and "y are two separate discounting factors for each modality. Here the diffusion objective follows directly from the derivation Eq. 9 and the sampling scheme proposed in Wallace et al. (2023) (Eq. 12-14 in that paper). Notice here that the imgge generation model is conditioned on the corresponding refined prompt zz We can define r(x', y') similarly and optimize the DPO objective:
lloger (re,(("",)) Te,p - (x,, (2x,""))]
LDPOR# - - (xy
(18)
We demonstrate a particular real use of such a system in Figure 4. The user request an image of a Minotaur in the streets New York. However, we see that the rejected image does not actually represent Minotaur, but bull. The prompt refiner generates valid description and specifically includes the phrasing "the body of a man and the head of bull", but the image generation model fails to follow that prompt. At the same time in the case of the chosen image (which does reflect the prompt) the prompt refiner generates more descriptive text, which the image generator is able to follow more closely. While it is possible to train each component separately, joint training can in theory, optimize the refiner to generate prompts that the image generator can more closely realize, while also training the generator to directly produce more aligned images.
D.1 Hybrid Video Generative Models
recent line of work on long-form video generation Hu et al. (2023); Gupta et al. (2023) by combining auto-regressive transformer generation with diffusion model decoding or uspcaling of the actual video frames to obtain temporally consistent and high-fidelity generations. We could deploy similar RLHF pipelines the rideo-generation problem as well. It is also straightforward to extend the DPO joint optimization framework, presented in the previous section in Eq. 18 to this stetting as well. Instead of textual prompt refiner tokens, the variables would represent the latent token generations of the autoregressive component Tg, which would be decoded into actual video frames y via the diffusion decoder Ng.. We believe this is exciting direction to pursue for the emerging video generation technologies.
23

---

Published as a conference paper at COLM 2024
Number of beams vs. winrace on TLDR
Number of beams vs. length on TLDR
PD (GP14)
150
140
OPF wirretes
0.3
Near
110
100
Number of sears
Figure 5: PPO model performance using beam search. Left: Win-rate model summaries over preferred summary. Right: Average answer length based number of beams.
E
Beam Search Trends for PPO
We consider whether the beam search trends in Figure hold for PPO, since PPO is known to exhibit reward over-optimization well (Gao et al. (2023)). We
PPO-tuned GPT-J-6B
model released by CarperAll, fine-tuned sampling parameters as with the DPO experiments (T = 1.0, k = 50) and generate 256 samples from test set prompts, with GPT-4 as the evaluator. We only report results for 2, 5, and 7 beams; higher number of beams were tried but exhausted memory on an NVIDIA A40 GPU.
the same TL,DR dataset. We use the same
In Figure 5, we observe similar over-optimization phenomenon with DPO, with beams both increasing winrate and decreasing sample length (even under the length- biased evaluator GPT-4). However, more than 2 beams leads to decline in downstream performance and large uptick in sample length, similar behavior DPO in the over- optimization regime. We find that the benefits of increasing the beam size are more limited when using this model.
Vtttesss/lagggiaaaaaeccc/////rrp summarize tldr ppo
24

---

Published as a conference paper at COLM 2024
F
TL;DR Sample Generations
E1 Samples Across Different Beam Searches
All samples generated from the p = 0.1 checkpoint.
SUBREDDIT: r/ AskReddit
TITLE: Question regarding learning/teaching
POST: I apologize in advance if this is not the right sub reddit for this kind of question.
for the past years, i've been slowly but surely developing my illustration skills. I am 18, recently out of high school, and will be going to college for bachelors in illustration in likely 1-2 years.
Despite teaching my self to be better, by using resources from the internet ext, but improvement has been slow. It can be seen, but still feel am behind where should be at my age, and for how long have been drawing/ drraing/ppintiing painting.
I've met many artists who are self taught, and never went to college at all for what they do, yet i have it in my mind that college will help me greatly, since learn faster when taught by another, rather then myself.
My question is, is this actually possible? do some people just have harder time of teaching theirself? have the dedication and interest to pursue education, but i respond so much better to being instructed, rather than instructing myself. Is this normal?
Num beams 1. TL;DR: want to continue my art education, but still be able to teach myself as much possible, is it possible to do both at the same time? appreciate all inputs.
Num beams = TL;DR: 18, recently out of high school, and will be going to college for bachelors in illustration in likely 1-2 years. Slow but steady improvement in my drawing/ paintiinninnggggg skills, but respond better to being taught rather then instructing myself. Is this normal?,
Num beams = 20. TL;DR: 18 year old self taught myself to be better at drawing ppanntin over the past 2 years, but still feel am behind where should be at my age, and for how long have been drawing/ haangg/paaning.... Do some people just have a harder time of teaching themselves? Is this normal?
25

---

Published as a conference paper at COLM 2024
SUBREDDIT: r/legaladvice
TITLE: Mom was hurt at work, think it's something that is a little past worker's comp. Any advice?
My mom works at an old navy for something upwards of almost ten years. Over the span of her working their they have been lingering in and out of being abusive employers. Not that this isn't I something I wouldn't expect out of a big corporation, something happened recently that I'm close to positive should be lawsuit worthy something in my mom's favor.
POST:
The other day on the clock my mom was cleaning one of the bathrooms and while taking the garbage out there was a faced up syringe under some paper towels! She found out after jabbing her hand with the syringe and noticed that there was preexisting blood on this syringe!!! She was then brought to the hospital (obviously) to be screened for all sorts of diseases and had the option of starting intense medications prevent the microscopic chance of that needle being contaminated with HIV though the medications would make her very ill. All old navy did about the situation was lock the bathrooms so the public can't use them.
Is there anything about this situation that my mom can use in her favor? She didn't turn the syringe over to Old Navy because they were being very nonchalant and dismissive about the situation.
Num beams = 1. TL;DR: mom is a working at old navy, had a bloody syringe found under some paper towels on the garbage disposal, likely contaminated with HÍV/AIDS Any help in what she can use in her favor in future lawsuits or otherwise?
Num beams = 5. TL;DR: Mom found syringe with preexisting blood under paper towels at work. Old navy was dismissive and locked the bathrooms so the public couldn't 't use them. Is there anything my mom can use in her favor? She didn't 't turn the syringe over to Old Navy because they were being very nonchalant and dismissive about the situation.
Num beams = 20. TL:DR: Mom found a syringe in the bathroom at work that had preexisting blood on it. Old navy was dismissive about the situation and locked the bathrooms the public couldn't use them. Is there anything my mom can use in her favor? She didn't 't turn the syringe over to Old Navy because they were being very nonchalant and dismissive about the situation. A: My mom works at an old navy for something upwards of almost ten years. Over the span of her working their they have been lingering in and out of being abusive employers. Not that this isn't something wouldn't 't expect out of big corporation, something happened recently that I'm close to positive should be lawsuit worthy or something in my mom's 's favor. The other day on the clock my mom was cleaning one of the bathrooms and while taking the garbage out there was a faced up syringe under some paper towels! She found out after jabbing her hand with the syringe and noticed that there was preexisting blood on this syringe!!! She was then brought to the hospital (obviously) to be screened for all sorts of diseases and had the option of starting intense medications
26

---

Published as a conference paper at COLM 2024
SUBREDDIT: r/self
TITLE:
Always seize the opportunity to help others
This is my first post so here's little intro about what do for work. I'm security guard. work the graveyard shift in the downtown part of my city. Nothing special. Being security guard is easy money.
POST:
Anyways, while making my first round I noticed a larger man a wheelchair across the street trying to make it onto the curb. The transition for the handicapped access wasn't smooth enough and he was stuck. crossed and pushed him onto the sidewalk.
He needed to go to the hospital five blocks the road. called my supervisor and said I'd be back in a few had to help this guy. pushed him to the hospital and walked back.
If had my headphones in like any other day, wouldn't 't have seen him and he'd be struggling to get there.
I'm not sure what was wrong but he had one arm and one leg missing.
Num beams = 1. TL;DR: I helped man who needed emergency attention on the street by pushing him for five blocks because I had headphones in. I didn't see him until after had to walk back.
Num beams 5. TL;DR: I saw handicapped guy stuck on the sidewalk. I pushed him to the hospital. If had my headphones in wouldn't have seen him and he'd ''d be struggling to get there. I'm not sure what wrong but he had one arm and one leg missing.
Num beams = 20. TL;DR: I saw handicapped guy stuck on the sidewalk. pushed him to the hospital. If I had my headphones in, wouldn't have seen him and he'd be struggling to get there. I'm not sure what was wrong but he had one arm and one leg missing.
27

---

Published as a conference paper at COLM 2024
F2 Samples from DPO with and without SFT Pre- tre-Trainnng
SUBREDDIT: r/Parenting
TITLE: Young adults/ reddit parents: how have you found ways of asking a parent to loan you money and figure out a way of paying it back? Diplomatically.
POST: know this question has lot to do with the kind of relationship one has with each parent and at what stage in life you are personally and professionally.
I'm 27, moved two states away for grad school, and i'm living strictly off loans. It's my first semester and I'm taking 12 hours. I'm an ex-teacher, so have some professional background. No savings. I've travelled quite bit though. Shooting for an assistantship /internship in the Spring. I've never been this low on funds. It's also scary realizing how much deeper I am putting myself into the dark debt hole of student loans.
I'm pretty much living on credit card for the next two months and am too ashamed to ask my parents for any money. can nanny here or there, but it takes time and effort (yes, like everything) to find families needing a babysitter in giant college town. I will get organized enough to pay my father back, of course. He's already loaned me close to $1,000 in the past four five months and I feel little disgusting since he has a hard time making any income himself. Mother another story-she's s got really tight grasp on her pocket and a little aggressive when it comes to financial advice. So I never approach her.
My dad is the kindest man know and has the cleanest reputation among his contacts, which is why people trust him and partner with him much. His kindness does get the best of him however, sometimes to his detriment. In other words, people have walked all over him in the past. Mom, the other hand, is constantly reminding me about payments, asking me about my loan, really approaching things in a way that push me far from wanting to face finances. Fabulous parenting. It's always been this way, with anything growing up. Dad never seemed to say no, he always gave me options/ / choices. Mom was a straightforward, dry, clear, British NO. She has always said I don't 't have my feet firmly grounded. I'm getting off on tangent now so I'll stop.
With SFT. TL;DR:
Since inherited money and owe my father a significant chunk of it, my mother doesn't allow herself to see any sense in dealing with my finances. She constantly reminds me need to pay my loan, yet seems to be little aggressive and pushy about things. How do I confront that without making my mother uncomfortable?
Without SFT. TL;DR: [CHINESE SCRIPT]
28

---

Published as a conference paper at COLM 2024
SUBREDDIT: r/ t/skReddi
TITLE:
[Serious] How can I tell my mother politely, that she is dealing with our dog in an inappropriate way?
POST: My mother watched Cesar Millan a lot before we got our dog, and she acted like she knew everything there was to know about training and raising dog.
Its been about 2 years with our dog, and he has a problem with barking at the door when people go to open it (doesn't seem like that much of problem to me). My mother's reaction is to yell at him, or he does it sequentially, she will strike him in the thigh/ rear end.
Whenever step in and try and take control of the situation, she will get mad, and try and tell me how to handle this kind of thing, because she thinks she a dog expert. (This upsets me because it feels like I am being talked down to)
The last strand was crossed today, when was in the basement watching tv, heard my dog bark at the door, and then my mother get up and yell at him. He ran into sun room, which is above the basement, and I could hear her hit him, and his paws skid on the floor. I went up check on him (he was fine, no damage, happy little guy when he's with me) and my mother yelled at me to put him back in the room where he to stay for long long timebecause tried to bite her when she was disciplininghim.
Her solution to my dogs problem is completely inappropriate, tristtate and no way to treat dog. I believe positive reinforcement, and am getting very frustrated listening to her complain about my dog, and her inappropriate ways of dealing with his barking.
So anyone knows a way can tell her what she is doing is wrong, or if there a way can try and get my dog to stop barking at the door, please let me know.
With SFT. TL;DR: My mother's 's ways of dealing with our dog's barking problem in- appropriate and is getting worse. How can I tell her otherwise, or get her to change her behavior?
Without SFT. TL;DR: |CHINESE SCRIPT]
29

---

Published as a conference paper at COLM 2024
SUBREDDIT. r/legalaviice
TITLE: Speeding down highway, Can I get ticket?
POST: 18, Male, Tennessee.
I was speeding, like 120 in a 65. Some girl kept trying to speed up with me shit car, drive turbo g35 it was no match but she wouldn't budge she'd d push her little car around trying to pass me, and obviously putting her life in danger, I'm no godly driver but know my car, and the road. knew what I could do speed wise, and I constantly took it back to around 80 before she'd speed up. Eventually had to turn, and she held up her phone, and piece of paper, assumed it was her number but now that I think about it she probably got my tags and plans to report or something, want to know could get in trouble?
On mobile.
With SFT. TL;DR: speeding 120 in a 65, girl tried to speed up with me in shitty car, had phone with street redsess on and number for police, could get in trouble for following too close.
Without SFT. TL;DR: Female driver started tailgating, almost hitting in my Then tried to make passes. know my car works well, but she wanted to prove me wrong, even though I'm pretty sure I'd outrun her at lower speed. She constantly kept slowing me down so I'd be at speed she could pass me. But after getting
30

---

Published as a conference paper at COLM 2024
G Dataset and Hyperparameter Details
The TL;DR dataset contains 64,832 summary comparisons and is derived from the Webis TLDR dataset, with human feedback collected by OpenAL. Each example is scraped from Reddit and belongs to one of several "subreddits" (topic forums), with an associated sate/posthhuman TL;DR (summary). Around 5% is held out for validation.
Each TL;DR model was pre-trained Pythia 2.8B model. We first SFT with learning rate of 0.5 x 10 6 and batch size of 128 for one epoch using 8 gradient accumulation steps NVIDIA A40 GPUs. All DPO models with the same setup and hyperparameters as the original paper (Rafailov et al., 2023). Specifically, we train for 1 epoch with learning rate of 0.5 x 1 and linear warmup of 150 steps. We use batch size of 32 (example example is positive and negative) and clip gradient norms to 10. We use ß = 0.1 for DPO.
All generation samples performed with temperature 1.0 and max length 512 unless otherwise specified. Evaluations (winrates, lengths, ett) are computed with 256 samples from the held-out set in TL;DR.
31

---

Published as a conference paper at COLM 2024
H GPT 4 Evaluation
We use the following prompt in all our GPT evaluations. 'hee order of the model sample and the reference response is randomized in each evaluation to avoid positional bias in the judge.
Which of the following summaries does better job of summarizing the most important points in the given forum post?
Post: QUERY
Summary A: A
Summary B: B
FIRST provide a one-sentence comparison of the two summaries, explaining which you prefer and why. SECOND, on new line, state only "A" "B" indicate your choice. Your response should use the format:
Comparison: jone-sentence comparison and explanation Preferred: i"A" or "B"
32


Reviews:
[{"review_id": "7b6Qp67XBs", "reviewer": "Reviewer_UQbN", "summary": "This paper compares the classic pipeline adopted in the GPT series with DPO, a more recent method proposed by the research to overcome the complexities and intricacies of the RLHF pipeline with PPO. The main issue with DPO is that it is formulated as a contextual bandit problem where the entire response is considered as an arm ignoring the fine-grained decision-making process underlying language modelling and its relationship with credit-assignment. \n\nAs a first contribution, the authors demonstrate that it is possible to reframe DPO in the token-level MDP which gives the ability to shed some light on the credit assignment ability of this method. Additionally, because it is now possible to define a Q-function in terms of single tokens, it is possible to formulate search-based algorithms (such as Monte-Carlo Tree Search) via likelihood-based search implemented as beam-search. Finally, thanks to this theoretical framework, the authors study a well-known problem in DPO training which causes implicit rewards to decline during training and conclude that this is a result of the DPO training process (i.e., this is a feature of DPO and not a bug).", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the detailed feedback and kind words!\n\n**“Section 5.1 reports a case study..”**\n\nWe follow the same training procedure as original DPO work, including SFT pre-training and DPO fine-tuning including the official codebase and default hyperparameters, besides varying the beta coefficient, which we report in our results.\n\n**“Again connected with the previous point, the evaluation feels very anecdotical…”**\n\nWe appreciate your point! Note that we have included additional examples in the Appendix. Furthermore, additional evaluations on TL;DR have exhibited similar trends -- we can include more of these examples in the final paper. \n\nWe believe larger scale studies are indeed warranted, but face challenges:\n\n1. Quantifying credit assignment in reinforcement learning in general is challenging, but becomes increasingly so in free-form text generation, where overall quality might not be attributable to specific tokens but larger scale features of the writing.\n2. Datasets for such analysis are quite limited. \n\nOne potential domain where credit assignment might be easier to evaluate is multi-step reasoning tasks. Recent work [1], trains a DPO implicit reward model as a verifier for sequence-level reasoning. It is possible to compare step-level results from such a verifier to data with explicit step-level annotations [2]. We believe this is a valuable study, but in of itself can be quite expansive work to correctly train such models (which has warranted several publications in its own right) and is beyond the scope of our current work!\n\nWe would appreciate it if the reviewer has any particular suggestions on this front!\n\n**“Did they try completing this step for other open-source models...”**\nWe only considered the 2.8B Pythia model due to compute constraints, but we believe our results will transfer to stronger models as well.\n\n**“Does a model trained only with SFT behave in a similar fashion…”**\n\nOur credit assignment experiments all evaluate the DPO implicit reward, as outlined in Section 4.2. As such in of itself the SFT model does not induce a directly comparable reward function.  \n\n\n[1] V-STaR: Training Verifiers for Self-Taught Reasoners, Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, Rishabh Agarwal, 2024\n\n[2] Let's Verify Step by Step, Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, Karl Cobbe, 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Really well-written paper with a strong theoretical foundation that clarifies many of the issues around DPO and RLHF training\n2. This paper represents a must-read for many practitioners and researchers interested in training LLMs but I believe it can extend further to other applications as well. In fact, the authors highlight LLM agents as a promising avenue for this work because in this domain the agent has to generate very long trajectories and sources for credit assignment are limited considering that they might happen only at the end of the trajectory (i.e., when the task is finished).", "reasons_to_reject": "Overall, I think this paper represents a great contribution to the community and in general can be considered as a great work for many to learn from. However, I was a bit disappointed by the level of detail of the case studies reported in Section 5 which I will list below. Additionally, I will report more details and suggestions for authors in the \"Questions for Authors\" box.  \n\n1. [Section 5.1 - training details] Section 5.1 reports a case study on text summarization to study the ability of DPO to learn credit assignment but it somehow lacks some details regarding the concrete steps involved in training this model.\n2. [Section 5.1 - credit assignment comparison] Again connected with the previous point, the evaluation feels very anecdotical. I believe this credit-assignment evaluation is valuable but it should be executed in a more principled way.\n3. Overall, all the sections involving some training should report more details to facilitate other researchers to reproduce your experiments. I understand that the space is limited and authors decided to prioritise their theoretical contribution but I think it's important to report these details in the appendix.", "questions_to_authors": "## Suggestions regarding weaknesses of the case studies\n\n1. [Section 5.1 - training details] The authors specify that they used Pythia as a base model but it's not clear how they completed the training for the Reddit TL;DR dataset. Was this model instruction-tuned first using SFT or not? Did they try completing this step for other open-source models? I believe it would be good to comment on this to provide more information regarding how their work will influence the overall LLM training pipeline.\n2. [Section 5.1 - credit assignment comparison] I believe that the authors should execute this evaluation in a more systematic way. They only report very specific examples for the DPO model only. Does a model trained only with SFT behave in a similar fashion? Is there any difference between the two?\n3. I would recommend the authors use the appendix to report more details regarding the training steps required to reproduce their case studies. I think it's important to at least include hyperparameters and other steps required to reproduce the training pipeline that they have used.\n\n## Minor edits\n\n1. Typo at Page 4: belmman equaitons\n2. I would suggest the authors consider moving their related work at the end of the paper considering that the \"Preliminaries\" section provides a very good overview of all the technical background required to understand their theoretical contribution.\n3. Authors should consider citing older work on using RL for image captioning such as [1]\n\n## References\n[1] Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi, and Rita Cucchiara. 2020. Meshed-memory transformer for image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10578–10587.", "extra_scores": {}}, {"review_id": "bSmgTCb2xj", "reviewer": "Reviewer_hRDL", "summary": "This work offers an alternative derivation of Direct Preference Optimization using Markov Decision Processes (MDPs) that enables to define token-level rewards. This has theoretical and practical applications. Fundamentally that DPO can learn token level rewards and assign different credit to tokens, a relation between guided decoding and search-based algorithms and an explanation of the phenomenon of decreasing likelihoods during DPO training.\n\nThe core of the paper is a derivation that follows closely that of DPO but from an MDP perspective i.e. [S4.1]\n\n1. Determining the optimal policy for the KL-controled reward maximization loss. Compared to DPO here they consider a more general case of the loss that uses a MDP characterization with states and actions as a function of time. The optimal policy for this case is a know result, as it happened with DPO.\n\n2. Deriving a the reward from the this optimal policy. Compared to DPO, here the use of the  Bellman Equations is needed to obtain the token level relations. The value function cancels in the same way the partition function cancels in DPO. I may have missed details on this part, but it seems solid. Result of this is Eq (9) a derivation of DPO as an MDP.\n\n3. In addition to this they generalize the definition of equivalency of DPO for the MDP case. This exploits very similar properties as in the case of DPO (S4.2).\n\nIn the following sections they use these derivations to deduce certain properties of DPO and empirically confirm these i.e.\n\n4. The MDP derivation of DPO, indicates that it can learn token level rewards. \n\n5. Guided decoding that maximizes a reward, also based in the MDP formulation, is the same as maximum likelihood DPO optimization.\n\n6. Explain the known phenomena that \"the implicit rewards of both the chosen and rejected response decline, though the margin between them increases\", which once is seen from the perspective of the MDP it is explainable.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the detailed review and kind words!\n\n\n**“While the original derivation of DPO relies on the fact that Q∗(x, y) = r(x, y)...”**\n\nIndeed, the original DPO work uses a bandit setting which can be represented as a trivial single-step MDP, which does not require any sequential considerations. In that setting the value function Q(x,y) is trivially r(x,y), but as we argue in later sections, this does not seem to represent the real sequential nature of language problems. \n\n**“Question 2: Could you expand on this difference? It seems to be and important aspect to judge the novelty”**\n\nGarg et al. (2022) is an imitation learning algorithm, while we focus on the problem of improving language models from (offline) user feedback. Hejna & Sadigh (2024) trains a parameterized value function using the feedback dataset and then separately attempts to extract a policy from that value function. All experiments are based on continuous control and the approach has been hard to scale to LLMs. In our work we have a unifying formulation through max-entropy RL, where we represent the LLM itself as both a policy and an implicit value function and do not use separately parameterized models. Moreover, the proofs in those works state something similar, but critically different: that there is a bijection between optimal rewards and value functions under a fixed policy. Both our setting and results are different. We show that there is a bijection between the reward function and *optimal* Q-function (not Q-function of a fixed policy) and that this holds in the Token-MDP with discount factor one (while prior works require discount factors less than one in order to invoke contraction properties of the Bellman operator). This requires a different proof technique than used in prior work.\n\n\n**“I fail to see why this is not self-evident. Scaling the logits by f(x) does not affect the probability distribution. Could you please clarify this?”**\n\nThis comment links the theoretical reward-smoothing formulation of Ng et. al. (1999) as represented in Section 4.2 to the practical LLM implementation. We agree that from the logit's perspective, it is clear that a constant shift does not affect the final model outputs, but represents the reward shaping potential $\\Phi(s)$. However, from a token-level RL perspective this was originally non-obvious until Ng's. work. Here we extend this reward shaping formulation to the feedback setting, which we state in Theorem 1.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper provided novel and relevant insights into DPO, in particular \n\n1. a theoretical derivation token-level reward DPO bringing concepts from Markov Decision Processes. \n2. showing that DPO maximum likelihood search is equivalent to decoding to maximize a reward function. \n3. provide insight and empirical results showing the importance of the starting policy initialization in DPOs training.", "reasons_to_reject": "Not really reasons to reject, but more clarity on some RL terms NLP people may not be fully familiar with. Thanks to RLHF's impact in LLMs, the average reader on this conference will be familiar with a lot of the material (I am one of those), but maybe some concepts may remain unfamiliar, such as, bandits, MDP and Q function. \n\nI have to say I did not fully understand some of the claims and some aspects seem self-evident, I put these on the question section. Probably after discussion this should all be ok.", "questions_to_authors": "### S 4.1\n>While the original derivation of DPO relies on the fact that Q∗(x, y) = r(x, y)\n\n*Question 1:* Can you clarify this assertion? The original derivation in (Rafailov et al 2023) does not use MDPs or Q. It basically consists on two elements\n\n-  derivation of the form of the optimal policy for the KL-controlled reward maximization objective (appendix A.1). This just needs algebraic manipulation of the original objective and checking the solution satisfies this and is unique.\n\n- derivation of the DPO objective, which follows again by algebraic manipulation of the optimal distribution derived above and replacing this in the Bradly-Terry model.\n\nI am aware that there are earlier derivations for the optimal policy, but this one seems simple and not need of the assumption above. \n\n### S4.1\n>This style of relationship was first explored by Garg et al. (2022) in imitation learning and later in Hejna & Sadigh (2024) for preference-based RL. However, these works require the use of a discount factor γ < 1 which is typically not used in RLHF.\n\n*Question 2:* Could you expand on this difference? It seems to be and important aspect to judge the novelty. Maybe add some summary of the derivation used in those works in the Appendix A.  \n\n### S.1\n>Second, we demonstrate that likelihood search over a DPO model is analogous to searchign over a reward function during decoding as done by contemporary works (). \n\n*Question 3:* can you provide this missing reference? \n\n### S 4.2\n\n>Interestingly, in practice, the potential function Φ(st) represents the free parameter in the logits of the language model. An equal shift along all logits yields the same policy, but different Q-functions and corresponding rewards. The above Theorem proves that all of these are in the same equivalence class and induce the same set of preferences.\n\nI fail to see why this is not self-evident. Scaling the logits by f(x) does not affect the probability distribution. Could you please clarify this?", "extra_scores": {}}, {"review_id": "HjWTmeuqBc", "reviewer": "Reviewer_Qos1", "summary": "This paper formulates DPO in the token-level MDP and provides practical insights into DPO policy based on the derivation. The derivation is carefully step-by-step, and the paper is well organized. The paper provides interesting insights into the token-level behavior of DPO.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the detailed feedback! We already provided an answer about the evaluations in Section 5.1. in our response to reviewer UQbN and to your second question in our response to reviewer hRDL, which we will not repeat due to response length limitations. \n\n**“Section 5.2 attempts to make a connection between guided decoding and DPO by showing that DPO improves…”**\n\nA number of recent works ([1]-[3] for reference), have explored using search-based methods with a value function during inference time, which has shown to improve the quality of outputs. Moreover [1] has also shown such benefits when using a DPO base model as well. The goal of section 5.2 is to theoretically show the equivalence between such search methods and DPO likelihood-based search and provide empirical proof. As such we do not aim to explain the performance improvement of beam search, but to make the point that we do not need an explicitly trained separate value function.  Instead we can utilize the implicit value function learned by the DPO model, by leveraging the connection between implicit value functions and LLM likelihoods in DPO, which is the main theoretical contribution of our work as outlined in Section 4. \n\n\n**“I do not clearly see the equivalence between Eq (2) and the formulation of RLHF in the previous studies. Are they exactly the same?”**\n\nPrior classical RLHF works usually formulate the RLHF problem using bandit terms, however in practice the PPO algorithm (and similar policy-gradient based methods) optimize the token-level objective of Eq. 2. \n\nThank you for pointing out a number of typos, we will fix these in our final version!\n\n\n[1] Controlled Decoding from Language Models, Sidharth Mudgal, Jong Lee, Harish Ganapathy, YaGuang Li, Tao Wang, Yanping Huang, Zhifeng Chen, Heng-Tze Cheng, Michael Collins, Trevor Strohman, Jilin Chen, Alex Beutel, Ahmad Beirami, 2024\n\n[2] Don't throw away your value model! Generating more preferable text with Value-Guided Monte-Carlo Tree Search decoding, Jiacheng Liu, Andrew Cohen, Ramakanth Pasunuru, Yejin Choi, Hannaneh Hajishirzi, Asli Celikyilmaz, 2024\n\n[3] Alphazero-like Tree-Search can Guide Large Language Model Decoding and Training, Xidong Feng, Ziyu Wan, Muning Wen, Stephen Marcus McAleer, Ying Wen, Weinan Zhang, Jun Wang, 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The token-level formulation provides a novel token-level interpretability of DPO.\n* The derivation is well explained.\n* The paper is well-organized and easy to follow.", "reasons_to_reject": "* I understand that the main contribution is the token-level formulation of DPO, and I find the practical insights based on this formulation interesting. However, the empirical results are too weak to support the practical insights the paper insists on.\n  - Section 5.1 shows only several qualitative results (including the appendix) to argue that DPO learns token-level rewards. \n  - Section 5.2 attempts to make a connection between guided decoding and DPO by showing that DPO improves performance with moderate beam size. However, the relationship between the argument and the results is unclear. In text generation, it is well-known that beam search with a moderate size of beam improves performance on (e.g., [1]). Why should the cause of that improvement be attributed to DPO?\n* The derivation provides a new interpretation of DPO but does not accompany improvement on DPO. This is a minor point and not the focus of this paper, but the current introduction makes such an expectation (e.g., \"we rectify this difference\").\n\n[1] If beam search is the answer, what was the question? (EMNLP 2020)", "questions_to_authors": "* I do not clearly see the equivalence between Eq (2) and the formulation of RLHF in the previous studies. Are they exactly the same?\n* The first line in Section 4.1 says, \"While the original derivation of DPO relies on the fact that $Q^*(x, y) = r(x, y)$,\" but I could not find this statement before this line. Where is it explained?\n* Isn't M and N opposite in Eq. (9)?\n* Section 5.3 provides interesting findings, but what is its connection with the token-level formulation of DPO?\n* Typos\n  - Abstract: multi-tun -> multi-turn\n  - Section 1, paragraph 4: searchgn -> searching; missing citation in the next line.\n  - Section 3.1, after Eq (1): \"loss\" -> \"lose\"?\n  - Section 3.3, paragraph 1: redundant \"entirely\"\n  - Setion 4.1, paragraph 1: $\\pi*$ -> $\\pi^*$", "extra_scores": {}}]

Author rebuttal:

Meta-review: This theoretical paper sheds new lights into connections with the status quo algorithm of RLHF (PPO) and a popular alternative introduced not so long ago (DPO). In particular it provides an alternative derivation of Direct Preference Optimization that shows that it does produce token-level rewards (instead of single final action at the end of the sequence).

This alternative derivation then allows the authors to leverage beam search for decoding and also to explain the unsatisfactory phenomena where the rewards of both completions (better & worse) decline all while the difference between them increases.

Most reviewers loved this paper, despite some experimental conclusions which are not of the same level than the theoretical insights. It would certainly be a valuable addition to COLM's programme and trigger insightful discussions.
Decision: Accept