A Critical Look At Tokenwise Reward-Guided Text Generation
Authors: Ahmad Rashid
Ruotian Wu
Julia Grosse
Agustinus Kristiadi
Pascal Poupart
Venue: COLM 2025 (2025)
Keywords: LLM
RLHF
Alignment
Model Efficiency
Reward Models
Sampling
Test-time
TL;DR: We analyse some of the pitfalls of contemporary reward guided text generation methods, and present a principled approach with strong performance on several language generation benchmarks.

Abstract: Large language models (LLMs) can be improved by aligning with human preferences through fine-tuning -- the so-called reinforcement learning from human feedback (RLHF). However, the cost of fine-tuning an LLM is prohibitive for many users. Due to their ability to bypass LLM fine-tuning, prediction-time tokenwise reward-guided text generation (RGTG) methods have recently been proposed. They use a reward model trained on full sequences to score partial sequences during decoding in a bid to steer the generation towards sequences with high rewards. However, these methods have so far been only heuristically motivated and poorly analyzed. In this work, we show that reward models trained on full sequences are not compatible with scoring partial sequences. To alleviate this, we propose to train a Bradley-Terry reward model on partial sequences explicitly, and autoregressively sample from the implied tokenwise policy during decoding. We study the properties of this reward model and the resulting policy: we show that this policy is proportional to the ratio of two distinct RLHF policies. Our simple approach outperforms previous RGTG methods and performs similarly to strong offline baselines without large-scale LLM fine-tuning.

Published as a conference paper at COLM 2025
A Critical Look At
Tokenwise Reward-Guided Text Generation
Ahmad Rashid
Ruotian Wu
University of Waterloo
University of Waterloo
Vector Institute
Vector Institute
Julia Grosse
Agustinus Kristiadi* Western University
Pascal Poupart"
University of Tübingen Tübingen AI Center
University of Waterloo
Vector Institute
Vector Institute
Abstract
Large language models (LLMs) can be improved by aligning with human preferences through fine-tuning the so-called reinforcement learning from human feedback (RLHF). However, the cost of fine-tuning an LLM is pro- hibitive for many users. Due to their ability to bypass LLM fine-tuning, prediction-time tokenwise reward-guided text generation (RGTG) methods have recently been proposed. They use a reward model trained on full sequences to score partial sequences during decoding in a bid to steer the generation towards sequences with high rewards. However, these methods have so far been only heuristically motivated and poorly analyzed. In this work, we show that reward models trained on full sequences are not com- patible with scoring partial sequences. To alleviate this, we propose to train a Bradley-Terry reward model on partial sequences explicitly, and autore- gressively sample from the implied tokenwise policy during decoding. We study the properties of this reward model and the resulting policy: we show that this policy is proportional to the ratio of two distinct RLHF policies. Our simple approach outperforms previous RGTG methods and performs similarly to strong offline baselines without large-scale LLM fine-tuning, Code for our work is available at https:/ /github..com //giaubccccm//aamaaaaa) Haamadrahh/PARG
Introduction
Large language models (LLMs) provide modern foundation for most, if not all, text generation tasks (Radford et al., 2019; Brown et al., 2020; Touvron et al., 2023a;b). In practice, significant improvements in the quality of text generation are achieved by aligning LLMs to human preferences (Stiennon et al., 2020b; Ouyang et al., 2022). This is typically performed via reinforcement learning from human feedback (RLHF), which involves two steps: learning reward model from preference data and ii) fine-tuning an LLM to maximize expected rewards by reinforcement learning (Ziegler et al., 2019b). Usually, this is done via reinforcement learning algorithm such as proximal policy optimization (PPO, Schulman et al., 2017). Nevertheless, recently, Rafailov et al. (2023) showed that the reward modeling step (i) can be bypassed by directly fine-tuning
LLM with preference data, resulting in a method called direct preference optimization (DPO). Although this simplifies RLHF, the fine-tuning step (ii) remains prohibitively costly for most users, since it requires high- performance computational resources with large GPUs.
In order to alleviate the computational issue above, Khanov al. (2024); Deng & Raffel (2023) explored tokenwise -warddguided text generation (RGTG) techniques that avoid any fine-tuning of the LLM. More precisely, the LLM remains frozen (i.e., not fine-tuned), and the reward model is used at the decoding time to adjust the softmax scores of the tokens directly. Unlike DPO, this line of work retains the reward modeling step, but training
'Equal Supervision

---

Published as a conference paper at COLM 2025
reward models is typically much cheaper endeavor than fine-tuning text-generation LLMs since smaller models can be utilized for reward modeling. Furthermore, reward models are modular: they can easily be composed and reused without any cost to guide text generation in conjunction with any base LLM. In contrast, RLHF via DPÔ requires fine-tuning of every LLM that we wish to improve based human preference data.
While RGTG is an interesting alternative to the standard offline RLHF, it is often based on heuristics and still poorly analyzed. For instance, ARGS (Khanov et al., 2024) proposed to simply use a reward model trained on full sequences to score each partial sequence during autoregressive decoding. Meanwhile, Deng Raffel (2023) used custom tokenwise loss to distill reward model trained on full sequences. Thus, it is unclear whether these approaches can give rise sound tokenwise text generation policy. Controlled decoding (CD; Mudgal et al., 2024), on the other hand, uses rollouts from the base model along with reward model trained on full sequences to distill the partial reward.
In this work, we analyze this common RGTG approach. First, we show that the usage of full-sequence reward models to score partial sequences in a tokenwise policy is pathological. To alleviate this, we propose to explicitly train a Bradley-Terry (BT) reward model on partial sequences. We prove that this text generation policy a ratio two different RLHF policies trained on sequences of different lengths. Ideally, the policy would be derived from single RLHF policy, but as we shall also show in Section 3, the exact computation of such a policy is intractable. By deriving the policy from ratio of distinct RLHF policies, obtain tractable sampling procedure. This is akin to the argument of Zhao et al. (2024) where they use a similar ratio to derive a sequential Monte Carlo method.
We empirically validate our analysis on three different text generation datasets on two recent LLMs. Evidence shows that our RGTG approach achieves better performance compared to ARGS and CD, matching the performance of the more expensive, offline PPO and DPO baselines. In summary:
(i) We analyze the recent practice of using full-sequence reward models for guiding the LLM decoding process. In particular, we show deficiency in this approach.
(ii) We thus propose to explicitly train a BT reward model on partial sequences and sample from the induced per-token policy induced by it during the decoding time.
(ii) We show that this reward model induces a ratio of two distinct RLHF policies over sequences with different lengths. This is a trade-off that one must make to make tokenwise RGTG free of the aforementioned deficiency and yet still tractable.
(iv) Extensive experiments with multiple LLMs on text generation validate insights.
2 Preliminaries
We denote a prompt by x and its response by y where the bolded letters indicate sequences of tokens. The i-th token in is denoted by x", while the partial sequence starting at token i and ending at token is denoted by (t, The length of a sequence is denoted by |x]. The same notation applies to y.
2.1 Reinforcement Learning from Human Feedback
LLMs generally consist of probabilistic models that can generate response y given prompt x. More specifically, the generation of is done token-by-token by sampling the next token from conditional distribution 77 <y (:--).
y
xx,
containing K triples of token sequences yx, y)), Ziegler et al. (2019b) and Ouyang et al. (2022) proposed technique based on reinforcement learning (RL) align an LLM with the preference dataset. They train parametric reward model rg x) that assigns a higher score to the "winning" (i.e., preferred)
Given a preference dataset D = {(xx, Yu.) { j==
utterance yu and lower score to the "losing" utterance yı. This is done via the BT model (Bradley& & Terry, 1952) which minimizes the loss:
LR
log (rp (yw (x) ro(yi(x)),
(1)
2

---

Published as a conference paper at COLM 2025
where o is the logistic function. Note that Te is trained to score entire utterances y. Once To is trained, it can be used to infer the probability of generating sequence y in response to x, i.e, Pø(y[x) = exp(r,(y/x))/,, epprr// explry(y')x). Given a reference LLM, we denote by (ye()//)) the conditional probability that it will generate response to prompt x (also referred to as policy). We refer to the LLM and its policy interchangeably. One can then copy the LLM and fine-tune to maximize
max
E X X
[r+(y|x)] - DKI. [KL[/T0 [xo(()x) l| [4ct(y|x)]I
(2)
where the KL term forms a regularizer that ensures that the fine-tuned model will not differ too much from the reference model. The optimization problem above be solved by many RL techniques, including the popular PPO algorithm (Schulman et al., 2017). This RL
optimization is quite costly in practice due to the size of the LLM.
The optimization (2) has a closed form solution of the form (Peters & Schaal, 2007)
Hgly|x)
Z(x) Tref(y exp(pr((y)x))
where Z(x) Ly Tre (y/x) exp (ylx)) is the intractable partition function. Notice that we can reorganize (3) to express the reward function in terms of the policies Ty and Tee
log (y(x))) + log Z(x),
r(ylx)
B
The (yl))
which can be used to replace (x(xy)) in (1) to obtain the following optimization problem:
max E logo (3 log nT(yw)l) log YYooY!-D
TT(y)ix) The (y ))
Nhat (y
Maximizing the above objective with respect to 0 directly fine-tunes the LLM without the need to learn a reward model. Furthermore, this maximization is done by supervised learning, which is generally simpler than RL. This approach, known direct preference optimization (DPO, Rafailov et al., 2023), reduced the cost of RLHF while solving an equivalent optimization problem. However, note that both PPO and DPO based RLHF are still very costly in practice since they require fine-tuning (a copy of) the target LLM Te
2.2 Reward-Guided Text Generation
In a separate line of work, Khanov et al. (2024) proposed reward-guided text generation (RGTG) techniques that do not require any LLM fine-tuning, but can obtain sequences y with high reward. This is done by freezing the reference LLM Tet and at decoding time, the next-token probability Ty' x.yl:-1) is adjusted by a reward model r..- More specifically, possible values for y' are scored by a weighted combination of logits of Tref and the rewards:
score(y' |x, y 1 = log ( y + (r
The next value for y is then selected greedily by maximizing their score or by sampling from a softmax distribution of the scores that has a similar form to the RLHF policy in (3):
softmax(score(() 1 1)) Z(x,y Tret (y' y exp(Brx
where the partition function 7(x yii--)))) is now tractable since the summation is now over all possible values of just single variable it is summation over possible tokens in the vocabulary.
However, is unclear whether the resulting distribution is equivalent/ approximating the RLHF policy in (3). Khanov et al. (2024) do train the reward model with the BT loss, but it is trained only with complete sequences, i.e. r (ylx), while it is used to score partial sequences,
3

---

Published as a conference paper at COLM 2025
En
Context x
y²
y
y
ef = nuil
Context X
y¹ y
y
Figure 1: A pathology of using reward model trained on full-sequence to predict partial sequences in decoding-time RGTG. We denote r' = rry' x yl: While the total reward over the full sequence y = (y' y"", might be nonzero c, it could be in the extreme case that the values over previous partial sequences are all zero-thle is a perfectly valid result for a sequence-level reward model (top). This means we can have an unguided decoding in reward-guided decoding. By explicitly training on partial sequences, could avoid this issue (bottom): While might achieve the same final reward c, nonvanishing reward signals over partial sequences could be avoided.
ie. (x))) Hence, is unclear whether the inferred scores for partial sequences are reasonable. In Section we show that reward models trained only with complete sequences can assign arbitrary scores to partial sequences, and in Section we show empirically that the resulting RGTG policy therefore underperforms that of PPO DPO. Meanwhile, Deng Raffel (2023) learn the reward model by minimizing a cumulative squared loss to distill full sequence reward model instead of using the BT loss (1), making the connection to RLHF policy looser. Mudgal et al. (2024) also distill a partial reward model from full sequence reward model, but the tokenwise policy is not the marginal of the full-sequence policy. Nevertheless, tokens are sampled from a different tokenwise RL formulation that follows similar derivation as RLHF.
Zhao et al. (2024) proposed to match each of the marginal distribution of ( by learning a series of parametric functions {/
y.y This turn induces policy:
rly |x 11 yxx,yl: 1) (yeryy (y y exp (1), (l)).
The generated sequences y are then approximately equal to the sequences generated by the RLHF policy (3). However, their method is general and does not specifically target RGTG- indeed, Zhao et al. (2024) focused using the implied approximation of the partition function Zxx). Finally, Rafailov et al. (2024) modifies DPO to obtain partial-sequence reward model.
Tp x, xxy 1:1-1)
r(y [x) = log [ (x, y
B
Similar to the sequence-based DPO, this reward model is then used to obtain per-token loss function fine-tune the LLM and thus, while defining a partial-sequence reward model, is not RGTG method.
Pitfalls of RGTG and How to Fix Them
First, we start by analyzing the partial sequence rewards inferred from reward model trained with full sequences only. Proof in Appendix A.
Theorem 1. A reward model r trained to minimize the BT loss (1) on full sequences y1:ly) may assign arbitrary rewards to partial sequences yii (where < ly)). More precisely, r(y]:1|x) x where 1 la E IR can be any value.
This leads to an unidentifiability problem-see see Fig. 1 for an example. If we learn reward model based preferences over full sequences only as proposed by Khanov et al. (2024)
4

---

Published as a conference paper at COLM 2025
Training
Prediction
(x, yhi)
Small Reward Model
P (y!l (b))
Taget LLM
(x.
5+) Cat(solluax(
The proposed approach to alleviating the problem in Theorem 1. First, we train the reward model r on partial sequences explicitly (when yyv * lyyl, we pad to the longest sequence) and (ii) sample from the weighted sum of the logits and the rewards of the next token during decoding. This is in contrast to some previous RGTG methods where the reward model trained on full sequences, but decoding relies on partial sequence scoring.
Figure 2:
and Deng Raffel (2023), then we may not obtain adequate rewards for partial sequences. As concrete example, suppose that is reward model such that (Fig. 1)
r(y'xx,y'it1) [r(y)x) 0
i= lyl i< lyl.
This reward model satisfies the identity in (8) and therefore, could be the solution when minimizing the BT loss (1). If we use this reward model to sample from the induced RLHF optimal policy in (3), then the token-level sampling distribution is the same as for the reference LLM (y' 1 for all tokens except the last one. This is problematic since RLHF generally changes the token-level distribution at each position, not just the last token. Hence the ARGS method (Khanov et al., 2024) may utilize reward model that does not score partial sequences properly, and negatively impact token-by-token generation.
To alleviate this issue, we propose to explicitly train the reward model with partial sequences-still using the BT model in contrast to Deng & Raffel (2023) follows (Fig. 2). We create separate loss function for all prefix lengths i:
C Ex log o (r x) T ro(x)i(x))
(4)
Then, given that full sequence yw is preferred to full sequence yı, we assume that the partial sequence yl is also preferred to the partial sequence y) Strictly speaking, is hard for human annotators to compare partial sequences due to their incomplete nature, and most preference datasets do not include preferences over partial sequences. Nevertheless, we can interpret ylu as the prefix of a winning sequence that is preferred over a losing sequence with prefix y!.. The following lemma shows that the resulting reward model ensures that the probability that a first partial sequence is preferred to a second partial sequence corresponds to the probability that the first sequence is extended to winning full sequence while the second sequence is extended to losing full sequence according to the preference data distribution Paata- Proof in Appendix A.
Lemma 2. In the limit of infinite preference dala, optimizing sufficiently expressive reward model according to (4) under the assumption that partial sequences inherit the winningg/osing label of full sequences yields a reward model r with the following property:
ofr(y|1*1x) ( 1x)) = Panta (y1 7 Y2 y Y2
1:j
(5)
where Paitu is the distribution the preference dataset was sampled from and yı b y2 indicates that yı preferved to y2.
Hence, optimizing the partial-sequence objective (4) for all lengths determines a reward model for all response prefixes that is adequate in the sense that it induces distribution over partial sequences that approximates the true underlying preference distribution (due to finite data) instead of assigning arbitrary rewards in the sense of Theorem 1.
5

---

Published as a conference paper at COLM 2025
Once the partial-sequence reward model r is trained, we use it to sample the next token y conditioned on the previous tokens x, y according to the following conditional distribution:
my x, y - 1)= Z(x,yI:i-1) (y |x, ylu 1) exp (y
1
(6)
Algorithm 1 summarizes the decoding procedure. Contrary to the previous approach of Khanov et al. (2024), it directly follows the policy induced by the explicitly trained reward model on partial sequences. Meanwhile, compared to Deng & Raffel (2023), it uses the standard BT model instead of a custom squared loss function that distills full-sequence reward model.
Let us now analyze the tokenwise sampling distribution in (6). By the definition of con- ditional distributions, we can rewrite it as a ratio of two partial sequence distributions: (x)"///""""-->>>
r uyy'), ([y'|x,yl=-11 to RLHF policies - the main point of the tokenwise RGTG methods. The following theorem shows how the decoding process by following this distribution relates to RLHF-induced policies. Proof in Appendix A.
[x). However, it is still unclear how this distribution relates
Theorem 3. Given a reward model trained according to the partial-sqquence BT objective in (4), the induced token generation distribution (6) is proportional to the ratio:
πt xx 111-1
TRIHEA
(7)
where TRLHE, and TRLHE:-11 are two distinct policies over prefix sequences of length and i 1, respectively, induced by RLHF optimization (2).
Ideally, we would like a decoding proce- dure that samples the next token from distribution that is mathematically equiva- lent to the conditional distribution restutii from an RLHF over full sequences. How- ever, as shown in Theorem 3, a partial- sequence reward model re leads to multiple RLHF decoding policies with different con- ditional distributions for each prefix length
Algorithm 1 Decoding with our approach.
Input: Pretrained partial-sequence reward model TØ Prompt x, number of candidates k, hyperpa- rameter $ > 0, any reference/SFI model Tge generation length
Output: A generated response to of length 1: for iori=11 to do
2: v(k) top ><<TTer( yii
3:
for v E v(k) do
i. Hence, it is not possible to have equiv- alence with single RLHF policy, e.g. obtained via PPO or DPO.
4:
Reward rp (i v(x))
5:
Logit log net (1) (/lxxy/yttt 1
6:
logı = 0 x, (-y-11-1) 11:--1)
One may then ask: Which RLHF policy is best? We argue that none of them nec- essarily better than the others since they simply arise from considering different pre- fix lengths. Note that the reward model r leads to a distribution that approximates the true underlying preference distribution on partial sequences. The problem is inherent to RLHF which takes reference LLM with a consistent distribution over response pre- fixes induced by a reward model and yields different decoding policies for different prefix lengths.
log Te (D) x, 1 + fr (y
7:
end for
8:
y Cat((sottaxx(og T(y |x, yli--))))
9:
end for
10: return y
Since all the resulting RLHF decoding policies have merit, one could argue that we can keep things simple by selecting only one policy, perhaps the RLHF policy induced by full sequence preferences (i.e., TRLHF (y] x)). However, as discussed by Rafailov et al. (2024) and Zhao et al. (2024), conditional distribution over full sequences does not give immediate procedure for token-wise sampling. Mathematically, we can derive a token-level policy from full-sequence policy as follows:
TRLHI (y [x]
Eyir yi TRL (ylx)
TRLHF (y' ( (i---
TRLHF (yli 1/x) Eyam TRLHF(y|xx

---

Published as a conference paper at COLM 2025
However, the summations in the above equation are exponentially large in the length ly] of the sequences. This exponential complexity was also noted by Zhao et al. (2024) who proposed twisted sequential Monte Carlo technique to approximate the computation and mitigate the exponential complexity. In contrast, our approach embraces the multitude of RLHF policies and leverages them in a linear time decoding procedure without any approximation of the partial sequence RLHF policies. The ratio policy (6) described here can thus be seen as a necessary tradeoff if one wants to perform tokenwise RGTG without the pathology in Theorem 1.
4 Related Work
Language model alignment Simple fine-tuning and instruction tuning (Wei et al., 2021) are ways to align LLMs to labeled data. Recently, RLHF methods (Christiano et al., 2017;
Ziegler et al., 90; Lee et al., 2021; Nakano et al., 2021; Snell et al., 2023) have provided a direct method to align LLMs to human preferences. The PPO algorithm has been especially popular and has shown promising results for a range of tasks (Askell et al., 2021; Bai et al., 2022; Ouyang et al., 2022). However, training RL models is compute intensive and researchers have turned their attention to supervised fine-tuning methods that can learn directly from preference data. Liu et al. (2023a) turns the preference data into prompts with which they fine-tune the LLM. Dong et al. (2023) uses the reward model to filter the training set to better fine-tune the model. DPO (Rafailov et al., 2023; 2024) models the LLM itself as Bradley-Terry model and optimizes the RLHF objective without any need for RL. TDPO (Zeng et al., 2024) incorporates token-level KL divergence into the DPO objective to improve content diversity. These methods, however, fine-tune the base LLM, which can be expensive as we scale. Some works have attempted to improve alignment by gathering more fine-grained rewards by using either LLMs (Cao et al., 2024) or human annotators (Wu et al., 2023).
Guided decoding There has been prior work in guided decoding using sequence- level (Welleck et al., 2022; Uesato et al., 2022; Lightman et al., 2023; Krishna et al., 2022; Li et al., 2023; Khalifa al., 2023; Yao et al., 2023) and token-level value functions (Dathathri et al., 2019; Krause et al., 2021; Yang & Klein, 2021; Chaffin et al., 2022; Liu et al., 2023b), PPLM (Dathathri et al., 2019) uses the gradients from an attribute classifier to guide LLM generation. Gedi (Krause et al., 2021) uses attribute conditioned language models as dis- criminators to update LLM generation probabilities using Bayes rule. These algorithms are different from our work as they do not align LLMs using human preference data. Deng & Raffel (2023) use a reward model trained on preference data in the decoding process, however, they use cumulative squared loss function that is different from the RLHF framework. Mudgal et al. (2024) uses a similar loss function with the key difference that instead of training with samples from a preference dataset, they take input full sequence reward model and train a partial sequence value function based on roll-outs (i.e., sampled token sequences) from the base LLM. Therefore, for each new base LLM, the value function needs to be retrained with new roll-outs, limiting portability to new or updated language models. The closest work to our method is Khanov et al. (2024), which also based on the Bradley-Terry model, but they use reward model trained on full sequences, which we have argued can lead to pitfalls. Different from our work, Zhao et al. (2024) present reward-guided decoding method based on sequential Monte Carlo and show that it can approximate RLHF.
Outside of preference data alignment and RLHF, prior work in reinforce- ment learning for language modeling has looked at partial reward models for improving text generation. Hao et al. (2022) show that a sequence to sequence model trained with supervised learning is a valid partial reward model for text generation under a Markov de- process, a ranking function which can rank the next token for partial sequences. Both these methods modify language model training.
Partial Rewards
cision process. Lee et al. (2023) do not train an explicit reward model but instead introduce
7

---

Published as a conference paper at COLM 2025
TL;DR Summarization
HH Dialogue
Method
LLM
Single y?
rd
Method
LLM
Single
y1
Top-k
frozen
yes
-0.11110.28
Top-k
frozen
yes
-1.42.10.21
CD
frozen
yes
0.32.10.33
CD
frozen
yes
-1.08.10.21
ARGS
frozen
yes
1.57:10.21
ARGS
frozen
yes
-0.97110.19
PARGS-G
frozen
yes
2.06++0220
PARGS-G
frozen
yes
-0.97
PARGS
frozen
yes
2.36:10.20
PARGS
frozen
yes
-0.88+1.19
Best-of-N
frozen
no
2.2 4-0.19
Best-of-N
frozen
no
0.17 4-018
DPO
trained
yes
0.81
DPO
trained
yes
-0.79:1031
PPO
trained
yes
2.41+0.23
Table 1: Average reward (over 100 samples) i standard error for the TL;DR summarization and HH dialogue tasks. The best technique that freezes the LLM and generates single response is bolded.
TL;DR Summarization
HH Dialogue
Method A
vs
Method B
Win-Tie (%)
Method A
Method B
Win-Tie (%)
PARGS
CD
75-0 0
PARGS
CD
52 8
PARGS
ARGS
73
PARGS
ARGS
49 11
PARGS
Best-of-N
-
PARGS
Best-of-N
36 11
PARGS
DPO
-9
PARGS
Top-k
56 15
PARGS
PPO
56
PARGS
DPO
27 14
Table 2: GPT-4 evaluation based the win-tie rate of PARGS over different baselines TL;DR summarization with GPT2-large, and on HH dialogue generation with Llama-2-7b.
5 Experiments
We evaluate our proposed approach, which we call Partial Alignment as Reward-Guided Sampling (PARGS)-in contrast to ARGS which considers full sequences and greedy decod- ing instead of sampling-od on two generation tasks: summarization and dialogue generation.
5.1 Setup
Summarization task We use the Reddit TL;DR dataset (Völske et al., 2017), where, the context x is a post on the Reddit forum and is the summary of the post. We use the human preference dataset from Stiennon et al. (2020a) to train the reward model and the relevant baselines. Our base summarization model is GPT2-large, fine-tuned on the TL;DR training set. We and train it with partial sequences for an additional epoch. Our baselines include top-k sampling (Fan al., 2018), Best-of-N generation, which involves sampling N sequences from reference LLM (N = 10 for all our experiments) and returning the best one according to the reward model, RLHF models based on PPO and DPO, the reward-base decoding method ARGS (Khanov et al., 2024) and controlled decoding (CD; Mudgal et al., 2024). We use CD-Fudge as the baseline all our CD experiments, noting that its performance is similar to CD-Q (see Table in Mudgal et al. (2024)).
pre-trained reward model based the DeBerta-v3-large architecture
Dialogue task Next, we evaluate our model on single-turn dialogue using the Anthropic Helpful and Harmless (HH; Bai et al., 2022) dataset. The goal is to generate helpful and harmless response to a general purpose query. Each sample provides a prompt x and two responses y with label indicating the preferred response. We use Llama-2-7b the base model and DeBerta-v3 as the reward model which is about 20 smaller. Details in B.
Fine-grained text generation task dataset (Ganqu Cui et al., 2024). We use Zephyr-7B as the base LLM and Phi-1.5 (1.3 billion parameters) as the reward model.
We also evaluate model the UltraFeedback
8

---

Published as a conference paper at COLM 2025
Ultra Feedback
Ultra Feedback
Method
LLM
Single y?
r SE
Method A
Method B
Win-Tie (%)
Top-k
frozen
yes
-0.1810.12
PARGS
CD
53 13
CD
frozen
yes
-0.04.10.01
PARGS
ARGS
42 23
ARGS
frozen
yes
0.01:1:012
PARGS
Best-of-N
29
PARGS
frozen
yes 0.21=00099
PARGS
Toр-k
52-15
PARGS-G
frozen
yes
0.21+:0.12
PARGS
DPO
65-7
DPO
trained yes
-0.57++00.9
Best-of-N
frozen
no
1.15 +0.08
Table Average reward (100 samples) 1 std. error and GPT-4 evaluation for Ultra Feedback.
Machine translation task We perform additional experiments on machine-translation the IWSLT-2017 dataset (Cettolo al., 2017). We used the post-edit dataset from Kreutzer et al. (2020) the IWSLT-2017 English-German dataset to provide token-wise reward signals. We use Gemma-2b as both the base model and the reward model. The evaluation is based on the standard BLUE score.
Evaluation Following Khanov et al. (2024), we compare all methods based on average reward, the test samples, as measured by the reward model. We use different full- sequence reward model and not the partial-sequence reward model (that we trained for our algorithm) to evaluate the models. Since evaluating language generation, especially unconditionally, is nuanced and human evaluation is very expensive, we use GPT-4-based evaluation, which has been shown to align with human assessment (Zheng et al., 2023; Rafailov et al., 2023). Following Chiang et al. (2023) we construct prompts for the two tasks and ask GPT-4 score and rank response pairs. We randomly shuffle the order of the responses to mitigate position bias (Zheng et al., 2023). Finally, we use the Rouge-L. score (Lin, 2004) and the BLEU score evaluate the dialogue and translation tasks, respectively.
5.2 Results
Table 1 (left) shows the average reward for the summaries generated by the different algorithms measured by the reward model. PARGS achieves the best average reward among the techniques that keep the LLM frozen and generate single response y. We also note that PARGS outperforms DPO and is competitive with PPO based RLHF that incurs large cost to tune the LLM, and Best-of-N that incurs significant overhead to generate multiple responses. Upon siginficance testing we observed PARGSS to be significantly better than all algorithms expect PPO. Details in Appendix C. Note that we also evaluate our algorithm with greedy decoding (PARGS-G) for a direct comparison with ARGS.
Similarly, Table (right) presents average rewards for the responses of the different algo- rithms the HH dialogue task. Note that in this setting, the reward model is 20x smaller than the base L.M. Again, PARGS achieved the highest reward among the techniques that freeze the LLM and generate single response. We observe that Best-of-N achieved the highest average reward followed by DPO, but incurred overhead to generate multiple re- sponses and fine-tune the LLM respectively. Finally, Table (left) presents average rewards on the UltraFeedback dataset. We observe that PARGS outperforms all methods except Best-of-N. Significance testing (see Appendix C) reveals that PARGS significantly better.
Next we evaluate PARGS using GPT-4. The prompt used to probe GPT-4 is presented in Appendix J. Table reports the win-tie rate (i.e., percentage of utterances where GPT-4 finds PARGS' response to be better than or equivalent to those of the baselines). Table 2 (left) shows that PARGS has a higher win-tie rate compared to all the methods, especially ARGS, for TL;DR summarization As noted by others Rafailov et al. (2023), ofst- is strong baseline, but it is computationally intensive. On HH, we observe (Table right) that PARGS is better than CD and ARGS, but worse than Best-of-N and DPO. As scale training based alignment methods, e.g., DPO become prohibitive. On UltraFeedback (Table 3 right) observe that PARGS outperforms all methods except Best-of-N. We perform human

---

Published as a conference paper at COLM 2025
evaluation for PARGS against ARGS, CD and DPO UltraFeedback. PARGS wins against all three baselines. Detailed results in Appendix F.
We perform additional experiments (Cettolo et al., 2017). We used the post-edit dataset from Kreutzer et al. (2020) on the IWSLI- 2017 English-German dataset to provide token-wise reward signals. We use Gemma-2b as both the base model and the reward model. The evaluation is based on the standard BLUE score.
on the IWSLI-2017 dataset
The translation direction is English to German and the edit- ed/ ed/corrected sequence is considered the winning sequence. Table 4 compares ARGS, PARGS-G and greedy decoding. We observe that applying ARGS reduces the BLUE score of the greedy baseline where PARGS-G increases it by 1.5 on average.
Table 4: BLEU Score
IWSLT-17 English-German
Method
BLEU
We evaluate the diversity of generation on 50 samples from the UltraFeedback dataset. We compare different methods by generating 10 responses for each prompt, evaluating the Rouge-L. score between each generated pair and averaging over all the samples. A lower Rouge-L score indicates a greater diversity. Table 5 shows that PARGS generates the most diverse responses compared to top-K and DPO.
Greedy ARGS
31.7 = 3.6
29.4 + 3.4
PARGS-G
33.2 3.5
Table 5: Diversity Results
Note that Best-of-N generates Nx the number of samples using top-K generation.
Method
ROUGE-L
Top-k
0.230 +0.011
DPO PARGS
0.206 +0.006
Additional Results A discussion on decoding costs is pre- sented in Appendix D. A study on effect of the hyper- parameter 8 and K in top-K is presented in Appendix G. We present an empirical analysis validating our assumption that the partial sequence of winning sequence wins over the partial sequence of losing
0.203 10.008
sequence in Appendix H.
6 Conclusion
We have discussed the pitfalls in tokenwise, decoding-tinn reward-guided text generation (RGTG) with reward models trained on full sequences. These pitfalls can lead to inadequate reward during the autoregressive decoding process and may lead to subpar performance To alleviate this, we train reward models on partial sequences and then sample from the implied per-token text generation policy during decoding. We proved that this policy is a ratio of two distinct reinforcement learning from human feedback (RLHF) policies. This means that this policy is not equivalent to the standard offline RLHF methods. However, we have also shown that it is intractable to obtain a tokenwise policy that is equivalent to single RLHF policy. Our experiment results validated our approach: it performs better than e recent RGTG methods such as ARGS, that leverages full-sequence reward models, and CD. We discuss limitations of our work in Appendix E.
Acknowledgments and Disclosure of Funding
Resources used in this work were provided by the Province of Ontario, the Govern- ment of Canada through CIFAR, companies sponsoring the Vector Institute https:// vectorinsti tute. a ai/partners/ and the Natural Sciences and Engineering Council of Canada. AR thanks Apple for support through the Waterloo Apple PhD Fellowship, Natural Sciences and Engineering Council of Canada for its support through the CGS-D program, and the David R. Cheriton Graduate Scholarship. JG thanks Microsoft Research for support through its PhD Scholarship Programme and the International Max Planck Research School for Intelligent Systems (IMPRS-IS). AK thanks Rob Brekelmans for fruitful discussion.
10

---

Published as a conference paper at COLM 2025
References
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as laboratory for alignment. arXiv preprint arXiv:2112.00861, 2021.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. ar preprint arXiv:2204.05862, 2022.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: The method of paired comparisons. Biometrika, 39(3/4), 1952.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurlPS, 2020.
Meng Cao, Lei Shu, Lei Yu, Yun Zhu, Nevan Wichers, Yinxiao Liu, and Lei Meng. Beyond sparse rewards: Enhancing reinforcement learning with language model critique in text generation. arXiv preprint Xiiv:2401.07382 2024.
Mauro Cettolo, Marcello Federico, Luisa Bentivogli, Jan Niehues, Sebastian Stüker, Kat- suhito Sudoh, Koichiro Yoshino, and Christian Federmann. Overview of the IWSLT 2017 evaluation campaign. In Proceedings of the 14th International Conference on Spoken Language Translation, pp. 2-14, Tokyo, Japan, December 14-15 2017. International Workshop on Spoken Language Translation. URL
Antoine Chaffin, Vincent Claveau, and Ewa Kijak. PPL-MCTS: Constrained textual genera- tion through discriminator-guidde MCTS decoding. In NAACL, 2022.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric Xing. Vicuna: An open-source chatbot impressing GPTT44 with 90%* ChatGPT quality, March 2023. URL https: //wsss/mm////////// otg/blogg/0233333 03- 30-vicuna/.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In NIPS, 2017.
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text generation. In ICLR, 2019.
Haikang Deng and Colin Raffel. Reward-augmented decoding: Efficient controlled text generation with a unidirectional reward model. In EMNLP, 2023.
Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, SHUM KaShun, and Tong Zhang. RAFT: Reward ranked finetuning for generative foundation model alignment. TMLR, 2023.
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In ACL, 2018.
Lifan Yuan Ganqu Cui, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with scaled ai feedback. In ICML, 2024.
Yongchang Hao, Yuxin Liu, and Lili Mou. Teacher forcing recovers reward functions for text generation. NewrIPS, 2022.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. Xiv preprini Xix::22001.0888 2020.
11

---

Published as a conference paper at COLM 2025
Muhammad Khalifa, Lajanugen Logeswaran, Moontae Lee, Honglak Lee, and Lu Wang. Grace: Discriminator-guidee chain-of-thought reasoning. In EMNLP, 2023.
Maxim Khanov, Jirayu Burapacheep, and Yixuan Li. Alignment as reward-guided search. In ICLR, 2024.
Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. GeDi: Generative discriminator guided sequence generation. In EMNLP, 2021.
Julia Kreutzer, Nathaniel Berger, and Stefan Riezler. Correct me if you can: Learning from error corrections and markings. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, 2020.
Kalpesh Krishna, Yapei Chang, John Wieting, and Mohit lyyer. Rankgen: Improving text generation with large ranking models. In EMNLP, 2022.
Kimin Lee, Laura M Smith, and Pieter Abbeel. PEBBLE: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. In ICML, 2021.
Youngwon Lee, Jinu Lee, and Seung-won Hwang. Learning to rank generation with pairwise partial rewards. In EMNLP, 2023.
Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. Making language models better reasoners with step-aware verifier. In ACL, 2023.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In ICLR, 2023.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summariza- fion branches out, pp. 74-81, 2004.
Hao Liu, Carmelo Sferrazza, and Pieter Abbeel. Chain of hindsight aligns language models with feedback. In ICLR, 2023a.
Runcheng Liu, Ahmad Rashid, Ivan Kobyzev, Mehdi Rezagholizadeh, and Pascal Poupart. Attribute controlled dialogue prompting. In ACI, 2023b.
Sidharth Mudgal, Jong Lee, Harish Ganapathy, YaGuang Li, Tao Wang, Yanping Huang, Zhifeng Chen, Heng-Tze Cheng, Michael Collins, Trevor Strohman, et al. Controlled decoding from language models. In ICML, 2024.
Rejichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. WebGPT: Browser- assisted question-answering with human feedback. arXiv preprint ar XrX::22122.09332, 2021.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. In NeuriPS, 2022.
Jan Peters and Stefan Schaal. Reinforcement learning by reighte regression for operational space control. In ICML, 2007.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl Blog, 2019.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In NewrIPS, 2023.
Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From r to Q": Your language model is secretly a Q-function. In COLM, 2024.
12

---

Published as a conference paper at COLM 2025
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithmss. arXio preprint urXiv:1707.0634/ 2017.
Charlie Snell, Ilya Kostrikov, Yi Su, Mengjiao Yang, and Sergey Levine. Offline RL for natural language generation with implicit language Q learning. In ICLR, 2023.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. In NeurlPS, 2020a.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize with human feedback. In NeurlPS, 2020b.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and efficient foundation language models. arXiv preprint ar XiXo:2302.13371, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023b.
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Trina Higgins. Solving math word problems with process-and outcome-based feedback. arviv preprint ar Xiv::2211.142755, 2022.
Michael Völske, Martin Potthast, Shahbaz Syed, and Benno Stein. TI; dr: Mining reddit to learn automatic summarization. In Proceedings of the Workshop on New Fronfiers in Summarizatioon, pp. 59-63, 2017.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners.
In ICLR, 2021.
Sean Welleck, Jiacheng Liu, Ximing Lu, Hannaneh Hajishirzi, and Yejin Choi. Naturalprover: Grounded mathematical proof generation with language models. In NeurIPS, 2022.
Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for language model training. NeurlPS, 2023.
Kevin Yang and Dan Klein. Fudge: Controlled text generation with future discriminators. In NAACI, 2021.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In NeurlPS, 2023.
Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang, Haifeng Zhang, and Jun Wang. Token-level direct preference optimization. In ICML, 2024.
Stephen Zhao, Rob Brekelmans, Alireza Makhzani, and Roger Grosse. Probabilistic inference in language models via twisted sequential Monte Carlo. In 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, al. Judging LLM-as-aajjdde with MT-bench and chatbot arena. In NeurIPS, 2023.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human prefer- ences. Xiv preprint ar Xix: 90..9099.,8,,, 2019a.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human prefer- ences. ar Xiv preprint ar Xix::199.00859; 2019b.
13

---

Published as a conference paper at COLM 2025
A Proofs
Theorem 1. A reward model r trained to minimize the BT loss (1) on full sequences yl:ly) may assign arbitrary rewards to partial sequences y1 (where i < ly)) More precisely, r r(yl:) (x) xxyy where D xxy' E can be any value.
Proof. Let rly") x, ) be the reward associated with token y in the context of x, y Then token-level and (partial) sequence-level rewards are related by the following identity:
(() = Ej= ,(
for all x, x,
(8)
Optimizing a reward model with full-sequence preference data yields specific values for ryy":yl|x) xx). Since partial sequence rewards are not directly optimized, it is not clear what val- ues they may converge to. The above system linear equations can be used to infer partial sequence rewards from full sequence rewards. However the system is underdetermined since there are more variables than equations: there is one equation for every combination of x, y, and while there is one variable per combination of x, y, and i on the left-hand side of each equation and many more variables on the right-hand side. Hence partial sequence rewards can take arbitrary values and yet satisfy (8).
In the limit of infinite preference data, optimizing sufficiently expressive reward model according to (4) under the assumption that partial sequences inherit the winning/losing label of full sequences yields reward model r. with the following property:
Lemma 2.
orrgyyl r 1: lx)) = Pasta (y1 b y2 X,
Y2 1:j
(5)
where Paata is the distribution the preference dataset was sampled from and y1 & Y2 indicates that y1 preferred to y2-
Proof. In the limit of infinite preference data, maximizing the log-likelihood in (4) is equiva- lent to minimizing the KL divergence between the learned preference distribution o and the preference data distribution for partial sequences.
argmax
E xy1.y22 Paata
logo 1: - ro ro 1: lx))
(9)
argmin
E Xyly2--Paaa
logo (xx(x)=((x)
|x) - rr(y)] 1:/ (x))
(10)
Panta (y1 x y2 x, yt y22
argmin
E X2
log
(11)
a(r (y| x) r ry 1:f (x))
argmin KL (y1 2 yl 1i 1:/ N22 (y -r0(y2" (x))
(12)
With sufficiently expressive reward model, the KI. divergence will be zero, and therefore, the distribution a equals the preference data distribution.
((roly]'1xx rr(22111) xx) Paata (y1 > y2/x, yl"
y 22
1:/
(13)
14

---

Published as a conference paper at COLM 2025
Theorem 3.
Given reward model trained according to the partial-sequence BT objective in (4), the induced token generation distribution TI (6) is proportional to the ratio:
TRIHE, (y):((
<<y²l X y 1)
(7)
1 (yl:-
where HRI and HF: two distinct policies over prefix sequences of length and i respectively, induced by RLHF optimization (2).
i, performing RLHF (2) under a reward model r induces a different policy i (x) for different values of To see this, notice that by (2):
Proof. We first note that for each prefix length
TLLHF, (y (y xx = 1/ exp((r(yli))
Then, for < j, we have by marginalization:
TRLHE, f (y*x*xx)
E Ty
y yty
x Σ
(y (xx((
y y'1
Σ Troe(yy 11//x, y yiu exp(Br(yli/xx)
- exp(Br(y
(x)((())))(x))
Tre 18 yx, exp(Br(y1))x)) yiu
(y
E
exp( Br (y (1))
(y
Since Egt by My x y exp(Dr(y"")) x))
[x) depends on yi it cannot be treated as a normal-
ization constant. Therefore / (y # TRLHF, (y (x). Based on this fact, then:
zly'lx xx,y 1 Tee yx, 1:i
exp((r(y (x)))
(by (3))
Teer (v x, y 12-11 exp(Br(y (Br(( (yl
(normalization constant)
exp(pr(yl:t (ll-1(x)
(y x) exp(Br(yl:(xx)
(conditional distribution definition)
Tyl 1 (xx exp 1xx))
& TRLHE, (y xx)
(by (3))
TRLHEF 1 (y-
This completes the proof of the theorem.
B Training Details
Software and hardware All experiments (24GB VRAM) and NVIDIA A40 GPUs(40GB VRAM). We use CUDA Toolkit version 11.7 and PyTorch 2.2.2 framework.
server with NVIDIA RTX6000 GPUs
Training Partial Reward Models Based on DeBerta-v3-Large We train two partial re- ward models the partial sequences retrieved from the HH-RLHF and TL,DR dataset respectively, utilize the TRL library to accelerate the training process. We report the training parameters on Table and 8.
Training DPO Models We train two DPO models on the original preference dataset, one is trained based on GPI2-Large on the TL;DR dataset, and the other is trained based on
hatiaggggpp2-larg--ll---mm
15

---

Published as a conference paper at COLM 2025
Llama-2-7b on the HH-RLHF dataset. We also adopt the TRL library to train the DPO models. The training parameters are reported on Table 9.
Partial Sequence Data Generation We randomly sample subset of the set of all partial sequences to maintain a reasonable computational budget. We present an ablation on the tldr summarization dataset where present the average reward achieved by PARGS when training on different subsets, as well as the wall clock time.
Dataset Size
Average Reward
Wall clock time (approx)
Tx
1.64
0.22
hour
1.5x
2.32 + 0.19
1.5 hour
2x
2.36 + 0.20
2. hour
3x
2.23 1 0.20
hour
Table 6: Average Reward with different partial sequence dataset sizes. is the size of the full sequence dataset.
We can observe from the results on Table that get diminishing returns when the dataset is more than 1.5x. On the TLDR dataset we sample 2x and on the other datasets 1.5x of the total dataset size. Note that the wall clock time is for training 4 RTX6000 GPUs.
C Significance Testing
We ran the Wilcoxon signed rank test, which does not make any distributional assumptions, to evaluate statistical significance. We report the p-values below (a p-value less than 0.05 indicates that PARGS achieves results that are statistically better than the alternative method). Table 10 and Table 11 show the p-values of the rewards of PARGS various baselines. We observes that PARGS is significantly better than all baselines on TL,DR Summarization and all but Best-of-N Ultra Feedback. The results HH-ddialguee are better average but only signficantly better than Top-K.
D Decoding costs
We present an estimate for the floating point operations (FLOPs) per token for inference with PARGS. The reward model adds linear layer with a single output the language model. The number of non non-embedding parameters in model, following the calculation of Kaplan et al. (2020), is approximately N z 12 model where Maaers is the number of layers and dmodel is the hidden dimension size. Additionally the FLOPs required by a forward pass is Corwarrd Av 2N + /odeeel where Netx is the number of context tokens. The additional operations include 4d for the embedding and 2d for reward predicting. But since model < N, CRM s Cforward- Also if model > Max/12 12
assume that CRM = Cfowardd = 2N (Deng & Raffel, 2023). At decode time we analyse k-tokens using the reward model. In our experiments k 10, the total inference cost is Gforward + 10CRM
Parameters
Value
Parameters
Value
11 training samples
170053
It training samples
218933
LR Batch size
5e-6
LR Batch size
5e-6
TL;DR
16
HH-RLHF
16
Gradient acc. steps
16
Gradient acc. steps
16
DeepSpeed Zero stage
DeepSpeed Zero stage
Max. sequence length
Max. sequence length
512
B
1.5
ß
2
Table 7: Training Hyperparameters for Deberta-large-v3 partial reward models
arrgsearch/Ilama-aa hlaaa/aa-aaaa---fffltttete
16

---

Published as a conference paper at COLM 2025
Parameters
Value
Number of epoches
1
Learning rate Batch size
2e-6
Floating point format gradient accumulation steps
fp16
phil_5
8
DeepSpeed Zero stage
3
Max. sequence length
512
f
Table 8: Training Hyperparameters for Ultra Feedback reward model
Parameters
Value
Parameters
Value
Number of epoches
Number of epoches Learning rate Batch size
1
Learning rate Batch size
5e-5
5e-5
1
2
warmup steps Floating point format
150
Floating point format
fp16
GPT2-L
grad accumulation steps
16
LLaMA-7b
grad accumulation steps
bfl6
16
LoRA r
16
LoRA r
16
LoRA
16
LoRA
16
Max prompt length
512
Max prompt length Max sequence length
512
Max sequence length
512
512
Table 9: Training Hyperparameters for DPO models
TL;DR Summarization
Ultra Feedback
Method A
Method B
p-value
Method A
Method B
p-value
PARGS
Top-K
6.67 10-14
PARGS
Top-K
1.15 x 10-5
PARGS
CD
7.41 x 10-13
PARGS
CD
4.07 x 10
PARGS
ARGS
4.82 x 10-6
PARGS
ARGS
3.98 x 10
PARGS
Best-of-N
7.75 x 10-3
PARGS
Best-of-N
1.0
PARGS
DPO
4.02 x 10-10
PARGS
DPO
2.46 10-11
Table 10: P-values of the reward of different methods compared to PARGS
HHH-Dallgue
Method
Method
p-value
PARGS
Top-K
1.0 10
PARGS
CD
2,01 x 10
PARGS
ARGS
4.60 x 10
PARGS
Best-of-N
8.9 x 10
PARGS
DPO
9.9 x 10-1
Table 11: P-values of the reward of different methods compared to PARGS
17

---

Published as a conference paper at COLM 2025
FLOPs per token. When the language model is GPT2-large and the reward model DeBerta- v3-large, plugging in the parameters, the inference FLOPs overhead 4.3x the base model. When the language model is Llama2-7b, with the DeBerta reward model the overhead is 0.47x. Note that the Best-of-N decoding cost overhead would always be 9x.
On Figure 3 we plot the average wall-clock time to generate a single token by the LLM and re- ward model on an NVIDIA A40 GPU. Note that this is the time for one call to the Ilm and k = 10 calls to the reward model.
0.5
LLM
Reward Model
0.4
0.3
Time
0.2
E Limitations
0.0
One limitation in our method is the overhead in- duced from performing forward passes through the reward model at each decoding step. How- ever, note that this acceptable compared per- forming large-scale offline PPO or DPO which often prohibitive. Moreover, this limitation is shared with other RGTG methods. Another limitation is that in applications such as mathematical reasoning instead of generating tokenwise rewards we may require step or process rewards.
TLDR Summarization
HH Dialogue
Figure 3: Runtime overhead.
F Human Evaluation
We performed a human evaluation of the responses of PARGS versus ARGS, CD and DPO on the Ultra Feedback dataset. We enlisted independent evaluators to score the instruction following, correctness and helpfulness, of two Al assistant responses, on a scale of 1 to 5, We used the score mark a win, tie or loss for PARGS. The evaluators did not know the identities of the AI assistants and the responses were shuffled in random order. We can observe from the results on Table 12 that PARGS has high winning rate. We also observed large percentage of ties.
Ultra Feedback
Method A
vs
Method
Win-Tie (%)
PARGS PARGS
ARGS
45 - 50
CD
50 20
PARGS
DPO
60 25
Table 12: Human Evaluation based on 20 evaluations
Arbitrary Rewards We ran another human evaluation for empirical verification that a full sequence reward model can lead to arbitrary rewards. We took the TLDR test set of human summaries and randomly sampled 40 examples. Then we randomly cut-off one-fourth of the examples at 25%, 50%, 75% of the sequence length. We kept the last one-fourth at full sequence length. For each prompt the dataset had two responses. We enlisted human subjects and asked them to select winning or losing partial summaries based on which one looked the most promising for completion. If they could not choose between the two they could mark tie. Next ranked each pair of summaries based on the reward from the full sequence reward model. We compared the results of the human evaluation with the ones from the reward model. We removed the ties from the evaluation scores.
We report the results on Table 13. We can observe that full sequence evaluations have higher conformity with human evaluation compared to partial sequence evaluation.
18

---

Published as a conference paper at COLM 2025
Sequence Length
Agreement
%
%
25
50
50
43 %
75 %
50 % 80%
100 %
Table 13: Conformity of Full reward model with human judgement for different sequence lengths
G Sensitivity Analysis
We conduct sensitivity test the summarization task, using 6 1.5, and k E {5, 10, 15), we report the average reward and the standard deviations in Table 14, and the diversity score measure in Rouge-L in Table 15.
K/B
6 0.5
== 1.0
B = 1.5
5 2.0
2.0
8 = 2.5
k k=5
0.12 + 0.33
3.31 + 0.22
3.65 + 0.22
3.72 + 0.20
2.20 + 0.30
k k 10
-0.02 + 0.38
3.35 + 0.20
3.88 + 0.17
3.88 + 0.16
2.65 + 0.25
k=
0.61 0.38
1.04 0.39
2.07 1 0.29
2.21 + 0.23
2.88 + 0.27
Table 14:
Average Reward of summarization task with different value of f and k
k/B
B 0.5
1.0
B ==15 1.5
8= 2.0
k=5
0.29 + 0.03
0.31
= 0.04
0.30 + 0.03
0.30 + 0.03
k= 10
0,27 + 0.03
0.26 + 0.03
0.27 + 0.03
0.28 1 0.03
15
0.25 1 0.03
0.25
1 0.02
0.24 0.03
0.28 1 0.03
Table 15: Diversity based on ROUGE-L with different value of and k. Lower score is better
For the reward scores, we observe that f 2.0 achieves the highest score for every value of k, and the score starts to drop when we further increase & to 2.5. Also k = 10 achieves the best reward scores while k = 5 is usually better then k 15.
Since k represents the size of candidates the generation algorithm will sample from, we expect higher would result in better diversity, and the empirical results also validate this intuition. On the other hand, it seems does not have significant effect on generation diversity.
H Analysis
We empirically evaluate the assumption that the partial sequence of a winning sequence wins over the partial sequence of losing sequence. We cut the losing sequence at 25% and 50% of the total length, and sample multiple completions from πe Then we use reward model to evaluate whether the winning sequence achieves higher reward compared to all completions of the losing sequence. Table 16 shows the win-rate of a single winning sequence over multiple completions from the losing sequence different lengths. This result demonstrates that our assumption is reasonable.
Sequence Length
Win-Rate %
%
25
62 %
50 %
60 %
Table 16: Win-rate of winning sequence over 3 completions of the losing sequence on the TL;DR test set
19

---

Published as a conference paper at COLM 2025
I Generated Samples
The following are two example generations on Ultra-Feedback and TL;DR all the baselines including our method.
Ultra Feedback Generation Samples
Prompt
In the Monkey King Story, how did Monkey king fight with vampire? Assistant:
Generations
[PARGS] As per the legend, Monkey King fought with vampire named Yin Yang Shou, who was the leader took possession of his magical sword. The sword was said to have the power to cut through even the strongest of demons and it was an important part of Monkey King's s arsenal.
demons. Monkey King defeated Yang Shou gruesome battle and
[DPO] Monkey King fought with vampire in the story of Journey to the West.
[ARGS] The Monkey King fought with the vampire by using his supernatural powers. He used his strength fight the vampire and eventually defeated him.
[CD] The Monkey King fought with the vampire by using his supernatural powers and martial arts skills. He used his strength to overpower the vampire and his agility to dodge its attacks. He also used his magical abilities to create illusions and distract the vampire. In the end, the Monkey King was able to defeat the vampire and save the day.
[Best-of-N] In the Monkey King Story, the Monkey King fought with vampires by using his magical powers and techniques. He used his speed and strength attack the vampires and defend himself from their attacks. Additionally, the Monkey King had the ability turn into different shapes and sizes, which he used to outmaneuver the vampires and gain an advantage in battle. Despite the vampires' abilities, the Monkey King ultimately emerged victorious, using his intelligence and skill defeat them.
20

---

Published as a conference paper at COLM 2025
TL;DR Generation Samples
Prompt
SUBREDDIT: T/ relationships
IITLE: Me [19 M] with my GF [19 F] broke up after almost years, hard time staying friends POST: We broke up for college last year and I know had rough time, going from constantly seeing best friend to not communicating, It was understandably mutual decision as we were both going forward in life, but it still hurt lot.
Anyways after a few months of college, I thought I was done with her. Then she be- gan drunk texting quite often. I was civil at first then ignored them as I realized it was just hurting and going nowhere. So got she's been wanting to hang out with mutual friends as we were all close. I thought was at peace and had gotten over her but just seeing her again ruined my day. felt (and still feel) like I'm going through the breakup all over again.
her again.Now that we're back for summer,
Avoiding her out of the question since my best friends close friends of hers as well. We were all one big happy family and now there's this awkward rift between my ex and I that is stemming from my confused feelings and was damn near palpable to my friends when we all got together today.
Also, would never want to leave my friends, especially because of
TL;DR:
Generations
[PARGS] Ex and I broke up few months ago, still hurting and I'm having trouble avoiding her, especially since best friends are close friends of hers.
[DPO] I'm confused about my feelings towards my ex and want to avoid her as much as possible during summer between college.
[ARGS] I'm confused about my feelings towards my ex and it's causing tension between me and my friends.
[CD] I am confused as hell and having trouble coping, I am wreck and would love some advice.
[Best-of-N] I think should stay friends with my ex, but it's difficult avoid her since we're close. Any advice on what should do?
J GPT-4 Evaluation
We use the following prompts for the summarization and the dialogue tasks.
21

---

Published as a conference paper at COLM 2025
GPT-4 prompt for Evaluating TL;DR Summarization
System Prompt
[System]
Which of the following two summaries does better job of summarizing the most important points in the given forum post. Each summary receives overall score scale of to 10, where higher score indicates better overall performance.
Please first output a single line containing only two values indicating the scores Summary 1 and 2, respectively. Second provide one-sentence comparison of the two summaries, explaining which you prefer and why.
User Prompt
[Post] Human: Hello reddit. I'm in need of your help. I need to have my classes picked for my high school junior year and have no idea what do.
live in Nevada, and I want to go to the University of Washington, Seattle right out of high school. Next year I will be in my junior year of high school, so I'm picking classes. Here's ' my problem:
I'm taking the hardest classes available my school. The problem is, suck at my foreign language class. I already have 2 years of French. My dream is to go the University of Washington, and I'm sure that it would look better to have more foreign language classes on my resume. My issue is that the French 3 teacher is my friends mom, and know that shes HARD teacher. am most likely going fail if take third year French. need to know how important standstill and I need to have my class recommendations in really soon.
for me to take a third year French. I'm stuck at
TL,DR:
[The Start of Summary need advice on whether or it would be worth it to take third year of french to get to university where my dream is to go, or if would detrimental to my future chances.
[The End of Summary 1]
[The Start of Summary 2]
I suck at foreign language, want to get into UW of Washington, Seattle next year, and need to pick classes within 6 months.
[The End of Summary 2]
22

---

Published as a conference paper at COLM 2025
GPT-4 prompt for Harmless and Helpful Dialogue
System Prompt
[System]
You are helpful, harmless, and precise assistant for checking the quality of the answer. We would like to request your feedback the performance of two Al assistants in response to the question. Please rate the harmlessness, helpfulness, and level of detail of their responses. Your evaluation should consider factors such as the helpfulness, harmlessness, relevance, accuracy, depth, creativity, and level of detail the response. Note that a response appears cut off at the end due to length constraints, it should not negatively impact the score. Also, base your evaluation solely on the given answer, disregarding any preceding interactions in the question. Each assistant receives an overall score on scale of 10, where higher score indicates better overall performance.
Please first output a single line containing only two values indicating the scores for Summary 1 and 2, respectively. Second provide one-sentence comparison of the two summaries, explaining which you prefer and why.
User Prompt [Question] Human: How do I get better at bowling? Assistant:
[The Start of Assistant I's Answer]
Bowling is skill, and like all skills, it requires practice to get better. The first step is to decide what you want get better at: do you want to roll more strikes, reduce the number of spares you leave? Once you have goal mind, you'll want practice the following skills: Striking: This is all about timing. You need to be able to get the ball to move the way you want to, at the right moment in your swing. [The End of Assistant I's Answer]
[The Start of Assistant 2's Answer]
Bowling requires skill and coordination, as well as good understanding of lane conditions and the way the ball moves. There are several ways you can work to improve your bowling skills. Practice is essential, and good place to start would be to go to the local bowling alley with friend two and try bowling few games. Try different approaches, varying your stance, your grip the ball, and your arm swing. Practice throwing the ball and timing your release to match your movements. And try make sure your ball rolls straight down [The End of Assistant 2's Answer]
23


Reviews:
[{"review_id": "aN7pqpsOhZ", "reviewer": "Reviewer_rFy9", "summary": "This paper investigates reward-guided text generation (RGTG) as an alternative to standard offline reinforcement learning from human feedback (RLHF). The authors identify a key limitation of tokenwise RGTG: when the reward model is trained solely on full sequences, it may assign arbitrary rewards to partial sequences, leading to inadequate guidance during autoregressive decoding. To alleviate this issue, the authors propose explicitly training the reward model on partial sequences using the Bradley-Terry (BT) model framework, thereby enabling more reliable token-level rewards during generation. The authors further provide a theoretical analysis showing that the resulting policy is proportional to the ratio of two RLHF policies over sequences of different lengths, reflecting a tradeoff between avoiding pathological behavior of the reward model on partial sequences and maintaining tractability. Experimental results on summarization and dialogue generation tasks show the effectiveness of the method compared to recent RGTG baselines.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is clear and well-structured, especially the Preliminaries section, which provides a thorough introduction to RLHF, Reward-Guided Generation, and the relevant formulas related to these concepts. This section effectively guides the reader through foundational concepts, enabling a better understanding of the subsequent analysis.\n2. The mathematical derivations are well executed, with sufficient details provided throughout. Furthermore, the proofs included in the appendix are clearly presented, facilitating easy comprehension for the readers.\n3. By avoiding the need for expensive fine-tuning of the base LLM, the method is computationally efficient compared to traditional RLHF methods like PPO and DPO, while achieving competitive performance.\n4. The method is shown to generalize across different LLMs and datasets, highlighting its robustness and flexibility in diverse text generation contexts.", "reasons_to_reject": "1. The theoretical discussion around the Bradley-Terry model applied to partial sequences feels underdeveloped. Although the paper explains the ratio of two distinct RLHF policies, it does not sufficiently justify why this particular approach is the best one or explore the limitations and potential drawbacks in depth. The discussion could benefit from a more thorough exploration of the implications of using a ratio of RLHF policies and how this may affect the quality of the generated text.\n2. The paper contains several grammatical issues that may affect clarity. For example, in the conclusion section, the sentence “it performs better than a recent RGTG methods ...” contains a number agreement error (\"a\" with a plural noun), and in the related work section, the sentence “Different from our work, Zhao et al. (2024) a reward-guided decoding method ...” appears to be missing a main verb. Addressing these issues would improve the readability and overall quality of the writing.\n3. There appears to be a minor issue in the reward function derived from Equation (3), as it appears that the coefficient 1/β is missing in front of . It might be helpful to verify whether this omission was intentional or an oversight in the mathematical derivation.\n4. One potential weakness is that the paper assumes preference over full sequences implies preference over corresponding prefixes. However, this assumption may not hold in practice, as annotators typically do not provide preferences over partial sequences, and preferences may depend on tokens later in the sequence. This assumption may introduce bias into the learned reward model.\n5.  Another potential issue is that while the paper cites Deng & Raffel (2023) in several places (e.g., in the Introduction as a related approach), it does not include this method as a baseline in the experiments. It would be helpful if the authors could clarify why this method was not included, or consider adding it if feasible.\n6. The paper uses a mix of different evaluation metrics (e.g., win-tie rate, reward score), but does not provide a clear rationale for why these specific metrics were chosen or how they directly relate to real-world performance. It is unclear how well these metrics align with human expectations or the actual task requirements.", "questions_to_authors": "Please refer to \"Reasons To Reject\"", "extra_scores": {}}, {"review_id": "HfSXkAorfo", "reviewer": "Reviewer_2dhF", "summary": "This work theoretically and empirically analyzes Tokenwise Reward-Guided Text Generation (RGTG) methods, where a reward model predicts the marginal token probability that is added to the original token probability of the frozen base LLM to approximate an RLHF-tuned policy. Existing works in RGTG overlooked the credit assignment problem, failing to compute partial token-level rewards properly. The paper proposes a simple remedy of training the reward model with *all* prefix pairs of the winning/losing sequences (dubbed PARGS), ensuring that the final reward is properly distilled to each token. Empirical studies on different tasks (summarization, helpful-harmful dialogue, UltraFeedback win/lose data) show the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. **Clear theoretical analysis of existing RGTG works.** The authors provide convincing theoretical analyses about how existing RGTG works overlooked the credit assignment problem when distilling sequence-level rewards to token-level rewards. Based on the results, the authors successfully derived the novel training loss (Eqn (4)) and decoding algorithm (Theorem 3).\n\n2. **Diverse experiments that back the theoretical analyses.** Experimental results (e.g., average rewards and LLM-as-a-judge) clearly show that the reward model trained with the proposed loss produces improved results than baselines that ignore the credit assignment. The fact that the performance of PARGS almost matches RLHF-trained method is quite promising, and opens up many different future directions (e.g., having small specialized token-level reward models specialized in different tasks for plug-and-play architecture, which is a more lightweight solution than plug-and-playing LoRA weights in terms of both training and inference).", "reasons_to_reject": "1. **Missing empirical justification for a controversial assumption.** The proposed loss function for training a reward model relies on the assumption *\"that partial sequences inherit the winning/losing label of full sequences\"* (Lines 174-175). I question the validity of such an assumption and believe that the paper requires more empirical justification. I have two questions:\n\n(1) **Does this assumption empirically hold?** The standard, unbiased, and correct approach for estimating the token-wise reward is to obtain the expected final reward by sampling multiple rollouts (e.g., MCTS). A straightforward way to prove that this assumption is correct is to show that the MCTS-calculated reward of the prefix of a winning sequence is actually higher than that of a losing sequence for all prefix lengths. While it is hard to conduct such an experiment in the preference domain, using tasks with easily verifiable rewards (see point 2) will enable such analyses.\n\n(2) **Does the token-level reward obtained from PARGS follow the true utility distribution?** As pointed out by the authors, there are multiple ways to assign the token-level rewards given the final reward. I think this assumption and the proposed loss function will favor a uniform reward distribution among tokens, rather than rewards being sparsely distributed to only a few tokens. Interestingly, [1-2] has empirically shown that rewards are sparse. For instance, in math/multiple-choice questions, when the value function is estimated by sampling 5 continuations from each prefix, the reward is usually focused on a *single* token [1]. **Question:** How is the final reward distributed to each token in PARGS (uniform vs. sparse), and does that follow/contradict [1-2]'s results?\n\n2. **Missing tasks with verifiable rewards** Following the previous argument, a good way to justify PARGS and its underlying assumption would be testing on domains with verifiable rewards, like math, multiple-choice problems, or coding. My personal experience is that improving performance in these tasks by steering (directly manipulating token probabilities) is more challenging than tasks like toxicity or summarization. It would be nice to explore the utility of the proposed PARGS on these tasks.\n\n[1] Bigelow, E., Holtzman, A., Tanaka, H., & Ullman, T. (2024). Forking paths in neural text generation. arXiv preprint arXiv:2412.07961.\n[2] Lin, Z., Liang, T., Xu, J., Wang, X., Luo, R., Shi, C., ... & Tu, Z. (2024). Critical Tokens Matter: Token-Level Contrastive Estimation Enhence LLM's Reasoning Capability. arXiv preprint arXiv:2411.19943.", "questions_to_authors": "(1) (2) See Reasons to Reject, point 1.\n\n(3) **Directly measuring credit assignment**: One interesting method for measuring the token-level credit assignment ability is token mean reciprocal rank, introduced in [1] (Section 5.2). In this approach, you sample multiple sequences (>=16), determine one with the highest reward (gold), and check the reciprocal rank of each token in the gold sequence under the manipulated distribution. The mean reciprocal rank closer to 1 will imply that the tokens with higher final reward have been assigned a higher probability. Can PARGS achieve a higher token mean reciprocal rank than other RGTG methods? (This is not necessary; given the limited time, I would be happy to increase my score when (1) and (2) are fully addressed.)\n\n[1] Lee, Y., Lee, J., & Hwang, S. W. (2023, December). Learning to Rank Generation with Pairwise Partial Rewards. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (pp. 6078-6092).\n\nNote: After checking the author's response, I increased the confidence from 3 to 5, as I am now confident that the core assumption in this paper's theoretical analysis is severely unjustified, which limits the overall contribution.", "extra_scores": {}}, {"review_id": "iqnS56LxU0", "reviewer": "Reviewer_Z2q2", "summary": "This paper examines and improves token-wise reward-guided text generation methods for LLMs. It proposes a new approach called PARGS to address issues with existing RGTG methods that use reward models trained on full sequences to score partial sequences during decoding. The authors show that such reward models can assign arbitrary scores to partial sequences, leading to sub-optimal performance. To fix this, they train a Bradley-Terry reward model explicitly on partial sequences and sample from the implied token-wise policy during decoding. This approach outperforms previous RGTG methods like ARGS and CD, achieving results comparable to expensive offline RLHF methods like PPO and DPO without large-scale LLM fine-tuning.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This problem, which involves using a reward model trained at the sequence level to improve token-wise alignment, is interesting. To my knowledge, this is the first work to discuss this issue.\n\n- The proposed method appears to be simple yet effective. We do not need to modify much code to implement it successfully.", "reasons_to_reject": "- One primary concern lies in Eq. 5, which simply discards part of the sequence, specifically y1 and y2. This raises the question of whether we might lose valuable information, potentially introducing bias. While the paper provides some theoretical motivation for this approach, I remain somewhat concerned about this aspect. For example, there might be a case where y2 performs better than y1 in the earlier part of the sequence, but due to some errors that occur in the sampling process of the latter part, y2 ends up being worse than y1 overall. In such a scenario, can we confidently claim that y1 is always superior to y2 in the preceding segment? \n\n- Another concern is the insufficient number of experiments. While summarization and dialogue are common settings, there is a growing trend towards using reward-based optimization to enhance reasoning. Therefore, the paper could benefit from including experiments related to reasoning.", "questions_to_authors": "- In Table 1, why is the PPO result lacking on the dialogue task?\n- Is this issue related to training a PRM? Can the proposed method be applied to train a better PRM?", "extra_scores": {"ethics_concerns_details": "N/A"}}, {"review_id": "6rxXJ0fbYI", "reviewer": "Reviewer_JW6d", "summary": "This paper addresses a significant problem in the field of reward-guided text generation. The use of reward models trained on full sequences to guide token-by-token generation, where the authors show can be problematic.\n\n1. A theoretical analysis shows why the reward models trained on full sequences can assign arbitrary rewards to partial sequences, leading to suboptimal guidance during decoding.\n2. It provided a novel solution namely PARGS (Partial Alignment as Reward-Guided Sampling) to train Bradley-Terry reward models on the partial sequences.\n3. It gave the theoretical proof that the sampling policy representing a ratio of two distinct RLHF policies. It is a necessary trade-off to make tokenwise RGTG both principled and tractable.\n4. It also made extensive experiments on multiple datasets (summarization, dialogue, text generation). It shows that PARGS outperforms the existing RGTG methods. In addition, it is competitive with more expensive fine-tuning approaches like PPO and DPO.\n\nThe paper is well-structured and clearly written. The theoretical contributions are sound and well-supported by proofs. The empirical evaluation is comprehensive, covering multiple tasks, models, and baselines. The approach is practical and addresses an important limitation of existing methods without requiring expensive LLM fine-tuning.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The work provided theoretical analysis on the proposed RGTG methods using full-sequence reward models for partial-sequence scoring on why it can be problematic. \n2. The empirical results show the PARGS outperforms the existing RGTG methods, e.g., on multiple tasks. It also performs than more expensive fine-tuning approaches, e.g., PPO and DPO, etc.\n3. It enables better alignment even without requiring expensive LLM fine-tuning.", "reasons_to_reject": "1. The work proposal introduced significant computational overhead during the inference, since the requirement of multiple forward passes through a reward model for each token generation step.\n2. The proposed approach assumes preferences are over full sequences transfer to partial sequences. It may not hold in practicealways. Although the paper has acknowledged this limitation, but it does NOT explore it.\n3. As the authors mentioned in the limitations section, the proposed approach maybe NOT suitable for the tasks which require step-wise rewards rather than token-wise rewards (e.g., mathematical reasoning).", "questions_to_authors": "None", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a solution to the problem of token-wise guided text generation, where sequence-level rewards are used to influence token-level decisions. The reviewers main pros and cons are:

**Pros:**
- The paper is clear and well-structured; the mathematical derivations are coherent, and the conceptual ideas behind the method are well articulated.
- Empirical results demonstrate that the proposed method outperforms existing RGTG baselines.

**Cons:**
- The theoretical justification is not well developed.
- There is a lack of experiments to show the generalizability of the proposed method.
- The proposed work involves a significant amount of overhead during inference.
- Some critical baseline comparisons are missing.
Decision: Accept