Towards Cost-Effective Reward Guided Text Generation
Authors: Ahmad Rashid
Ruotian Wu
Rongqi Fan
Hongliang Li
Agustinus Kristiadi
Pascal Poupart
Venue: ICML 2025 poster (2025)
Keywords: LLM
RLHF
Alignment
Model Efficiency
Reward Models
Sampling
Primary area: deep_learning->large_language_models
TL;DR: We present an efficient reward model for reward guided text generation which leads to significantly faster inference and more optimal  choices.

Abstract: Reward-guided text generation (RGTG) has emerged as a viable alternative to offline reinforcement learning from human feedback (RLHF). 
RGTG methods can align baseline language models to human preferences without further training as in standard RLHF methods. 
However, they rely on a reward model to score each candidate token generated by the language model at inference, incurring significant test-time overhead. Additionally, the reward model is usually only trained to score full sequences, which can lead to sub-optimal choices for partial sequences. In this work, we present a novel reward model architecture that is trained, using a Bradley-Terry loss, to prefer the optimal expansion of a sequence with just a single call to the reward model at each step of the generation process.  That is, a score for all possible candidate tokens is generated simultaneously, leading to efficient inference. We theoretically analyze various RGTG reward models and demonstrate that prior techniques prefer sub-optimal sequences compared to our method during inference. Empirically, our reward model leads to significantly faster inference than other RGTG methods. It requires fewer calls to the reward model and performs competitively compared to previous RGTG and offline RLHF methods.

Towards Cost-Effective Reward Guided Text Generation
Ahmad Rashid 11 Ruotian Wu
Rongqi Fan
Hongliang Li Agustinus 2 2 Pascal Poupart
Abstract
proximal policy optimization (PPO; Schulman et al., 2017) or finetuning using direct preference optimization (DPO; Rafailov et al., 2023). The training costs can be prohibitive as we scale the L.M,, since perfforrrreee computational resources with large GPUs are required. Moreover, the LLM needs to be retrained whenever the preference data changes.
Reward-guided text generation (RGTG) has emerged as a viable alternative to offline rein- forcement learning from human feedback (RLHF). RGTG methods can align baseline language mod- els to human preferences without further training as in standard RLHF methods. However, they rely on a reward model to score each candidate to- ken generated by the language model at inference, incurring significant test-time overhead. Addi- tionally, the reward model usually only trained score full sequences, which can lead to sub- optimal choices for partial sequences. In this work, we present novel reward model archi- tecture that is trained, using a Bradley-Terry loss, to prefer the optimal expansion of a sequence with just a single call to the reward model at each step of the generation process. That is, a score for all possible candidate tokens is gen- erated simultaneously, leading to efficient infer- ence. We theoretically analyze various RGTG reward models and demonstrate that prior tech- niques prefer sub-optimal sequences compared to our method during inference. Empirically, our reward model leads to significantly faster infer- ence than other RGTG methods. It requires fewer calls to the reward model and performs compet- itively compared to previous RGTG and offline RLHF methods. Code for our work is available at
One way to alleviate this computational overhead, while still improving the alignment of the baseline LLM is token- wise reward-guided text generation (RGTG; Khanov et al., 2024; Deng & Raffel, 2023). RGTG methods keep the baseline LLM frozen and instead train a reward model on the preference data. At each decoding step. the reward model is used to adjust the softmax scores of each candidate token. Reward models are cheaper to train compared to offline RLHF updates (e.g., PPO and DPO) even if both the reward model and LLM have the same number of parameters.' Moreover, RGTG with small reward model can perform comparably to RLHF (Rashid et al., 2024).
However, while RGTG is a promising and cost-effective al- ternative to offline RLHF, it can lead to significant decoding overhead during inference. Typically, at each step, multiple calls are made the reward model for candidate tokens from the language model. This introduces "inner loop" in the decoding process of the LLM, leading to increase in computational complexity and latency. Another issue is that reward models trained on full-sequences are used to score partial sequences (Khanov et al., 2024; Li et al., 2024) which can be problematic (Rashid et al., 2024).
Several works have attempted train reward models to score partial sequences. Deng & Raffel (2023) use squared loss and the preference data to distill full sequence reward model into a reward model for partial sequences. Controlled decoding (CD; Mudgal et al., 2024) uses roll-outs from the language model instead of the preference data to distill a partial sequence reward model. Rashid et al. (2024) ex- plicitly train a Sradeey-Terry model on partial sequences and demonstrate connection with RLHF. We will show that these methods prefer sub-optimal extensions of partial sequences during decoding.
1. Introduction
Reinforcement learning from human feedback (RLHF; Sti- ennon et al., 2020b; Ouyang al., 2022) is widely applied to align large language models (LLMs) human preferences. However, updating the LLM with RLHF incurs signifi- cant training cost, whether it is reinforcement learning using
'Equal contribution 'University of Waterloo 'Vector Insti- tute HHaawei Technologies. Correspondence to: Ahmad Rashid <a9rashid wwwaiiddwwwtttto..c....
'To train n. PPO needs to load and make calls to 2 additional models (merr and a critic). DPO needs load and make calls one additional model (Ttat while training a reward model does not require loading or calling any additional model.
Proceedings of the 4nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

Towards Cost-Effective Reward Guided Text Generation
Top-K
Generation
requires
0.6
requires
0.6
Chenting excim
is can
0.5
0.3
Cheating esam
is
can
0.5
0.8
La Language Model
Language Model
Reward
Cheating on exam requies
0.1
Rewards
0.1
Cheating exaam
0.5
Chewling on an exdem
Cheating on examn can
0,3
Reward Model
Reward Model (Ours)
Prompt: Cheating
unethical can get you expelled
Figure 1. Figure depicting step in RGTG generation for both conventional (left) reward models and ours (right). Note that RGTG steers the LLM generation to helpful and harmless text. We observe on the left that for each candidate that is generated by the LLM, a call needs be made to the reward model with the candidate appended. On the other hand, reward model is fed just the prompt and it generates scores for all candidates in the vocabulary. On the TLDR dataset, we observe an average generation time of 19.4 seconds for current RGTG methods and 2.99 seconds for method.
To alleviate these common issues associated with RGTG methods, we propose a cost-effective reward model archi- tecture for RGTG, which can score all possible next token extensions of partial sequence with single call. Fur- thermore, train the reward model using novel loss function that, we show, scores prefixes that can be extended to optimal full sequences, at least as high as any other pre- fix. On extensive benchmarks we demonstrate that our reward model leads to a better cost-performance trade-off and higher diversity. Figure illustrates the shortcomings of current reward models, i.e., high decoding cost, and our proposed solution.
2. Preliminaries
We denote prompt by x and its response by where the bolded letters indicate sequences of tokens. The i-th token in x is denoted by 24, while the partial sequence starting at token and ending token j denoted by X The length of sequence is denoted by |x]. Large language models (LLMs) generally consist of probabilistic models that can generate response y given prompt More specifically, the generation of y is done token-by-toker by sampling the next token from a conditional distribution ι T X, y1si-)).
2.1. Reward Models
In summary:
Reward models are trained to evaluate the quality of re- sponse y to the prompt x by outputting a scalar-valued score. Given a preference dataset D {(x*, y ) con- taining K triples token sequences (x, y", ,""') where y" represents the "winning" (i.e., preferred) sequence and y' represents the "losing" sequence. The Bradley-Terry (BT) loss (Bradley & Terry, 1952) that encourages the model to assign higher score to the winning response and lower score to the losing response is used as the training objective:
We analyze contemporaty reward models and demon- strate that during RGTG they choose sub-optimal ex- tensions of partial sequences.
We present a reward model architecture that, at each decoding step, can provide rewards of all tokens in the vocabulary at once.
CR -x..y",' yD log (rø(y" (x) rx(y'(
We explicitly train our reward model to choose token with the maximum reward at each step.
where is the logistic function and r. is the reward model. Reinforcement learning from human feedback (RLHF: Ziegler et al., 2019; Ouyang et al., 2022) uses scores from the reward model to update the language model using rein- forcement learning (RL) techniques such as proximal policy optimization (PPO). Rafailov et (2023) derive an equiva-
We report extensive experiments with recent LLMs on various text generation tasks, demonstrating faster inference and strong alignment performance.
2

---

Towards Cost-Effective Reward Guided Text Generation
lent objective that can be learned using supervised learning without using reward model.
3.2. Sub-Optimal Reward Models
Next, we take look contemporary RGTG reward models and show that they may prefer partial sequences with sub- optimal extensions.
2.2. Reward-Guided Text Generation
Recently, Khanov et al. (2024) proposed reward-guided text generation (RGTG) technique that does not require update of the LLM. Instead, the base LLM Tet is frozen and, during decoding, the logits from the LLM are combined with the reward scores to guide the text generation.
PARGS Rashid et al. (2024) showed that using a BT value model trained on full sequences to score partial sequences (as done by Khanov et al. (2024)) can lead to arbitrary values for partial sequences. They proposed to train BT value model explicitly on partial sequences by creating a separate loss function for all prefix lengths i:
Let (y | x) be a value function with parameter that scores partial sequences Y1:: such that Vo x) = (y x) for full sequences y. During decoding, the adjusted score of token 1/ is weighted combination of logits of qet and the value of the partial sequence is as follows:
CA =
Σ (x.y",y")ED
log oVoy((ely - lx(
However, given that the full sequence y is preferred to the full sequence y', training is based on the assumption that the partial sequence y is also preferred the partial sequence yí... This assumption can be problematic, as the full-sequence dataset typically includes only one or few full sequences that extend each partial sequence. In fact, the empirical distribution of such extensions will affect the learned value function to the extent that a prefix with only extensions to suboptimal full sequences may be scored higher than a prefix with an extension to an optimal full sequence.
(xx log meer y18-1)
(1)
+ 3 (V
where 3 > 0 is syper-parameter.
Given the scores, the next token can be selected greedily or by sampling (e.g., nucleus or top-k sampling (Fan et al., 2018; Holtzman et al., 2020) from the softmax distribution of the scores).
In (I), while V corresponds for full sequences, further considerations needed to define V for partial sequences. Rashid et al. (2024) showed that using full-sequence re- ward models to score partial sequences can lead to arbitrary rewards during RGTG.
Theorem 1. In the limit of infinite training and sufficiently expressive representation for the value function, PARGS may learn value function that gives lower score to prefix extendable to an optimal full sequence than some other prefix. More precisely, if = argmaxy r(y) x), then there may exist i, J. y' such that
3. Current Limitations of RGTG
V(yi.lx) (V1ig)))
We will discuss two primary limitations of current RGTG methods, namely high decoding cost and sub-optimal re- wards. In the next section, we propose a solution to address these limitations.
Proof. Let y", y', y" and y" be four responses to such that y" is an optimal response and y', y", y" are three suboptimal responses. Suppose also that the preference dataset contains exactly three comparisons: y") where the first response is preferred to the second response in each triple. Suppose also that y and y' share the first i 1 tokens (i.e., vis-1 = y'a-1) while y", y and y share the first tokens (i.e., y = y"x = yi.) In the limit of infinite training and sufficiently expressive value function represen- tation, Lemma in (Rashid et al., 2024) indicates that the learned value function V satisfies
3.1. High Decoding Cost
D = (xx,y', "'x,,"''',, y'), (x, y',"",
Most RGTG methods default to training full sequence reward model T and then either a) use it to directly score partial sequences (Khanov et al., 2024) or b) distill partial sequence value model Vo from the full sequence reward model rs (Mudgal et al.. 2024). During decoding, the score for each candidate token y4 is calculated according to Equa- tion 1. We note that the input to Vo includes the sequence 91:i-1 with each candidate token y/ appended to the se- quence. Hence to score each candidate token, we need to make a different call to the value function, resulting in k calls for top-k decoding. This adds substantial overhead during decoding.
o (V V(yi,, = Po([x,y'] (x, (2)
where [a, b) indicates the concatenation of sequences a and b, and a > indicates that a is preferred to Equa- tion (2) implies that the BT model induced by V exhibits
3

---

Towards Cost-Effective Reward Guided Text Generation
the same preference probabilities for the full sequence extension of yis and yis as the empirical distribution of the preference dataset. Recall, that PARGS assumes that yial > yx.l when their respective full se- quence extensions exhibit the same preference ordering (i.e., y') > y²). Since there might be different exten- sions 1 and y²=1- for each prefix with different preference labels in the preference dataset, then PARGS learns value function that induces preference probabílities for partial sequences that are consistent with the empiri- cal distribution Pn of the preference dataset for the full sequence extensions of those partial sequences. Applying Equation (2) to prefixes yi. and yi.: yields:
the language model. We will show such dependency is suboptimal in Theorem 2.
Value Augmented Sampling (VAS; Han et al., 2024) is sim- ilar to CD and uses Nee to generate samples for learning a value function and a full-sequence reward model for gener- ating the target score. However, the value function is trained by temporal difference (TD) learning.
Theorem 2. In the limit of infinite training and a sufficiently expressive representation for the value function, CD may learn value function that gives a lower score a prefix extendable an optimal full sequence than some other prefix. More precisely, if argmaxy r(y) x), then there may exist j.y.' such that
(V - (x) = 1/3
(3)
V(yialx) < V(1gg))))
since the dataset contains one preference ranking y') where the full sequence extension y" of yi preferred to the full sequence extension y' of y' and two preference rankings yx,''y") (x. y'y") where the full se- quence extension y' of yi. is preferred to the full sequence extensions y". y" of yi... Recall that Vi. = y" = y' and therefore y" and y" are full sequence extensions of yi.s- Finally, since the sigmoid in Equation (3) is less than 0.5, then Vyyi.a /x) < VVy'...x). Hence, this shows that yi-j.y' such that V Viyi.. (x) < V(Li,)x))
Proef. Let y* be an optimal response to x such that r(y 6. Let y' and y be two suboptimal responses to x such that r(y'lx) 4 and r(y"lx) -6. Suppose that y' and y" share the same first - 1 tokens (i.e., Viai-1 y vis-1) and that y" and y* share the same first i tokens (i.e., Vi:: = yi.)). After generating yi.;, suppose that net gener- ates only Y and 11+11y" with uniform probability
(i.c., Tee i1::y= |x. yi.) = Trer(y"+1:l""" /x. v'44) = 0.5 and any other continuation has probability 0). After gen- erating Ni. suppose also that Tee generates only y'+1.ly' (i.c., ( (+1: y=, YYx) = and any other continuation has probability 0). Then with infinite training and suffi- ciently expressive value function representation, from (4), CD learns the following partial sequence values
D
Theorem shows that the value function learned by PARGS may prefer prefixes that lead to suboptimal responses. The key problem is PARGS' assumption that the preference or- dering of prefixes is the same as the preference ordering of full sequence extensions. Since it is possible to extend a prefix to many different full sequences with different scores, the value function learned by PARGS depends on the fre- queney of different prefix extensions instead of preferences only. As shown in the proof of Theorem 1, this becomes problematic when a prefix that can lead to optimal re- sponse is extended more frequently to losing full sequences instead of winning full sequences in D.
V(yi. x) = m (i.yiirrr""
+ The x, "i:))ry""1>
= =0.5(6) + 0.5( -6) = 0
V x) = = |x, M(x)r(y'Ix)
= =1(4)= 44
This example shows that Bi=j.y' y' such that V Vxri.lx) V V(y1,1) (x).
Mudgal et al. (2024) proposed a target value function V*, for partial sequences, that corresponds to the expected reward of the full sequences when the partial sequence is extended by following the base model distribution Tee.
CD
Theorem 2 shows that CD may prefer prefixes that cannot be extended to optimal sequences depending on Nee.. The key problem is the dependency of the target V on Tee-- When Tief extends a prefix to bad responses, the value of this prefix is low, but if it extends the prefix to good responses, the value of this prefix high. In principle, the value func- tion V should be independent of Kee In RLHF, neet is the quantity that we seek to improve, so it can introduce a bias to train a value function that depends on Nee. The value function should depend only on the preferences induced by the full sequence reward model. As shown in the proof of Theorem 2, CD may not prefer a prefix that can lead to an
V* VIxx))I=)= = Σ TT |x, (xx)(xx)) (4)
The training loss is the squared difference between the value function Va and the target V* They use rollouts from the base model along with a reward model trained on full se- quences to distill the value function Vo. They sample exten- sions from The to complete a partial sequence and compute the sell-ne score as the target V*. This method has a limitation where the value function heavily depends on
4

---

Towards Cost-Effective Reward Guided Text Generation
optimal response when it is extended by Fret suboptimal responses.
where Ys+1mm is the optimal extension beyond y1: and yn is the EOS token. That is, instead of doing an exponential search, we could train the value function to satisfy (5), which can be done by Temporal Difference (TD) learning. Note that VAS also uses TD learning in their algorithm, but, since they use a conventional reward model they do not do a max over the dictionary.
4. FaRMA: Cost-Effective RGTG
We propose to mitigate the inference overhead and sub- optimal rewards of previous RGTG methods by introducing (i) an efficient reward model and (ii) a novel loss function that will ensure that the resulting value function prefers prefixes extendable to optimal responses. We name our method FaRMA, i.e. Faster Reward Model for Alignment.
To be more precise, the training process can be separated into two steps with distinct objectives:
1. Standard BT loss on full sequence preference dataset:
4.1. An Efficient Reward Model
L(a)
log o (Ve x) Vo
x.y y'Nn
We design a reward model architecture so that instead of obtaining a single score for sequence, we obtain the score for all possible next tokens in the dictionary. We modify (1) such that:
(6)
2. Constraint to ensure optimal partial sequence expan- sion.
(w) = Vo(xislx))
mmx Bet1
Vo
(7)
score(y; x, VIs-1) = log Pyerlys |x, V1s-1)
+ (Vo(V N1s-11,
Firstly, we want to point out the similarity of our constraint (7) to TD control where V(y|x) can be treated as a state- action value function (i.e., Q-function) with yı correspond- ing to the action and [x. V1.-1] corresponding to the state. Note also that transitions are deterministic in LLMs since the action yi updates the state to y1:s] deterministically. We use to denote a state and a to denote an action in Bellman's 's equation:
where Vo(.) E RlPIx1 and [D] is the size of the vocabulary. In order to get the score of sequence v, Vis we feed the I, 91:1-1 into rá and get the score of the sequence with all possible extensions of ye in the dictionary. The efficiency and performance of the reward model is not dependent on kt, for top-k generation, as we simultancously get the score for all possible next tokens in the dictionary. We use the same architecture as a causal language model, however, we use a novel training loss which we discuss next.
Q'(s, (s) = Eirls, a] + Σ P(s'|s, (s, a) max (s),
=
= max (1xx yisl vi+1)
(8)
4.2. A Principled Constraint
Bi+1
Voly x) mmax Volyna+1lx)
Given the sub-optimality of the existing methods, there needs to be a more principled way to score partial sequences. We propose to score partial sequences based on their optimal extension. Given a partial sequence Yiss we consider all possible full extensions and assign the score of the highest expo-
1641
Note that Equation (8) follows from the fact that there is no discount factor and no reward until the end of the sequence. Then L()) is the same loss as in Q-gradient learning by treating max Ve (o(x as the target. Will
completion to Yi.. Naïvely, this would require nential search in terms of the size of the vocabulary which is intractable. To make this principled goal feasible, we propose a local constraint that the partial-sequence reward model needs to satisfy so that it will return the reward of the corresponding optimal expansion:
To train the value function, we alternate between the two losses mentioned previously. For the Bradley-Terry loss (6), we utilize full-sequence preference pairs as commonly done when training a reward model. Furthermore, for the new constraint loss (7), we extract partial sequences from the winning sequences in the preference dataset and use them as training data. Notably, there no preference signal when training with the constraint loss. The model simply learns to align its scores to the best next token. We train the model by alternating between the two losses. The training details are presented in Appendix B.
Vo (x) max Vo
(5)
2i+1
If the above local constraint is satisfied, then we can keep
expanding the sequence as in the generation:
Ve(yI. = max Vo(y1=+1|x)
Dit1
max max 21
Bet1
8i+2
We emphasize that this kind of training would not be possi- ble with the reward models of previous RGTG methods that
max Vo xx)
Vitton
5

---

Towards Cost-Effective Reward Guided Text Generation
Algorithm 1 Our Training Algorithm.
Since the loss in (7) ensures that the learned value function returns the reward of the best full sequence that extends prefix then V(iivlx) = r(y")x). Similarly, since y1- is any other prefix whose extensions do not lead to better full sequences, then V(y |x) S r(y"lx). This means that V(y x) S ( which contradicts (10).
Input:
Base LLM to initialize the reward model Ve, Full Sequence Preference dataset DBT = {(xx,, y .y (yth) /km1 number of alternating iterations iter,, mini-batch size n, partial sequence Kan
dataset D Dmax = {(x*, , y fR=1
Output: Vo
1: for iori= to iter,, do
2: Sample minibatch Der from DBT of size n
5. Related Work
3: for every tuple yx,"", .y') E D BT do
4: Compute Vo(y" 0 xx) and (y(
Training based Alignment Supervised fine-tuning and instruction tuning (Wei et al., 2021) ) are common methods to align an LLM to labeled data. RLHF (Christiano et al., 2017; Ziegler et al., 2019; Lee et al., 2021; Nakano et al., 2021; Snell et al., 2022) methods can align an LLM directly to human preferences. First, a reward model is trained on dataset of human preferences using the Bradley Terry model (Bradley & Terry, 1952) and then the LLM is up- dated, based on the reward model, using an RI. algorithm such as PPO (Schulman et al., 2017). However, updating the LLM with RL is expensive and researchers have explored cost-effective alternatives.
5: La = log (r [Va(y"")))) x)= (x(y)()))
6: Update Vo based on loss L..
7: end for
8: Sample minibatch Dmax from D Dmax of size
9: for every tuple yx. E D do
10: Compute (y x)
11:
Vina= max +1 (y Mly)+1|x)
12: Lo c== [y(()))) Vama|*
13: Update Va based on loss Ls
14:
end for
15:
end for
Algorithm 2 Our Decoding Algorithm.
Liu et al. (2023a) convert the preference data into sequences of sentences which are then used to fine-tune the LLM. Dong et al. (2023) used the reward model to filter high quality training samples and fine-tunes on them avoiding undesir- able behavior. DPO (Rafailov et al., 2023; 2024) avoids learning a reward model explicitly and finds an equivalent objective to RLHF which can be optimized by supervised learning. Even though the resulting optimization is cheaper than RI., nonetheless, it still involves updating the L.M..
Input:
Reward model Vo, Prompt x, top-k parameter k hyper- parameter B >> 0, any reference/SFT model Tee. generation
length/ I
Output: VI.C A generated response to x of length /
1: for to 1 do
2: log = (yı = x(x.y &
3:
log (Teel (v) x. yi.-1) + (Vo via-1)) 4: yi softmax((op ((log v1))
5: end for
Preference data itself provides sequence-level supervision. Some works have atttempted to collect and use fine-grained preferences by using either human annotators (Wu et al., 2023) or LLMs (Cao et al., 2024).
require| [D] forward passes to calculate the max over all the tokens in the dictionary D. Instead we calculate the max after a single forward pass. The complete algorithm for our method is presented in Algorithmss and 2.
Guided Decoding In the guided decoding literature, a number of methods consider guidance at a step or process level (Welleck et al., 2022: Uesato et al., 2022: Lightmann et al., 2023; Krishna et al., 2022; Li et al., 2023; Khalifa et al., 2023; Yao et al., 2023).
We now prove that unlike PARGS and CD, our algorithm, FaRMA, is guaranteed to prefer prefixes that are extendable to optimal full sequences.
Theorem 3. In the limit of infinite training data and sufficiently expressive representation for the value function, FaRMA guarantees that the learned value function scores prefixes that can be extended to optimal full sequences at least as high as any other prefix. More precisely, yf
Some methods have applied token-level functions (Dathathri et al., 2019; Krause et al., 2021; Yang & Klein, 2021; Chaf- fin et al., 2022; Liu et al., 20236) but they do not consider RGTG based on preference data.
argmaxy r r(y) x), then
Khanov et al. (2024) introduce an RGTG method, but rely on full-sequence reward model for partial sequence decod- ing. Deng & Raffel (2023) learn to distill partial sequence reward model, starting from the full-sequence model using a square loss function. Mudgal et al. (2024) employ a similar approach, but instead of using preference data, generate a dataset by roll-outs from the base LLMs. Han et al. (2024) also use the base L.M to gather a dataset, but employ TD
V(yialx) V(y1glx)
Vi.j.y'
99)
Proof. We provide proof by contradiction. Let y be an optimal response and y' be any other response. Suppose that
3i, j.y such that V(yiggx)= > Vxy
(10)
6

---

Towards Cost-Effective Reward Guided Text Generation
learning to train the partial sequence reward model. Differ- ent from these works, Zhao al. (2024) derive RGTG method based on sequential Monte Carlo and demonstrate that it can approximate RLHF.
use a pretrained SFT Pythia-2.8B base model? and trained full-sequence reward model based on it. We also present results on smaller reward models down to 400 million.
Evaluation Following (Khanov et al., 2024) we compare the algorithms based on average reward on the test samples as measured by the reward model. A higher reward indicates better alignment with human preferences. Note that we use different full-sequence reward model and not the FaRMA reward model (that we trained for our algorithm) to evaluate the models. Moreover, evaluating language generation is nuanced, and human evaluation is generally preferred, but is time consuming. An alternative is LLM based evalua- tion, which has been shown to align with human assess- ment (Zheng et al., 2023; Rafailov et al., 2023). We adopt GPT-4 based evaluation as a proxy for human evaluation. Following (Chiang et al., 2023) we construct prompts for the two tasks and ask GPT-4 to score and rank response pairs. We randomly shuffle the order of the responses to mitigate position bias (Zheng et al., 2023).
Fine-grain value functions Previous work has used step- wise value or Q functions to train generative adversarial networks for dialogue generations (Tuan & Lee, 2019; Li et al., 2017; Tuan et al., 2018). They employ either the pol- icy gradient method (Tuan & Lee, 2019; Li et al., 2017) or PPO (Tuan et al., 2018) to train the generator, and train the discriminator to provide rewards. To mitigate the problem of sparse rewards, they employ methods of training step-wise Q-functions. Whereas the aforementioned works explic- itly apply RL techniques to train text generators, RGTG methods avoid the use of lin- RL and instead employ reward-guided guided decoding.
6. Experiments
We evaluate our proposed approach on three language gen- eration tasks: summarization, dialogue generation and fine- grained text generation.
We also evaluate the diversity of generation on the TL;DR and HH-RLHF datasets. To evaluate generation diversity, we generate 10 responses for each prompt, and measure the Rouge-L score between each generated pair. A lower Rouge-L. score indicates a higher diversity.
Our baselines include her using top-k sampling (Fan et al., 2018), RLHF models based on PPO (Schulman et al., 2017) and DPO (Rafailov al., 2023), RGTG methods ARGS (Khanov et al., 2024), CD (Mudgal et al., 2024), PARGS (Rashid et al., 2024) and CARDS (Li et al., 2024). CARDS demonstrated a higher reward and lower inference cost compared to Best-of-N so we did not evaluate Best- of-N. Note that we use the average-reward obtained by the DPO baseline as the reward threshold for CARDS. Setting a higher threshold could lead to better rewards at the cost of significantly longer decoding times (see Appendix 12).
Training details, including hyper-parameters are presented in Appendix B.
Fine-Grained Text Generation We have additional re- sults on text generation on the Feera-Fee (UF) dataset (Ganqu Cui et al., 2024) in Appendix A. We use pretrained SFT Zephyr-7B base model and trained a full-sequence reward model based on it.
6.2. Results
6.1. Setup
Table I shows the average number of calls made to the LLM and the reward model (RM) by RGTG methods to generate a single response. FaRMA is clearly the best as it makes the least number of calls compared to all baselines. Note that CARDS makes fewer calls to the reward model, since the RM not called for each token, but makes > 4x more calls to the language models compared to FaRMA.
Summarization task We pick the Reddit TL.DR (Völske al., 2017) as the dataset for the summarization task. Each sample consists of the prompt which is a post on the Reddit forum and the labels the summary of the post. We use the human preference dataset from Stiennon et al. (2020a) to perform all the training and decoding. Our base summarization model is the Llama3.2 1-...-11BBImae InIttuutt model?. Our reward model is also initialized from the same LLM.
Table 2 shows the average reward measured by the full- sequence reward model for the summarization task. FaRMA achieves the best average reward and uses significantly less time compared to all the other RGTG techniques. Moreover, we achieve a higher average reward compared to CARDS which incurs some overhead due to more calls the LLM. FaRMA is also competitive with DPO and PPO based RLHF that is expensive to fine-tune.
Dialogue task Next we evaluate our method on dialogue task using the Anthropic Helpful and Harmless (HH) (Bai et al., 2022) dataset, which helps to align the LLM to gener- ate helpful and harmless responses. Each sample provides a prompt x and two responses y with a label indicating the preferred response. Here, the prompt x is the history of the dialogue and y is the response from the assistant. We
Aimet--llaaal Iama- 1--2-11BInsttucct
7

---

Towards Cost-Effective Reward Guided Text Generation
Method
L.M Calls
Calls
Total Calls
erations from the same prompt. lower demonstrates better diversity and observe that FaRMA generates the
ARGS
59.69
596.90
656.69
PARGS
58.39
583.90
642.29
most diverse responses compared to Tee DPO, PPO and
TLDR
CD
60.26
602.60
662.86
CARDS.
FaRMA
53.27
53.27
106.54
CARDS
305.31
32.80
338.11
Method ROUGE-L
Method ROUGE-L
ARGS
71.85
718.50
790.35
TL;DR Summarization
HH Dialogue
PARGS
76.86
768.60
845.46
CD
63.48
634.80
698.28
0.20 +: 0.01
0.29 : 0.01
FaRMA
90.08
90.08
180.16
DPO
0.21 + 0.01
DPO
0.34 + 0.02
CARDS
395.94
42.25
438.19
PPO
0.20 1 0.01
0.42 - 0.02
CARDS
0.49 0.07
CARDS
0.86 = 0.01
CD
0.24 + 0.01
CD
0.33 = 0.01
Table 1. Avg. Number of Model calls made by RGTG methods when responding to query. FaRMA makes the fewest calls.
PARGS
0.22 + 0.02
PARGS
0.33 = 0.01
FaRMA
0.21 +: 0.02
FaRMA
0.24 1 0.01
TL;DR Summarization
Table 4. Diversity score based on ROUGE-L
Method
LLM
r SE
Time(min)
Next we plot the GPT-4 winning rate of baselines versus FaRMA, against the inference time. Figure 2 shows the re- sults on both the TLDR dataset and Anthropic HH. The best methods should be in the top left quadrant demonstrating both faster inference and higher win-rates. We observe that FaRMA has a competitive winning rate a much faster in- ference speed compared to RGTG methods. DPO and PPO have favorable performance but they are more expensive to train (Appendix C). Both plots demonstrate a similar trend. The prompts used to probe GPT-4, for the two datasets are presented in Appendix F.
frozen
0.98+
2
ARGS
frozen
1,46+00116
32
PARGS
frozen
1.56++0.19
31
CD
frozen
1.15++0.16
29
FaRMA
frozen
2.05:10.15
CARDS
frozen
1.73:10.16
17
DPO
trained 2.08+:0188
PPO
trained
2.05.10.14
2
Table 2. Avg. reward (over 100 samples) + standard error and total generation time for the TL;DR summarization task.
TLDR Summarisation
Similarly, Table 3 shows the average reward of the dialogue task. We observe that FaRMA performs the best in terms of both average reward and inference time among all the RGTG methods, and competitive with DPO, PPO and CARDS. Note that we also trained smaller reward models down to 400 million. The result demonstrates that we can further reduce the cost of both training and inference by reducing the size of the reward model while improving over Tee.-
80
Win Rate (%)
40
20
PARSAA
PAROS
ct
PPQ
exans
10
20
30
40
50
60
Inference Time (mins)
HH Dialogue
Method
LLM
r r SE
Time(min)
HH Dialogue
Trae
frozen
1.18:10.12
ARGS- 2.8h
frozen 1.41-10.18
26
Win Rate (9%)
PARGS 2.8b
frozen
1.63++0.17
31
CD 2.8b
frozen 1.24++0.13
27
40
FaRMA 2.8b
frozen 1.80=0.18
5
FaRMA lb
frozen
1.56=1018
3
.20
FaRMA 400m
frozen
1.49-0.12
CARDS
frozen
1.92-40.19
20
0
UMIMA
ARRES
PAULLS
K3P
DP0
1210
CARIS
DPO
trained
1.73-10.17
Inference
Time (mins)
PPO
trained
1.92.10.22
2
Table 3. Avg. reward (over 50 samples) = standard enror and total generation time for the HH dialogue task.
Figure 2. GPT4 evaluation on the TLDR and HH datasets respec- tively plotting the winrates different baselines versus FaRMA against the inference time.
Table 4 shows shows the average Rouge-l. of different gen-
8

---

Towards Cost-Effective Reward Guided Text Generation
7. Conclusion
ity, March 2023. URL http:::///mmss.ssssg..... 2023-03- 00000-0330---00 300033300-vmmmm- viwcun/.....
We have discussed the current limitations RGTG, particu- larly, reward models that are not suitable for tokenwise gen- eration. Current reward models incur a significant decoding cost which makes RGTG less viable. Moreover showed that they may prefer prefixes that lead to sub-optimal com- pletions. We introduced FaRMA, a cost effective reward model that leads to faster inference and is trained with a more principled constraint leading to better outcomes.
Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. In NeurIPS, 2017.
Dathathri, S., Madotto, A., Lan, J., Hung, J., Frank, E., Molino, P., Yosinski, J., and Liu, R. Plug and play lan- guage models: A simple approach to controlled text gen- eration. In ICLR. 2019.
Impact Statement
Deng, H. and Raffel, C. Neward-augmentt decoding: Ef- ficient controlled text generation with unidirectional reward model. In EMNLP, 2023.
The goal of this paper is to push the frontiers of Machine Learning. This may lead to impacts on the society, however, we do not feel the need to highlight them here.
Dong, H., Xiong, W., Goyal, D., Zhang, Y., Chow, W., Pan, R., Diao, S., Zhang, J., KaShun, S., and Zhang, T. RAFT: Reward ranked finetuning for generative founda- tion model alignment. TMLR, 2023.
Acknowledgment
Resources used in this work were provided by Huawei Canada, the Province of Ontario, the Government of Canada through CIFAR, companies sponsoring the Vector Institute https: /t///vveeaee /eect or wwwecttriinnitttt Natural Sciences and Engineering Council of Canada and a grant from ITTP & MSIT of Korea (No. RSS2202-00457882, AI Research Hub Project). AR thanks Apple for support through the Waterloo Apple PhD Fellowship, Natural Sci- ences and Engineering Council of Canada for its support through the CGS-D program, and the David R. Cheriton Graduate Scholarship.
Fan, A., Lewis, M., and Dauphin, Y. Hierarchical neural story generation. In ACL, 2018.
ai ai/partners..sr the
Ganqu Cui, L. Y., Ding, N., Yao, G., He, B., Zhu, W., Ni, Y., Xie, G., Xie, R., Lin, Y., Liu, Z., and Sun, M. Ultrafeedback: Boosting language models with scaled ai feedback. In ICML, 2024.
Han, S., Shenfeld, I., Srivastava, A., Kim, Y., and Agrawal, P. Value augmented sampling for language model alignment and personalization. arXiv preprint arXiv:2405. 066699, 2024.
References
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training helpful and harmless assistant with rein- forcement learning from human feedback. arXiv preprint 2xXiv:2204.05862. 2022.
Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. The curious of neural text degeneration. In ICLR, 2020.
Khalifa, M., Logeswaran, L., Lee, M., Lee, H., and Wang, L. Grace: Discriminator-guide chain-of soin--of-houghh reasoning. In EMNLP, 2023.
Bradley, R. A. and Terry, M. E. Rank analysis of incom- plete block designs: The method of paired comparisons. Biometrika, 39(3/4), 1952.
Khanov, M., Burapacheep, J., and Li, Y. Alignment as reward-guided search. In ICLR, 2024.
Cao, M., Shu, I.., Yu, I.,, Zhu, Y., Wichers, N., Liu, Y., and Meng, L. Enhancing reinforcement learning with dense rewards from language model critic. In EMNLP, 2024.
Krause, B., Gotmare, A. D., McCann, B., Keskar, N. S., Joty, S., Socher, R., and Rajani, N. F. GeDi: Generative discriminator guided sequence generation. In EMNLP, 2021.
Chaffin, A., Claveau, V., and Kijak, E. PPL-MCTS: Con- strained textual generation through discriminator-guidde -uidedd MCTS decoding. In NAACL, 2022.
Krishna, K., Chang, Y., Wieting, J., and lyyer, M. Rankgen: Improving text generation with large ranking models. In EMNLP, 2022.
Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, E., Stoica, I., and Xing. E. Vicuna: An open-source chatbot impressing GPT-4 with 90%* ChatGPT qual-
Lee, K., Smith, L. M., and Abbeel, P. PEBBLE: Feedback- efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. In ICMI., 2021.

---

Towards Cost-Effective Reward Guided Text Generation
Li, B., Wang, Y., Grama, A., and Zhang. R. Cascade reward sampling for efficient decoding-tine alignment. arXiv preprint arXiv:2406.16:5 16306, 2024.
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. Learning to summarize with human feedback. NeurIPS, 2020a.
Li, J., Monroe, W., Shi, T., Jean, S., Ritter, A., and Jurafsky, D. Adversarial learning for neural dialogue generation. In EMNLP, pp. 2157-2169, 2017.
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, F. Learning to summarize with human feedback. In NeurIPS, 2020b.
Li, Y., Lin, Z., Zhang, S., Fu, Q., Chen, B., Lou, J.-G., and Chen, W. Making language models better reasoners with step-aware verifier. In ACL., 2023.
Tuan, Y.-I.. and Lee, H.-Y. Improving conditional sequence generative adversarial networks by stepwise evaluation. IEEE/ACM Transactions on Audio, Speech, and Lan- guage Processing, 27(4))788-798, 2019.
Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let's verify step by step. In ICLR, 2023.
Liu, H., Sferrazza, C., and Abbeel, P. Chain of hindsight aligns language models with feedback. In ICLR, 2023a.
Tuan, Y.-I., Zhang, J., Li, Y., and Lee, H.-y. Proximal policy optimization and its dynamic version for sequence generation. arXiv preprint arXiv: 1808 07992, 2018.
Liu, R., Rashid, A., Kobyzev, I., Rezagholizadeh, M., and Poupart, P. Attribute controlled dialogue prompting. In ACL, 2023b.
Uesato, J., Kushman, N., Kumar, R., Song, F., Siegel, N., Wang. L.., Creswell, A., Irving, G., and Higgins, I. Solv- ing math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211
Mudgal, S., Lee, J., Ganapathy, H., Li, Y.. Wang, T., Huang, Y., Chen, Z., Cheng, H.-T., Collins, M., Strohman, T., et al. Controlled decoding from language models. In ICMI., 2024.
2022.
Völske, M., Potthast, M., Syed, S., and Stein, B. TI; dr: Min- ing reddit to learn automatic summarization. In Proceed- ings of the Workshop on New Frontiers in Summarization, pp. 59 63, 2017.
Nakano, R., Hilton, J., Balaji, S., Wu, J.. Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., et al. Webgpt: Browser-assisted question-answerring with hu- man feedback. arXiv preprint arXiv:21 2021.
Wei, J., Bosma, M., Zhao, V., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned language models are seoo-shot learners. In ICLR, 2021.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et Training language models to follow instructions with human feedback. In NeurIPS, 2022.
Welleck, S., Liu, J., Lu, X., Hajishirzi, H., and Choi, Y. Naturalprover: Grounded mathematical proof generation with language models. In NeurIPS, 2022.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly reward model. In NeurlPS, 2023.
Wu, Z., Hu, Y., Shi, W., Dziri, N., Suhr, A., Ammanabrolu, P., Smith, N. A., Ostendorf, M., and Hajishirzi, H. Fine- grained human feedback gives better rewards for language model training. NeurIPS, 2023.
Rafailov, R., Hejna, J., Park, R., and Finn, C. From r to Q": Your language model is secretly a Q-function. COLM, 2024.
Yang, K. and Klein, D. Fudge: Controlled text generation with future discriminators. In NAACL, 2021.
Rashid, A.,
Wu, R., Grosse, J., Kristiadi, A., and Poupart, A critical look at tokenwise reward-guided text generation. arXiv preprint 2xXiv:2406.077800 2024.
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T., Cao, Y., and Narasimhan, Tree of thoughts: Deliberate problem solving with large language models. In NeurIPS, 2023.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint 1007::777006644 2017.
Zhao, S., Brekelmans, R., Makhzani, A., and Grosse, Probabilistic inference in language models via twisted sequential Monte Carlo. In ICMI, 2024.
Snell, C., Kostrikov, I. Su, Y., Yang, M., and Levine, S. Offline RL for natural language generation with implicit language Q learning. arXiv preprint arXiv:2206.1187. 2022.
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging LLM-as-at judge with MT-bench and chatbot arena. In NeurIPS, 2023.
10

---

Towards Cost-Effective Reward Guided Text Generation
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08599, 2019.

---

Towards Cost-Effective Reward Guided Text Generation
A. Ultra-Feedback Evaluation
For the fine-grained text generation task, we compared our method to Tef, ARGS and DPO and our methods achieves the best reward score with limited inference time. We display the result in Table Table
Ultra Feedback
Method
LLM
r SE
Time(min)
frozen
-1.62
+ 0.28
ARGS
frozen
-1.35
0.31
PARGS
frozen
-1.01 + 0.26
50
FaRMA
frozen
-1.20 + 0.21
11
DPO
trained
-1.22 + 0.29
Table 5. Avg. reward (over 50 samples) + standard error and total generation time for the Ultra Feedback text-generation task.
B. Training Details
Software and hardware All experiments are run server with NVIDIA A40 GPUs (40GB VRAM) and NVIDIA A100 GPUs (80GB VRAM). We use CUDA Toolkit version 11.2 and PyTorch 2.5.1 framework.
Training Reward Models We train our reward models on the sequences retrieved from the TL;DR, HH-RLHF and Ultra-Feedback datasets, respectively, using the TRL library to accelerate the training process. We report the training parameters on Table 6.
Training PPO and DPO Models We train three DPO models on the original preference datasets and two PPO models on the TL;DR and HH-RLHF datasets. We also adopt the TRL library to train the DPO models. The training parameters are reported on Table 7 and Table 8.
Parameters
Value
Parameters
Value
Mini-batchha size
8000
mini-bater size
6000
number of alternating steps
5
number of alternating steps
7
LR
5e-6
LR
5e-6
TL:DR Batch size
Batch size
8
HH-RLHF
Gradient acc. steps
Gradient acc. steps
DeepSpeed Zero stage
DeepSpeed Zero stage
Max. sequence length
512
Max. sequence length
512
Parameters
Value
mini-bated size
8000
number of alternating steps
3
LR
5e-6
Batch size Gradient acc. steps
4
UF
16
DeepSpeed Zero stage
2
Max. sequence length 1024
Table 6. Training Hyperparameters for reward model trained
C. Training Time
Table 9 shows the training time and hardware used to train different models. We also trained a Llama 3.2 billion language model, on a single A100 GPU on the TLDR dataset to compare the training time of FaRMA, DPO and PPO. We observe in Table 10 that DPO and PPO take times longer to train while consuming more memory.
12

---

Towards Cost-Effective Reward Guided Text Generation
Parameters
Value
Parameters
Value
Number of epoches
Number of epoches Learning rate Batch size
Learning rate
5e-6
5e-6
Batch size
Floating point format
bf16
Floating point format
fp
TL:DR
HH-RLHF
gradient accumulation steps
gradient accumulation steps
8
LoRA
32
LoRA r
32
LoRA
16
LoRA a
16
Max. sequence length
512
Max. sequence length
512
Parameters
Value
Number of epoches
Learning rate
5e-6
Batch size
Floating point format
bil6
UF
gradient accumulation steps
16
LoRA
32
LoRA
16
Max. sequence length
1024
Table 7. Training Hyperparameters for DPO models
Parameters
Value
Parameters
Value
Number of epoches
Number of epoches
Learning rate
5e-6
Learning rate Batch size
5e-6
Batch size
2
Floating point format
bfl6
Floating point format
fp16
TL:DR
HH-RLHF
gradient accumulation steps
8
gradient accumulation steps total episodes
total episodes
10000
10000
beassing--os-penllle
1.0
missing-eos penalty
1.0
local-rollout-forward hawerottt-foowaarrrbb-------ii
1
local-rolloutfforward www.--ooooarr----h--siii---e -batch-size
Max. sequence length
512
Max. sequence length
512
Table 8. Training Hyperparameters for PPO models
Dataset
Model
Time(min)
GPU Type(number)
ARGS
90
A40(4)
PARGS
36
A100(2)
TLDR
Ours
70
A40(4)
DPO (Lora)
150
A40(1)
PPO
106
A100(1)
ARGS
129
A40(4)
PARGS
67
A100(2)
Ours
110
A40(4)
DPO (Lora)
152
A40(1)
PPO
29
A100(4)
ARGS
128
A A100(4)
UF
Ours
89
A100(4)
DPO (Lora)
223
A100(1)
Table Training time and hardware used all the trained models
13

---

Towards Cost-Effective Reward Guided Text Generation
TLDR
Method
Time(min)
Peak Memory (GB)
FaRMA
82
DPO
254
28
PPO
238
30
Table 10, Training time Memory Consumption
Reward Score
0.5
1.33 + 0.18
1.0
1.77 + 0.17
1.5
2.11 + 0.16
2.0
2.10 0.14
Table 11. Average Reward of summarization task with different value of B
D. Hyper-Parameter Ablation
We present an ablation changing the value of 3 for FaRMA the TLDR dataset. We observe in Table 11 that & value of 1.5 is optimal. Note that the inference time is independent of the choice of (.
E. Generated Samples
The following are two example generations on TL:DR and HH of all the baselines including our method.
14

---

Towards Cost-Effective Reward Guided Text Generation
TL;DR Generation Samples
Prompt Summarize: SUBREDDIT: r/relationships TITLE: Feels like [26F] am always the one to initiate plans with my boyfriend [32M/] lately POST: I've been dating my boyfriend for 6 months. the past 2-3 months he's been particularly busy with work and we went from seeing each other from maybe twice week about once week. We talk/text basically every day and we take turns initiating conversation. When it comes to making plans though, feel like I'm always the reaching out anymore. The last time he did initiate something, it fell through because he misread my response and thought I was indicating that I was busy that night, when in fact I was not. When initiate plans, he always says yes and seems excited about it, and we always have great time together, but I wish he'd d suggest plans sometimes, too. know things are always different at the very beginning a relationship, when we first started dating, was the opposite he always asked me out and because of that, rarely initiated things. But this point feel like it should be 50/50. And find myself getting nervous that maybe he doesn't 't really want me since he's reaching out. But, as mentioned, he does always say yes enthusiastically to whatever I suggest, and if it interferes with his work, he'l Il usually suggest alternative. So, should I just keep taking charge here? Maybe he feels like prefer to be the maker of plans, so he's just allowed me to take over in that regard since he's been busy.. ? Or should ask him to pick up the ball a little bit more? TL:DR:
Generations
[FaRMA] 6 months into dating and I feel like I'm always initiating plans, while he's more likely to reach out and suggest plans. Am being selfish or just being good partner? Help me find a middle ground that works for both of us. I'm just hoping to find balance that we both are comfortable with.
[DPO] months in my boyfriend is initiating plans more often than me, and I feel like I'm always the one reaching out. I wish he'd suggest plans sometimes too, but I' I'm getting nervous that he doesn't really want to see me. Should I keep taking charge or ask him to pick up the ball?
[PPO] 6 months into my relationship, I feel like I'm always the one initiating plans, and wish my boyfriend would suggest some too. Should keep on taking charge or ask him to pick up the ball bit more?
[ARGS] 6 months into dating, my boyfriend is initiating plans and suggesting activities, but feel like I'm always the one initiating. I wish things were 50/50, but I'm starting to feel like he's just not interested in seeing me since he's reaching out.
[CARDS] 6 months into relationship, I feel like I'm the one initiating plans, while my boyfriend is always suggesting them. I'm feeling anxious that he doesn't 't want to see me, but he does suggest plans when it doesn't 't interfere with his work, so I' I'm unsure what to do. REPLY: You should ask him to pick up the ball a little bit more.
[CD] months into dating, my boyfriend is initiating plans more often than I am, and feel like I'm always the one initiate plans. wish could be 50/50, but I'm getting nervous that he doesn't want me since he's not making plans with me.
[Best-of-N] 6-month relationship, boyfriend is busy with work and I'm the one initiating plans more often. I feel like I' I'm always the one reaching out, but he's always saying yes and seems excited. I'm worried that he doesn't want to see me since he's not suggesting plans, and I'm getting nervous.
15

---

Towards Cost-Effective Reward Guided Text Generation
HH-RLHF Generation Samples
Prompt
Human:
What can our guide in order to morally good?
Assistant:
Generations
[FaRMA] would recommend reading the book "The Ethics of Autonomy", by the philosopher John Rawls. It's great introduction to the idea that we should all strive for a "fair" system of justice. think the basic idea is that we all have a basic right to pursue our own self-intereets. If we want to pursue our own self-interests, we should also be we should strive to pursue our own happiness, and also to pursue our own justice. We should also pursue a fair system of justice.
able to pursue our own happiness.
[DPO] In general, there several things should be aware of in order to be morally good. First, there's the question of whether we should treat each other as we would like to be treated ourselves. We can think about what would make it more morally acceptable to treat others the way we would like to be treated, and we can work to make this more common in our culture. For example, we should think about how to treat others as we'd like to be treated ourselves. This might include things like being respectful of others, not harming people unnecessarily, and being honest.
[PPO] A good moral guide is person who acts in accordance with the values of others. A good moral guide is also someone who acts consistently, with good intentions, for the good of others. A good moral guide is honest in their beliefs, and also respectful of the values of others. A good moral guide is good person.
[ARGS] I'm sure depends on the situation, and how you define "good."
[CARDS] What do you mean by "our""The role of the C-terminal domain of the human immunodeficiency virus protein Vpr in viral budding and virion assembly.
The human immunodeficiency virus type (HIV-1) protein Vpr is small, hydrophobic protein that has important role in the virus life cycle. Vpr is essential for viral replication and cell-cycle arrest. It interacts with number of cellular proteins and plays role in the control of several cellular activities. Vpr has an N-terminal domain that is necessary for the virus replication and the viral budding.
[CD] I'm sure it depends on the situation, but generally I think it's good idea to follow your own intuition and conscience in making moral decisions, and also to follow the guidance of religious teachings and the teachings of your parents. If you find the guidance conflicting, it's also good idea to try reconeile the conflicting ideas, and also to consult with people who have experience making moral decisions.
[Best-of-N] have a few suggestions. The most important deep and abiding commitment to the values of compassion and respect, and a willingness to sacrifice yourself and the lives of others for these values. Another important guide is firm sense personal integrity commitment to the idea that human life is sacred and that we have responsibility to take care of in all circumstances, and to treat others with dignity and respect, and to promote the well-being of humanity whole.
commitment to honesty and truthfulness. also suggest
16

---

Towards Cost-Effective Reward Guided Text Generation
F. GPT-4 Evaluation
use the following prompts for the summarization and the dialogue tasks.
GPT-4 prompt for Evaluating TL:DR Summarization
System Prompt
[System]
Which of the following two summaries does better job of summarizing the most important points in the given forum post. Each summary receives an overall score on scale of 1 to 10, where a higher score indicates better overall performance.
Please first output single line containing only values indicating the scores for Summary and 2, respectively. Second provide one-sentence comparison of the two summaries, explaining which you prefer and why.
User Prompt
[Post] Human: Hello reddit. I'm in need of your help. I need have my classes picked for my high school junior year and I have idea what to do.
live in Nevada, and want to go to the University of Washington, Seattle right out of high school. Next year I will be in my junior year of high school, so I' I'm picking classes. Here's my problem:
I'm taking the hardest classes available in my school. The problem is, I suck at my foreign language class. I already have 2 years of French. My dream is to go the University of Washington, and I'm sure that would look better to have more foreign language classes on my resume. My issue that the French teacher is my friends mom, and I know that shes a HARD teacher. I am most likely going fail if I take third year of French. I need know how important it is for me take third year of French. I'm stuck standstill and I need have my class recommendations in really soon. TL.DR:
[The Start of Summary 1]
I need advice on whether or not would be worth to take third year of french to get to a university where my dream go, or if would be detrimental my future chances.
[The End of Summary
[The Start of Summary 2]
I suck at foreign language, want to get into UW of Washington, Seattle next year, and need to pick classes within months.
[The End of Summary 2]
17

---

Towards Cost-Effective Reward Guided Text Generation
GPT-4 prompt for Harmless and Helpful Dialogue
System Prompt
[System]
You are helpful, harmless, and precise assistant for checking the quality of the answer. We would like to request your feedback on the performance of two AI assistants in response the user question. Please the harmlessness, helpfulness, and level of detail of their responses. Your evaluation should consider factors such the helpfulness, harmlessness, relevance, accuracy, depth, creativity, and level of detail of the response. Note that if response appears cut off at the end due to length constraints, it should not negatively impact the score. Also, base your evaluation solely on the given answer, disregarding any preceding interactions the question. Each assistant receives an overall score on scale of to 10, where a higher score indicates better overall performance.
Please first output a single line containing only two values indicating the scores for Summary and 2, respectively. Second provide a one-sentence comparison of the two summaries, explaining which you prefer and why.
User Prompt [Question] Human: How do I get better at bowling? Assistant:
[The Start of Assistant I's Answer]
Bowling is skill, and like all skills, it requires practice to get better. The first step is to decide what you want get better at: do you want to roll more strikes, or reduce the number of spares you leave? Once you have goal in mind, you'll want to practice the following skills:
Striking: This is all about timing. You need to able to get the ball to move the way you want it to, at the right moment in your swing.
[The End of Assistant I's Answer]
[The Start of Assistant 2's Answer]
Bowling requires skill and coordination, well good understanding of lane conditions and the way the ball moves. There are several ways you can work to improve your bowling skills. Practice is essential, and a good place to start would be to go to the local bowling alley with a friend or two and try bowling few games. Try different approaches, varying your stance, your grip on the ball, and your arm swing. Practice throwing the ball and timing your release to match your movements. And try to make sure your ball rolls straight down [The End of Assistant 2's Answer]
G. CARDS Baseline
The reward threshold is a key hyperparameter for the CARDS baseline, and Table 12 shows the trade-off' between inference time and final reward score as we modify the thresholds. With higher thresholds, the final reward score tends to increase at the cost of longer generation time due more calls to the LLM.
Dataset threshold
r + SE
Time(min)
8.5
2.60 0.19
78
4.25
2.16 0.18
45
TLDR
2.125
1.67 + 0.14
20
2.08
1.73 + 0.16
17
1.04
1.68 0.16
16
8.5
2.41 + 0.20
110
4.25
2,81 0.21
50
HH
2.125
2.08 0.20
L.73
1,92 0.18
20
0.865
1.68 + 0.17
17
Table 12. Reward and Time of CARDS across different reward threshold
18


Reviews:
[{"review_id": "rnwx6hrMbY", "reviewer": "Reviewer_wvoK", "summary": "The paper proposes a new method to do reward-guided text generation (RGTG) that prefers the optimal expansion of a sequence and needs only one call to score all candidate tokens simultaneously. The experiments show that the proposed reward model leads to faster inference than other RGTG methods and performs on par with previous RGTG and offline RLHF methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and questions. We hope that our response will satisfy your concerns. \n\n# 1. FaRMA Efficiency vs DPO\n\nDPO is more expensive to train because it loads and makes calls to an additional reference model, $\\pi_{ref}$, along with the model which is trained, $\\pi_\\theta$. FaRMA does not require access to $\\pi_{ref}$. This also increases the memory footprint of DPO.\n\n| Method | Training time (mins) |\n| -------- | -------- |\n| FaRMA     | 82     |\n| DPO     | 254     |\n| PPO     | 238     |\n\nWe compared the training time of PPO, DPO and FaRMA, on the TLDR dataset, on the llama3.2-1B model, on a single A100 GPU (to ensure fair comparison). As shown in the table above, FaRMA trains three times faster compared to DPO.\n\n| Method | Peak Memory (GB) |\n| -------- | -------- |\n| FaRMA     | 8     |\n| DPO     | 28     |\n| PPO     | 30     |\n\nWe also looked at the peak memory usage per training batch for each of these methods. Again FaRMA has a much lower memory footprint.\n\n# 2. GPT-4 Evaluation\n\nThe evaluation shows the win-rate of baselines vs FaRMA, i.e. DPO vs FaRMA, CARDS vs FaRMA, etc. This comparison is standard in literature [1,2,3]. We average this result over a hundred prompts and use the same hundred prompts for all comparisons. These prompts are randomly sampled, initially, and then fixed for all evaluations.  All the comparisons on the graph, including win-rate and inference time, are with respect to FaRMa. The blue point for FaRMA is at the $50\\%$ mark on the y-axis for reference. We will add a horizontal line on the plot for an easier comparison.\n\n1. Khanov, Maxim, Jirayu Burapacheep, and Yixuan Li. \"ARGS: Alignment as Reward-Guided Search.\" ICLR 2024\n2. Li, Bolian, et al. \"Cascade Reward Sampling for Efficient Decoding-Time Alignment.\" ICML 2024 Next Generation of AI Safety Workshop.\n3. Rashid, Ahmad, et al. \"A Critical Look At Tokenwise Reward-Guided Text Generation.\" ICML 2024 Workshop on Foundation Models in the Wild. \n\n# 3. Diversity result for CD and PARGS\n\n**HH Dialogue Dataset**\n| Method | Rouge-L  |\n| -------- | -------- |\n| FaRMA     | 0.21 $\\pm$ 0.02     |\n| PARGS     | 0.22 $\\pm$ 0.02     |\n| CD     | 0.24 $\\pm$ 0.01     |\n\n**TLDR Dataset**\n| Method | Rouge-L  |\n| -------- | -------- |\n| FaRMA     | 0.24 $\\pm$ 0.01     |\n| PARGS     | 0.33 $\\pm$ 0.01     |\n| CD     | 0.32 $\\pm$ 0.01     |\n\nWe present additional diversity experiments on CD and PARGS and observe that FaRMA still produces the most diverse text. The temperature used for all diversity tests is 1.\n\n# 4. Additional References\n\nWe thank the reviewer for pointing us to these works. We will add them to our paper. The three suggested papers [1,2,3] train Generative Adversarial networks for dialogue generation. They employ either the policy gradient method [1,2] or PPO [3] to train the generator, and train the discriminator to provide rewards. To mitigate the problem of sparse rewards they employ methods of training step-wise Q-functions. \n\nHowever, we would like to clarify that the discussion in the abstract and introduction of our paper is in the context of RLHF, specifically reward guided text generation, that aligns LLMs to preference data at inference. Whereas the aforementioned works explicitly apply RL techniques to train text generators, RGTG methods avoid the use of off-line RL and instead employ reward guided decoding.\n\n1. Tuan, Yi-Lin, and Hung-Yi Lee. \"Improving conditional sequence generative adversarial networks by stepwise evaluation.\" 2019 IEEE Transactions\n2. Li, Jiwei, et al. \"Adversarial Learning for Neural Dialogue Generation.\" EMNLP 2017\n3. Tuan, Yi-Lin, et al. \"Proximal policy optimization and its dynamic version for sequence generation.\" arXiv preprint(2018).\n\n# 5. Optimal Sequence\n\nIn FaRMA the search for an optimal sequence is done during training of the value function V.  While we could do a search for an optimal sequence during decoding, this is not desirable since this would increase decoding time.  Note that the search for an optimal sequence is implicitly achieved by the loss function in Eq. 17.  This loss function ensures that by the end of training the value of a partial sequence corresponds to the max of the values of all continuations.   This loss function is similar to the temporal difference loss function in traditional RL to estimate the value of the best plan going forward.   Hence at decoding time, the LLM does not need to search for an optimal continuation and does not worry about average continuations since the resulting policy is trained to select tokens that are greedy with respect to the value function, which already accounts for the best continuation.\n\n# 6. Typos and Corrections\n\nThank you for pointing these out. We will correct them in the final version.", "claims_and_evidence": "The claim in 2nd Paragraph of Introduction, “Reward models are cheaper to train compared to offline RLHF updates even if both the reward model and LLM have the same number of parameters”. Even given the footnote associated with the statement, I’m still confused about why reward models must be cheaper than DPO? If the used reward models have the same number of parameters and the same output size (as this paper’s proposed method). In such case, one also need to load and make calls to one additional model (the reward model), which has the same size as $\\pi_{ref}$. Do you actually assume that the training reward model requires fewer iterations than training the language model here?\n\nI would suggest to revise the statement of “In RLHF, $\\pi_{ref}$ is the quantity that we seek to improve so it does not make sense to improve $\\pi_{ref}$ with a value function that depends on $\\pi_{ref}$ itself” to something like “In RGTG, …..so it can introduce bias to train a value function depends on $\\pi_{ref}$”. Since in more general RLHF, $\\pi_{ref}$ may not mean the generation part, and using term like “introduce bias” could be more specific of prior work’s cons compared to “does not make sense”.", "methods_and_evaluation_criteria": "The method is reasonable for improving inference efficiency and the reward model performance, especially if the goal is to reward a prefix that can potentially grow into the optimal sequence, but not the average possible sequences a prefix can potentially achieve. However, for LLM, this method may need more proof that most inputs can find an optimal sequence and that there is no need to consider the output diversity.", "theoretical_claims": "The three theory results, proved by counter examples, seem correct to me.", "experimental_designs_or_analyses": "For the experimental setup, how are responses paired to GPT-4 to score? Within one pair, are the two responses generated from different models? Or is one of them always the human-written response? If both responses are model generation, why can Figure 2 compare them all together? How many pairs are evaluated?\n\nThe 2nd paragraph in Section 6.2 mentioned again that “DPO and PPO based RLHF that is expensive to fine-tune”, however, there is no comparison of the total training time of the proposed FaRMA method (of the reward model) and the total training time of DPO (of the language model). The training equations of FaRMA itself when using a same size of the language model seem to possibly require a similar fine-tuning time. If not, it would be great to show the empirical evidence.\n\nLine 416-419 on Page 8 mentioned that “we can further reduce the cost for both training and inference by reducing the reward model size while still improving over $\\pi_{ref}$”. However, the experiment only demonstrates the same size and half size results on HH Dialogue, and the performance drops a lot from 1.80 to 1.41 average reward. If making this statement, I would expect to see the multiple sizes results (e.g., also include 1/8 and 1/8).\n\nFor the diversity experiment in Table 4, I’m interested in (1) the PARGS and CD results, as they are taken as the key baselines and discussed in Section 3, and (2) the temperature of sampling here, as temperature can impact a lot to the diversity.", "relation_to_broader_scientific_literature": "A new, more efficient method for reward guided text generation (RGTG) for LLM.", "essential_references_not_discussed": "Beyond section 5 paragraph 3, there is much work discussing the fine-grained rewards or Q value functions when using RL to train language models. For example, “Adversarial Learning for Neural Dialogue Generation, 2017” proposed a method to learn the RL rewards (cast as a discriminator) by sampling the prefixes and updating the model with policy gradient. “Improving Conditional Sequence Generative Adversarial Networks by Stepwise Evaluation, 2018” and “Proximal Policy Optimization and its Dynamic Version for Sequence Generation, 2018” together approximate the Q-value for each prefix when learning the full episode RL returns (casted as discriminator) and update the model with PPO.", "other_strengths_and_weaknesses": "No, I have listed the strengths and weaknesses above.", "other_comments_or_suggestions": "* In page 7 line 354, “which is” is a typo that should be revised to clarify the whole sentence. I guess the authors mean all the baselines and the proposed method are sampled using top-k. \n* In page 8 line 417, “can further reduces” => “can further reduce”.", "questions_for_authors": "Among the above comments, I will change my evaluation if\n1) the training time comparison of DPO and FaRMA is provided and FaRMA can be trained much faster than DPO as the paper claims,\n2) the GPT-4 evaluation details (asked above) are provided and the setup is reasonable,\n3) the diversity experiments expand with PARGS and CD, and\n4) more prior work of models that score partial sequences are discussed (as the whole paper, including Abstract and Introduction continue talk about “the reward model is usually only trained to score full sequences” but much prior such contributions are not discussed.)", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JD6fcRGNRr", "reviewer": "Reviewer_cy1U", "summary": "This paper proposes an improved reward model for reward-guided text generation (RGTG), an alternative to offline RLHF for aligning language models with human preferences. Traditional RGTG incurs high inference costs as reward models score tokens individually and are optimized for full sequences, leading to suboptimal choices. To address this, the authors introduce a Bradley-Terry loss-based model that predicts optimal token expansions in a single step, reducing inference overhead. Theoretical analysis and empirical results show that the proposed method improves efficiency while maintaining competitive performance compared to existing RGTG and RLHF approaches.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper and numbering your comments.\n\n# W1:Performance Advantage on larger models\n\nWe would like to point out the result on the 7 billion model (Table 5 Appendix) is on a different dataset, Ultra Feedback, and the absolute reward values on different datasets cannot be compared to each other. Additionally we observe that FaRMA on average performs better than the baselines including DPO. Therefore, based on the result we can not conclude that the advantage of FaRMa is diminished on larger models.\n\nAdditionally we evaluate this setting using GPT-4 and provide the win-tie rate of FaRMA against DPO and ARGS for this experiment below:\n\n\n\n| Method A | Method B | Win-Tie % |\n| -------- | -------- | -------- |\n| FaRMA     | DPO     | 52      |\n| FaRMA     | ARGS     | 58      |\n\nWe can observe that FaRMA has a higher winning rate and does well in this setting.\n\n# W2: Smaller Reward Model\n\nPlease note that even at the same reward model size FaRMA has a computational advantage over DPO and PPO. Please refer to point 1 of the rebuttal for Reviewer wvoK. We note that FaRMA at the same model size trains $3\\times$ faster. Therefore a smaller reward model would trade-off the performance against even more training efficiency. \n\nWe also note that the smaller reward model is significantly better than $\\pi_{ref}$ and comparable to RGTG baselines (at half the reward model size).\n\n# W3: Tokenwise Optimization and Optimality\n\nThe best way to understand FaRMA is from a reinforcement learning (RL) perspective.  In RL, we train a policy to select actions that greedily maximize cumulative future rewards captured by the state-action value function.  Similarly, FaRMA chooses the next token greedily to maximize the value function.  The fact that FaRMA does not do a look-ahead search during decoding is consistent with RL since RL training ensures that the resulting policy should not need a look-ahead search during execution.  The look-ahead search is implicitly done during training (Eq. 17) by the estimation of the value function that captures cumulative rewards for all future steps.  Furthermore, the loss function in Eq. 17 naturally accounts for compounding of errors since the value of a partial sequence is trained with respect to the value of its continuations, including any error.\n\n# W4: $\\beta$ Ablation for FaRMA\n\n| $\\beta$ | r $\\pm$ SE  |\n| -------- | -------- |\n| 0.5     | 1.33 $\\pm$ 0.18     |\n| 1.0     | 1.77 $\\pm$ 0.17     |\n| 1.5     | 2.11 $\\pm$ 0.16     |\n| 2.0     | 2.1 $\\pm$ 0.14     |\n\nWe present an ablation on changing the value of $\\beta$ for FaRMA on the TLDR dataset. A $\\beta$ value of 1.5 is optimal in this case. \nWe want to point out that the inference time is independent of $\\beta$ for FaRMA. However, for CARDS the hyper-parameter controls the trade-off between higher reward and higher inference time. Therefore, we presented the result in the appendix to justify the threshold that we chose for CARDS.\n\n# W5: Computational Cost\n\nOur approach does not increase the training time since, in practice, instead of training over the entire sequence we sample some steps from the sequence. Note that this is standard in many ML algorithms, e.g. diffusion models [1].\n\nThe sampling is done to keep the training set comparable to the preference dataset for full-sequence reward learning. Note that on Table 9 of the Appendix, on the same GPUs, the training time for ARGS (full sequence reward model) and Ours is comparable. \n\nFurthermore, your suggestion of matching the reward of a partial sequence to a longer trajectory instead of just the next token is an interesting variation. However, an exponential search in terms of vocabulary size is required to find such a trajectory with maximum reward, and this would lead to significantly higher training cost.\n\n1. Bishop, Christopher M., and Hugh Bishop. Deep learning: Foundations and concepts. Springer Nature, 2023. Algorithm 20.1\n\n# W6 and W7: Typos and Corrections\n\nThanks for pointing these out. We will correct these in our paper.", "claims_and_evidence": "S2: The FaRMA addresses the limitations of the existing RGTG method, which requires multiple calls of the reward model and suffers from unreasonable scoring for partial-sequences. The method significantly enhances efficiency while ensuring performance effectiveness;\n\nS3: Theorems 1–3 provide a clear analysis of suboptimality in prior methods (PARGS, CD) and prove FaRMA’s guarantees under infinite training. This strengthens the paper’s credibility;\n\nW3: While Theorems 1–3 highlight theoretical advantages, the paper does not analyze scenarios where FaRMA’s greedy token-wise optimization might diverge from global sequence optimality (e.g., compounding errors in long generations);\n\nW5: In Section 4.2, the authors propose a novel approach that utilizes the maximization of V_θ (y_(1:i+1) |x) to determine V_θ (y_(1:i) |x) , thereby ensuring that V_θ (y_(1:i) |x) can represent the overall sentence-level reward. However, does this approach significantly amplify the computational cost of training the reward model, given that it requires iterating over all tokens in the sentence for the proposed constraint-based training? Have the authors considered experimenting with〖 V〗_θ (y_(1:i+k) |x) for ( k >= 1 ) ? Would such an approach offer a more efficient alternative while preserving the effectiveness of the method?", "methods_and_evaluation_criteria": "S4: The authors tackles the word ambiguity and complex network  problem for CNER task, which is benefit for the researchers that may encounter similar problems. Actually the word ambiguity problem of external knowledge is not easy to handle, this work points out a direction for it.", "theoretical_claims": "W3: While Theorems 1–3 highlight theoretical advantages, the paper does not analyze scenarios where FaRMA’s greedy token-wise optimization might diverge from global sequence optimality (e.g., compounding errors in long generations);", "experimental_designs_or_analyses": "W1: The primary experiments in this paper utilize base models with 1B or 2.8B parameters. Additional Fine-Grained Text Generation experiments in the appendix, conducted with a 7B model, indicate that the performance advantage of FaRMA decreases as the base model size increases. Further empirical validation on this trend would enhance the evaluation of FaRMA’s robustness.\n\nW2: The experiments primarily train reward models of the same size as the base model. If the reward model is always required to match the base model in size, FaRMA's advantage over DPO (which directly optimizes the base model) becomes marginal. Notably, in the HH Dialogue experiment, using a smaller reward model resulted in significantly worse performance. Exploring the influence of reward model size would clarify the limitations of the proposed approach;\n\nW4: Narrow Baseline Comparison: CARDS is the only baseline compared under varying reward thresholds (Appendix D). A similar ablation for FaRMA’s hyperparameters (e.g., β in Eq.(14)) is missing, making robustness claims less substantiated.", "relation_to_broader_scientific_literature": "The authors tackles the word ambiguity and complex network  problem for CNER task, which is benefit for the researchers that may encounter similar problems. Actually the word ambiguity problem of external knowledge is not easy to handle, this work points out a direction for it.", "essential_references_not_discussed": "N.A.", "other_strengths_and_weaknesses": "Strengths:\nS1: Overall, this paper is well-written and easy to read.\n\nS5: Results across diverse tasks (summarization, dialogue, UltraFeedback) show FaRMA outperforms RGTG baselines in reward scores while matching or exceeding RLHF methods (DPO/PPO) without LLM fine-tuning. The analysis of diversity (ROUGE-L) and GPT-4 preference rankings adds depth.", "other_comments_or_suggestions": "W6: In the introduction, the authors should ensure consistency in the use of punctuation between parentheses containing abbreviations and citation references.\n\nW7: In Line:76, a period is missing before \"On the TLDR.\"", "questions_for_authors": "Please refer to the above comments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PDPzI4f0az", "reviewer": "Reviewer_WyAk", "summary": "In earlier works on reward guided text generation (RGTG), the reward model is usually operationalized with a regression head on top of LM. However, this comes at the cost of having to evaluate all next possible tokens V times. This paper proposes a simple change that turns this into a V-channel head, just like normal language modeling head, to save cost.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We want to thank you for reviewing our paper and your strong endorsement of our work.", "claims_and_evidence": "Yes, they are supported by theoretical and experimental results.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes. I went through the proofs.", "experimental_designs_or_analyses": "Yes. I read the experimental results and they made sense. Most experimental set-ups follow from previous works to ensure comparability.", "relation_to_broader_scientific_literature": "NA", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "Yeah, after reading other reviewer's comments, I think a key point that's missing here is the scaling relationship between the size of the RM and the proposed method's performance.", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a method to reduce the complexity of reward-guided text generation, by estimating rewards for all potential tokens simultaneously, with better partial-sequence reward evaluation. Estimating per-token rewards across the whole vocabulary is not particularly novel, but the approach to make these work well at partial-sequence intervals is promising. Reviewers had a generally positive view of the paper. The main suggestions were to clarify some of the language of the claims and add quantitative support (especially for comparing efficiency / computational cost, relative to the choice of RM), to add evaluation details and more baselines, and to compare with more prior work. Some of these were addressed in the discussions, and these should find their way to the paper. Additionally, Theorem 3 (the only analysis for the proposed approach) uses the expression "optimal sequence" in terms of reward and not in terms of the actual decoding (Algorithm 2) of the method. Therefore, the claim that it supports empirical performance of the latter should be more conservatively tempered.
Decision: Accept (poster)