Preference-grounded Token-level Guidance for Language Model Fine-tuning
Authors: Shentao Yang
Shujian Zhang
Congying Xia
Yihao Feng
Caiming Xiong
Mingyuan Zhou
Venue: NeurIPS 2023 poster (2023)
Keywords: Preference Learning
Training Guidance Learning
Language Model Fine-tuning
Text Sequence Generation
TL;DR: An alternate training process that iterates between grounding preference into token-level training guidance and improving the language model with the learned guidance.

Abstract: Aligning language models (LMs) with preferences is an important problem in natural language generation. A key challenge is that preferences are typically provided at the *sequence level* while LM training and generation both occur at the *token level*. There is, therefore, a *granularity mismatch* between the preference and the LM training losses, which may complicate the learning problem. In this paper, we address this issue by developing an alternate training process, where we iterate between grounding the sequence-level preference into token-level training guidance, and improving the LM with the learned guidance. For guidance learning, we design a framework that extends the pairwise-preference learning in imitation learning to both variable-length LM generation and the utilization of the preference among multiple generations. For LM training, based on the amount of supervised data, we present two *minimalist* learning objectives that utilize the learned guidance. In experiments, our method performs competitively on two distinct representative LM tasks --- discrete-prompt generation and text summarization.

Preference-groundee Token-level Guidance for Language Model Fine-tuning
Shentao Yang', Shujian Zhang', Congying Xia',
Yihao Feng",
Caiming Xiong'. Mingyuan Zhou'
'The University of Texas at Austin "Salesforce Research shentao, yang@mccombs utexas edu, yihao aclimail. com
mingyuan.. zhou@mccombs. wuw.waa......................... edu
Abstract
Aligning language models (LMs) with preferences is an important problem in natural language generation. A key challenge is that preferences typically provided at the sequence level while LM training and generation both occur at the token level. There is, therefore, granularity mismatch between the preference and the LM training losses, which may complicate the learning problem. In this paper, address this issue by developing alternate training process, where we iterate between grounding the sequence-level preference into token-level training guidance, and improving the LM with the learned guidance. For guidance learning, we design framework that extends the wewwwise-prffrrnceeee learning in imitation learning both variable-length LM generation and the utilization of the preference among multiple generations. For LM training, based on the amount of supervised data, we present two minimalist learning objectives that utilize the learned guidance. In experiments, our method performs competitively on two distinct representative LM tasks discrete- prommpt generation and text summarization. Source codes are released at https://githubb.com www./Sentoo------ Baaittao-YAG//Peee YANG/Preferrence
http://ffeeenceGrrunneeeee Grounded Guidance.
Introduction
Language models (LMs) have been successfully trained with token-level cross-entropy losses, where each token position has corresponding term in the overall training losses [1-11]. Recent studies have shown that LMs can be further improved by aligning them with preferences from human feedback [12-15] or automatic evaluation metrics [16-18]. Typically, the preferences are only provided at the sequence level, e.g.. "Which of the two generated text sequences is better?" To align LMs with sequence-level preferences, there exist variety of approaches, such applying external filters the training texts [19], performing supervised learning on some curated/improved datasets [20-22], and optimizing the LMs based on learned sequence-level (pairwise-) preference predictor [14, 23 22]..
While these approaches have contributed to the development of several revolutionary products [e.g., 18, 15), mismatch issue has emerged from the perspective of guiding LM fine-tuning. Concretely, the sequence-level preference is not grounded into the token level, where LMM training losses occur. This means that there is mismatch in granularity between the feedback and training losses the preference is soarse-grained while the training losses are fine-grained. This issue is similar to the Selayed-feedback problem in reinforcement learning (RL) [26-28], where informative feedback is available only at the end of the trajectory (sequence) and not at any of the intermediate timesteps. Pre- vious studies have noted that this problem could have a negative impact on the empirical performance of the resulting LMs [29, 30), as it introduces a more challenging learning problem characterized by higher gradient variance and lower sample efficiency achieve the learning goal [31, 32].
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

To address this granularity mismatch, we focus on the following question: How can we effectively ground sequence-level preference into token-level guidance for LM fine-tuning? We propose an alternate training process that alternates between two stages: learning weefrence-grounnde level guidance and improving the LM using the learned guidance. This alternate process reduces the requirement supervised data and targets the low-data regime, e.g., few/zero-shot learning, where task-specific supervised (pre-)training is infeasible and initial LMs have weak zero-shot abilities.
token-
To ground the sequence-level preference into token-level guidance, we propose a framework for learning a token-level "reward" function', inspired by reward- Serardd-earning-ffeee crrtiinn-fro--rfffeee pree-prrfeeeeces in the imitation learning (IL) literature [33-35]. Specifically, we train the token-level rewards such that the corresponding evaluation for generated text sequence reflects the preference among multiple alternative generations, where the preference comes from task task-specific evaluation. While summation classically used in to aggregate the learned soken-level rewards into the text-sequence evaluation, L.M tasks can be different from classical II. tasks. To cater to LM generations, our ceettacce--earning framework can accommodate more careful choices of the aggregation function beyond the classical summation. For instance, in generating text prompts to steer LM for text classification, a "key token" in the prompt may be more effective than several mediocre tokens. Hence, using maximum to aggregate the token-level rewards may better reflect the text-sequence quality than summation.
To utilize the learned defference-rrunndee guidance LM training, we present two minimalist learning objectives that contain only minimal number of hyperparameters. These two objectives respectively target different amounts of supervised data in the specific LM task. We evaluate our framework on two distinct representative LM tasks: generating discrete text prompts for few-shot text classification and text summarization. On both tasks, our method exhibits competitive performance.
2
Main Method
Before diving into technical details, we will first establish the notation, provide some background on classical pairwise-preference learning, and offer an overview of our preference- grounding process.
Notation, In most LM tasks, we are given a dataset 'D = {(**, y of N supervised examples, where is the input to the LM, which can be a dummy, and y is the target text-sequence. We denote the LM parameterized by 0 as :y. The 8th generated token is denoted as de, given by at ~ TT(- 8t), where the context for token generation step Z 0 denoted as 8t, consisting of the LM input xr and the previously generated tokens d2t (Op.
1) Specifically, su x and 74 > 0, 8g = (r, (zz). The full generated text- sett-eeuence of length T' denoted as a (oo. ... ar-1). In most LM tasks, we have task-specific evaluation metric R(sT,3) E R that depends on the final context sy of the generated sequence and the target sequence y. with ST (x, a). The objective of LM training is often to maximize the expected task-specific evaluation, which can be expressed as
max. ((x.)))-BaBTIIT To(a:|ss) [R(sT (z,, a.. v)]
We model the learned token-level guidance as a bounded (reward) function (s(, ae) E [0, 1]. parametrized by 6. Unlike the original sequence-level preference or evaluation that is only available the final step T, the trained re can densely guide the token selection at each step < T..
Pairwise Preference Learning. In reward-leammn -erraadd--eannnn---e Jearning-from-preee prefeeenesss [e.g., 33-35], a dense reward function is learned such that the sum-aggregated reward for the entire generation trajectory aligns with the pairwise preference between two trajectories. In the context of LM generation, suppose we have two text-generation trajectories r' and 7¹ associated with the same LM input and target (s, v). taking the form 7' = {($6. ab),... (s/=-1, ")p=1)) with sequence lengths 7 and T7, respectively. Assume that 7' is preferred over TJ, denoted as T' > T². A token-level reward function r (st, az) is learned by requiring _t=0 ra at) > ET" ro(s), a.). Following the Bradley-Terry model of preferences [36], the seirwise--refereene loss for
learning
(()) - log eep
(IL'r+ ro(s).t)) re(4t) / EEeeuay exp (I2' re(st.af) [
which is often interpreted as binary classification the literature (37-39) In Eq. (1), summation Σ() is used to aggregate the learned token-level rewards into a parametrized sequence-level evaluation.
'We use the words "guidance" and "reward". "fine-tuning" and "training" interchangeably, depending on the specific context.

---

Token level Guidance Learning
Guidance Utilization
r' The place
& (sb)
Preference
LM
pf
x RRdding in city
Seq Evaal Po(rt)....eff
Guidance T
a Riding
Reward Tp(50_.0) .b))
Reward ra (a
apk
Roward Model Tip
Figure 1: Overview of the proposed framework. "AVG" denotes average, which is an example of the aggregation function f()) discussed in Section 2.1. "Seq Eval" refers to the parametrized sequence-level evaluations. The model choice of the reward function and LM depends on the specific task and is discussed in Section 4.
Overview. To ground the sequence-level preference into token-level guidance for LM training and thereby address the granularity mismatch discussed in Section 1, we present an alternate learning process that alternately learns the token-level guidance and trains the LM using the learned guidance.
For learning the token-level reward function that reflects the preference among multiple generated sequences. To utilize the learned snfference-groundee guidance, based on the amount of supervised data in the minimalist LM training approaches that require only
guidance, Section 2.1 we propose framework that learns
specific task, in Section 2.2 we present minimal tuning. In our framework, we iterate between the above two steps to mitigate the distribution shift between the text sequences used train the reward function and the text sequences evaluated by the reward function during LM training, taking into account that LMs can evolve during the training process. Our alcernate-eearing procedure is illustrated in Fig. 1.
2.1 Token-level Guidance Learning for Preference Grounding
Instead of applying the pairwise approach discussed above, we utilize the preference among multiple generated text sequences learn the reward function (5 ar). Intuitively, use more information to train the reward function at each optimization step. Therefore, our approach can be more efficient and effective, especially when the optimization budget limited.
Concretely, suppose we have K 2 2 generated text-sequences (a) input and target (x, with the associated generation trajectories (7 (T)..... Kj and with possibly unequal sequence-lengths (T ..... T ). Assume that there is preference ordering among these K sequences, where by "preference" full text-sequences. For description simplicity, let the preference ordering be al > .... > a < r! .... , We make assumption about the source of preference. may come from human ranking or some task-specific evaluation metric R on the full text-sequences, e.g., the descending ordering of the text-sequence evaluations R(s v)
K for the same LM
mean a ranking of text sequences based on
evaluations of
R(s
For trajectory n² {(>) at)..... (.....(5)) (st ,)), desired token-level reward function Te generates reward for each step as (so at) -1 A sequence-level evaluation es(T*) for trajectory y can be obtained by Ep(r²) f((ro(st (at.aa)) (") where f(-) is the aggregation function over all per-step rewards, e.g., the classical summation E.). Our goal is to train Te such that these parametrized sequence-level evaluations ( align with the given preference ordering 71 K Through this, the sequence- preference is grounded into token-level rewards re-
Under the Plackett-Luce choice model [40, 41]. the parametrized sequence evaluations ( induce probability distribution over all possible permutations of the integers (1....,K K}. We want maximize the likelihood of the given preference ordering ord- (I,..., K), K),ie.
/ (2)
min ((0) =: log P (ord (oo(+)))/<i), P (ard (co(++))
II exples(r"))
3

---

Algorithm 1 A learning routine for the preference- preference-grounded token-level reward function rá-
Input: The LM TT. initialized reward ra. aggregation function f()), reward-training steps Mrew- for iter E {1,.... Mrew) do
Use Th to generate K sequences {a* Jk= ii and get the preference ordering among {a LK fk=1 {a*}1 LK
With f()), get sequence evaluations {ea (T=(TT)) from Tii and optimize vi by Eq. (2). end for
When K = 2 and f() denotes summation, Eq. (2) reduces to the classical pairwise- prefeeeeee loss in Eq. (1). Therefore, reward-learning loss can be viewed extension of the classical pairwise loss. Further, Eq. (2) extends the ListMLE loss [42] in recommender systems into preference learning under multiple variable-length trajectories.
Algo. summarizes our reward-learnigg learning framework by describing an online-learning routine for training rs. An offline or hybrid version can be obtained with minor changes.
The Choice of Aggregation Function f(-). In classical IL tasks such as robotics [43], the robots are trained to stand or walk as long as possible. In this scenario, summation is a natural choice for the aggregation function f (()). However, in many text generation tasks, such as summarization, the generation quality may not be directly associated with the length of the generated text sequence. Nev- ertheless, suppose the token-level rewards are positive (i.e., r > 0), a longer sequence naturally has a higher sum of per-step rewards than shorter one, which can bias rs towards automatically ranking longer sequences higher. This bias can hinder our reward-learnine goal of aligning (T H with the given preference ordering. A naive numeric example is additionally provided Appendix C.
To mitigate the potential length bias the classical summation, we discuss three alternative choices of the aggregation function f()): average, soft maximum, and soft minimum.
Average. We define the average-aggregatee sequence-level evaluation " for trajectory 7* as
( TIE E Et'o " ro(s),a))) at), C
where C' is the average length of the K sequences. Multiplied by the average length C' has the benefit of scaling Eg the scale of egmm, which ensures numerical-ses -ccale consistency with cx euum and thus reduces hyperparameter tuning when switching from summation average aggregation.
Soft Maximum. We define the soft-maximun-aggrees
sequence-level evaluation eg (rh) as
max (7*) C 3 1- log EtTo exp(ra(s) /b)/))
where C is the average trajectory-length in Eq. (3) and is the temperature parameter.
Soft Minimum. The soft-miniiumm-ggreeaatee sequence-level evaluation e (min((*") follows Eq. (4) except for changing 3 to -3.
2.2 LM Training with Preference-groundee Token-level Guidance
Considering the supervised-date availability, we present two minimalist LM training objectives that utilize the learned preference- groundded guidance: 1) a REINFORCE-style update when there is no supervised data; 2) reward- weighted MLE when there are sufficient data. Our LM training directly starts from raw pre-trained checkpoints, without task -ask-specific supervised pre-training. This choice is to keep the algorithm general and consistent in both situations we consider, since task-specific pre-training may not be feasible the setting of few/zero-sho learning.
As shown in Algo. 1. we train the reward function r by the sequences sampled from LM Ty. Since task-specific pre-training to πT is not assumed, over the course of training, Re itself can evolve from a dess-preferred distribution to highly-preferee and keep r& as accurate guidance for LM training, we periodically re-estimate T during the first half of the LM-training process?, motivated by recent works in model-based RL [44-47].
one. To mitigate the impact of this distribution shift
In our preliminary study, we observed that this choice ("retraining the reward model only during the first half of the LM-training process") can save about 25-30% compute without hurting the performance much compared the vanilla reward retraining ("retraining the reward model throughout the entire LM training process").

---

Algorithm 2 An alternate-learning process for the reward function and the LM mp.
Input:
The dataset D. initialized LM ne, initialized reward function Te, LM-training steps MIM- reward-retrain period Mre. inputs for training the reward function specified in Algo. 1.
Initialize Tá by Algo. 1.
for iter E [LI...,,MM].. do if iterr S MLMM2 and iter % M, == 0 then
Re-train re by Algo. without re-initialization. end if
Optimize To by Eq. (5) or Eq. (6) with and rd- end for
Without Supervised Data. When the LM Te needs to discover good text generations by itself, the learned token-level reward ra can be used to provide dense guidance on generating each token, i.e. given the generation context 8t, select the next token d, such that r.a (8,, az) is high. Intuitively, for a generation trajectory т, if A (st, (8t,,a)) ) E T, (8 (x) is high, then the corresponding sequence-level evaluation (g(T) f(fr (st, ) can be also high, e.g., the average or summation of token- level rewards. The associated text sequence will thus be preferable since is trained to reflect the sequence-leve preference (Section 2.1). Through ra. the sequence-level preference is grounded into dense token-level guidance for LM training, without granularity mismatch or feedback delay.
With the learned r'g. minimalist implementation of this LM-training idea is the discrete-variable optimization problem
max; Ex- (..
( at)],
for each timestep t of which we calculate its gradient by the classical REINFORCE method [48- 50] since it can cope with large vocabulary size. Here, T denotes a generic sequence length. Additionally, since we want multiple text generations in typical LM tasks, instead of only one, we relax the convergence of the REINFORCE method by adding a standard max-entropy gradient, which can help capture multiple good eehavior-modes [51-53]. Thus, the LM Te trained by the gradient
[p--UnteemO....TT111 (Barsxo(- a.) wolro(ss an) Vo log (aa 8x)] + a VoH(TO( ( (-1)))) (5)
where H(TT(-1 s:)) is the Shannon entropy of nT( (t) and a is balancing coefficient.
With Supervised Data. With a labelled dataset = ((x",y)) } and with the learned preference- grounded guidance ra. minimalist enhancement of the classical MLE LM-training is the token-leve weighted-MM MLEE, where the per-token weight is given by the learned -arrarrf--onoonnnnoo Te. Our intention emphasize the important tokens in the given sequence y and downweight the unimportant ones, where the token importance given by ra grounds the sequence-level preference. Intuitively, this weighting scheme better utilize the L.M capacity and the optimization budget, and may thus improve upon the vanilla supervised loss [54, 16]. Specifically, the LM T is trained by
ming E( El 1lll1
(e(ee)))) We log No (ve st) with 8t = (xc, yee) and we EPP ro(s))))))) (6)
where lyl is the length of the target sequence y and w; is the self-normalized token-level reward. The standard seff-normalization is used to reduce the gradient variance among the samples in D.
Algo. 2 sums up the entire alternate-learning process, with the reward-learning routine in Algo. 1.
3 Related Work
Guiding Signals for LM Training. One string of works in LM training directly optimizes the LMS against the native sequence-level feedback such the test-time metric [e.g.. 3, 55-59, 32). This choice, however, may directly suffer from the delayed-feedbare issue discussed in Section and the subsequent high gradient variance and low sample efficiency [31, 32]. the recent trend of RL-based LM training, it has been common to incorporate token-level KL penalty towards the uniform model [12, 62-64, 14), or the base momentum model [65], to add to the delayed/ungrounded feedback. Although that KL. penalty does impact the RI-baaedd LM training at the token level, it is not tailored to the concrete task or the desired sequence-level feedback. When combined with the delayed-feed issue,
distribution [31, 60], the initial LM [23, 61], the sepervvisdd-ine-tuuee
5

---

could distract the LM training from improving the received fecdback/evvlluation. especially at the beginning of the text-sequence generation, which can however affect all subsequent token selections. By contrast, as seen in Eq. (5), even when added a max-entropy gradient, our preference-groundee token-level guidance can still provide dense, task-speeific, and feedback-oriented guidance on the selection of each token. For more detailed discussion on the RI. formulation of LM generation, the helayed-feedbace issue in RL-based LM training, and delayed feedback with KL penalty, please refer to Appendix F.
In some relatively "ideal" settings, prior works have attempted learn lask-specific token-level guidance for LM training. For instance, Shi et al. [66] use inverse RL, Guo et al. [67] propose a hierarchical approach, and Yang et al. [68] learn LM discriminators: but these methods require abundant expert data for supervised settings we consider. Under the same requirement of sufficient expert data, Lin et al. [69] learn a sequence-level adversarial-rankiin Monte-Carlo rollouts to simulate intermediate rewards, which can be computationally expensive and have high variance. Le et al. [71] use some values related to the sequence evaluation without explicitly learning per-token rewards. Pang et al. [72] learn a token-level error predictor for machine translation, but they rely on expert error-span annotations for each translation, which is highly demanding.
prra-ttranning, making them infeasible for the few/zero-shor
reward and Yu et al. [70] train GAN structure. They both use
By contrast, we propose a versatile framework for learning task -ask-specific token-level guidance for LM training that can ground the sequence-lever preference. Our approach is not limited to standard LM tasks and is also suitable for the low-data regime, with few assumptions about expert-data availability or preference source. In our experiments, we compare our method to recent RL-baaed approaches that train LM under delayed/ungrounded feedback with KI. penalty. We discuss additional related works on prompt generation, text summarization, and aligning LMs with preferences in Appendix E.
4 Experiments
We test our framework on two distinct representative text-sequence generation tasks: 1) input-agnostic discrete text-prompt generation for few-shot text-classificttion (Section 4.1), 2) the classical text summarization (Section 4.2). Our LM training directly starts from raw pre-trained checkpoints from HuggingFace [73], without task-specific supervised pre-training. Depending on the LM me used in the specific task, our reward function r. be implemented either decoder-only encoder-decoder model. Similar to prior works [e.g.. 32, 25]. given a text sequence a and an LM input x, the causal mask in transformers enables to get the learned guidance ro(st, ae) at each step of the sequence in parallel. Source codes have been publicly released.
an
4.1 Input-agnostic Discrete-prompt Generation
Overview. In discrete text- eext-proppt generation [e.g., 10. 74], we input discrete text-prompt a and an observation sequence o to a large pre-trained downstream LM TDLM a, o) to directly classify text o, without finetuning TDLM We follow the classical setting [e.g., 75, 60] to perform classification by selecting tokens corresponding to some predefined class labels. In our input-agnostic setting, the generated prompt is independent of the observation o. During inference time, only the learned prompts are used and the LM Ty is discarded. The initial input x to To is dummy, and the target the class label. We also adopt the standard few-shot setting [76], where both the training and validation sets have 16 (o, v) -piis per class. With a fixed length T. the goal is to find discrete text-prompts a = (ug. (T-1) that have high test accuracy. We simulate the sequence-level preference by the stepwise metric in Deng et al. [60], i.e., the higher value the better prompt. This choice ensures a fair comparison and avoids a potential overfitting - training and testing the LM on the same evaluation metric "accuracy". Appendix D discusses more details about the prompt task.
LM Training, Implementation, and Datasets. Since the prompt-generation task does not as- sume the availability of supervised data- the ground-truth prompts, the LM TΦ is trained by the REINFORCE-styLE update in Section 2.2 to discover eighly-accurate prompts by itself. We implement our framework on the codebase of RLPrompt [60], and adopt the standard datasets and most hyperpa- rameter settings in it. Reward training is reconducted every 1000 steps during the first 6000 steps of the LM training process and has early stopping. Reward function is learned with 5 sampled sequences and the temperature Eq. (4) is set as B = 2. The coefficient a in Eq. (5) a = 2-3. Appendix A.2 discusses the choices of these hyperparameters. The length of the generated prompts is fixed at 5. We

---

Table I: Test accuracy the prompt lask. Best overall result is bold and best discrete- pnacrete-prompll result is underlined different. The reported results are mean (standard deviation). We denote "BB Tuning-50" for Black-Box Tuning with mixed discrete and soft prompts that tunes the 50 soft tokens; and "AVG", "SUM", "MIN". "MAX" for our method with aggregation function average, summation, soft minimum, and soft maximum (Section 2.1).
SST-2
Yelp
AG News
Finetuning
Fow-shot Finctuning
80.6 (3.9)
88.7 (4.7)
84.9 (3.6)
Soft Prompt Tuning [83]
73.8 (109)
(2.6
(2.6
Continuous
Tuning-50 [78]
89.1 (0.9)
93.2 (0.5)
83.5 (0.9)
Prompt
AutoPrompt [84]
75.0 (7.6)
79.8 (8.3)
65.7 (1.9)
Manual Prompt [85]
82.8
83.0
76.9
In-Context Demo [10]
85.9 (0:7)
89.6 (0.4)
74.9 (0.8)
Instructions [86]
89.0
84.4
54.8
Discrete Prompt
GrIPS [87]
87.1 (1.5)
88.2 (0.1)
65.4 (9.5)
RLPrompt [60]
90.5
94.2 (0.7)
(2.1)
Ours (AVG/ SUM)
92.6 (1.7)
94.7 (0.6)
82.8 (1.5)
Ours (MIN)
91.9 (1:8)
94.4 (0.8)
82.4
Ours (MAX)
91.2 (2:5)
94.8 (0.5)
83.3 (1.4)
test on three popular few-shot datasets prior work [e.g. 77, 78]: two sentiment datasets SST-2 [79, 80] and Yelp Polarity [81], and topic four way-classifiictiionn dataset AG News [81, 82]. Additional details on the experiment and datasets are provided in Appendix B.
Results. We compare three variants of our framework with finetuning and with baselines in discrete- and continuous-promm generation. Since the generated prompts all have length in this task, the average aggregation is equivalent to summation. Table I shows the test accuracy, where we rerun the codebase of RLPrrmpt [60] under the same random seeds and evaluation script as our method." Other baseline results are from the literature [60, 88].
On all three tested datasets, our method shows competitive and stable results against the strong baselines not only in discrete-prompt generation, but also in heavier continuous-prompl tuning and finetuning the large downstream LMM. Based on Section the performance improvement achieved by our method compared to RLPrompt suggests that utilizing the token-level guidance learned by our approach, which grounds the task-specific preference, can be more effective than learning under golayed/ungroume feedback with KL penalty. Further, on both Yelp P. and AG News, using MAX aggregation better than the classical summation. Table 3 in Appendix shows examples of good generated prompts and their test accuracy. For instance, high-quality prompts on the AG News dataset often contain topic classification keyword, such as "Tags" and "Category". This aligns with our intuition that good prompts may be identified by a (few) "key" token(s), as discussed in Sections I and 2. Thus, the (soft-) maximmmma aggregation may better reflect prompt quality than summation.
4.2 Text Summarization
Overview. In the summarization task, we follow the standard setting [e.g.. 89, 61], where a set of supervised samples is available. The LM input r is the text to be summarized and the target y is the given summary. We simulate the sequence-leve preference by the classical Meteor score [90] and report the standard ROUGE scores [91], to avoid overfitting evaluation metrics as in the prompt task.
LM Training, Implementation, and Datasets. Since a supervised dataset ' is available in this task, the LM Ty can be trained by the weighted-MI.E objective in Section 2.2. This objective could be more stable and computationally efficient than REINFORCE-style methods in tasks of long-sequence generation. Due to limited computing resources, unless explicitly mentioned, we use the standard T5-small model [89] for both the LM and reward function. The reward training simply epoch of training on randomly sampled 10% of the training set and is repeated every 0.5 epochs during the first 2 epochs of LM training. Reward function learned with sampled sequences and again the temperature 3 = 2 in Eq. (4), Additional experiment details are in Appendix B.2. We test on the standard setting of two news summary datasets: CNN/Daily Mail (CNN/DM) [92] and XSum [93].
Results. We compare four variants in our framework with the standard supervised fine-tuning and RL-based methods PPO and NLPO in RL4LMs [61] under the environmental reward Meteor both
'There are small discrepancies between our reported Rroorppt results and the original paper's. We have confirmed our reproduced results both with RLPrompt's authors and with the recent TEMPERA paper [88].
7

---

Table 2: Results text summarization. We bold the best result of each metric on each dataset. The results of
Lead-3 on CNN/DM from Ramamurthy al. [61] and on XSum are from Lewis et al. [7]. Other baseline results are from our reruning RL4LMs' codebase [61] using T5-small. Number reporting formats follow Table
CNNiDailyMail ROUGE-2
XSum ROUGE-2
ROUGE-1
ROUGE-I
ROUGE-1
ROUGE-L
Lead-3
40.10
17,50
36,30
16,30
1.60
11.95
Supervised
38.88 (000)
16.22 (0.05)
32.58 (0.04)
31.79 (0.02)
9.68 (0.01)
24.70 (0.03)
PPO
39,16 (0.51)
17.37 (0.33)
33.77 (0:37)
23.18 (0.31)
4,46 (0.19)
16.07 (0.52)
Supervised + PPO
39.17 (1.5))
17.29 (1.44)
33.76 (0.53)
28.24 (0.39)
7.68 (0.13)
20.02 (0.23)
NLPO
38.90 (0.35)
17.22 (0:35)
33.51 (0:4))
22.97 (0:23)
4.53 (0.13)
15.62
Supervised + NLPO
39.27 (0.60)
17.41 (0.30)
33.85 (0.42)
28.08 (0.16)
7,68 (0:20)
19.88 (0.15)
Ours (AVG)
40.94 (0.02)
18.78 (0.03)
38.17 (0.03)
33.62 (0.05)
11.17 (0.02)
26.33 (0.05)
Ours (SUM)
40.70 (0.06)
18.48 (0U55)
37.93 (0.05)
33.27 (0.09)
10.83 (o.u
25.90 (0.95)
Ours (MIN)
40.78 (0.06)
18.67 (0.33)
38.01 (0.04)
33.57 (0.02)
11.14 (0.02)
26.30 (0.03)
Ours (MAX)
39.98 (0:08)
18.06 (0.03)
37.26 (0.06)
32.50 (0 14)
10.46 (0.12)
25.58 (2)
with and without task-specific supervised pre-training. For a fair comparison, the baseline results are from our rerunning RL4LMSs codebase with T5-small model as our method. Table 2 shows the mean and standard deviation of ROUGE- 1/2/L score across three random seeds.
On both datasets, our method shows favorable and stable performance against the classical and recent baselines. The better results of our method over supervised fine-tuning confirm the improvement of our swaar-wwiighted MLE over the vanilla supervised loss, discussed in Section 2.2. As in the prompt task, the gain of our method over RL-based baselines may indicate the benefit of utilizing our preference-groundee
token-level guidance over learning under delayed feedback with KL penalty. In this task, using average as the aggregation function outperforms the classical summation. This confirms our idea in Section 2.1 avoiding the interference of unequal sequence-lengths in training ra- Using MIN is also suitable for this task, since is not confounded by individual lengths and reflects overall text quality. Unlike the prompt task, using MAX unsuitable, since good summaries can hardly be identified by a few keywords. Overall, these results show the importance of customizing the aggregation choice for the specific LM task, key feature of our guidance- framework.
Further, to verify the performance of our method under a larger LM, we change the average variant of our method in Table 2 from T5-small to using T5-base LM. Fig. 2 (a) (e) compares our method on CNN/DM against the baselines, with an additional metric BertScore [94]. The baseline results directly cited from RLALMS [61] and are the per-metric best across their three environmental rewards. 5 Table 4 in Appendix A. shows the detailed numbers. It is clear that our method performs favorably against these strong baseline methods, especially in the ROUGE-L, BERTS Meteor, and ROUGE-2 metries. To further varify our method, we conducted a human study under the TS-base LM. The results are in Fig. 2 (f), with detailed setup and numeries in Table 5 of Appendix A.1. It is clear that this human evaluation on the summarization task supports the improvements in ROUGE, Meteor, and BertScore by our method. Further scaling-up of our method left as future work.
4.3 Ablation Study
This section discusses the following three research questions to better understand our framework.
(a): What will be the performance if we switch to using preference- -based sequence-level guidance?
To further study the gain of grounding preference into token-level guidance, we change the preference- based token-level reward in our method to the corresponding sequence-level reward. Fig. 3 shows the results when applying this change to our best variants in the prompt and summarization tasks in Sections 4.1 and 4.2, including the T5-base LM in Section 4.2, in comparison the best corresponding baselines. For summarization, we plot the average ROUGE scores, i.e., (ROUGE-1 + ROUGE-2 + ROUGE-L)/3 / 3. Table 6 in Appendix A. shows each ROUGE metric with standard deviation.
We see that learning and using preference- based sequence-leve guidance does not provide a significant advantage over those basclines that mostly directly work with the task-specific native sequence- feedback - the results are even much worse than the basclines in some datasets. Besides, the results of our sequence-level variants are generally less stable. These echo the harm of the delayed-feedhack feedback issue discussed in Section 1. Overall, this set of comparisons confirms that the gain of our framework
WWe carefully tuned the RL.LMMs baselines on several hyperparameters, which is detailed in Appendix B.2. 'The "ROUGE-L" here refers to "Rouge-LSum" in RL4LMs and HuggingFace, as detailed in Appendix B.2.
8

---

43
20
20.2
10.0
42.6
10.4
19.4
41
41.0
18
18.2
18.0
36
36.3
36.3
10.5
17.7
31
64.3
84.4
17
Sup
PPO
Sup
NLPO
Sup
Ours
Sup
PPO
Sup
NLPO
Sup
Ours
Sup
PPO
Sup
NLPO
Sup
Ours
PPO
NLPO
PPO
NLPO
PPO
NLPO
(a) ROUGE-1
(b) ROUGE-2
(c) ROUGE-L
1.0
89..
3.83
89
3.5
3.51
32
90.0
31.6
98
68.2
3.0
3.17
3.29
50.1
29.2
87.6
87.6
$7.5
2.52
2.5
Sup
PPO
Sup PPO
NLPO
Sup
Ours
Sup
PPO
Sup NLPO Sup PPO
Ours
Ref
Sup
Sup PPO
Sup
Ours
NLPO
NLPO
NLPO
(d) Meteor
(e) BertScore
(l) Human
Figure 2: CNN/DM summarization of our method and baselines under T5-base LM. "Sup" denotes "Suuervised". "Ref" denotes the ground-trure reference summary. Except for the human study in (f), baseline results directly cited from RLALMs [61] and are the per-metric best across their three environmental rewards.
Best Baseline
Ours under Sequenco-level
Ours Token-level Guidance
Figure 3: Performance of our method using sequence-leved and token-level preference preference-based guidance. "Best Baseline" refers the best result in the baseline discrete- promptt methods for the prompt task, and the best result over baseline methods
summarization
Error bars show one standard deviation.
mainly comes from our preference-groundee perspective, i.e., learning and using preference-based token-level guidance, rather than simply learning and using "a preference-base based guidance."
(b): How does method perform if we remove the rewardffuctionn retraining scheme?
To study the effect of guidance re-estimation, we remove the reward- funrd-funniion retraining scheme from our best variants in the prompt and summarization tasks in Sections 4.1 and 4.2, including the T5-base LM Section 4.2. Fig. 4 compares our methods with the best corresponding baselines. For the summarization task, again plot the average ROUGE scores. Table 7 in Appendix A.I shows each ROUGE metric with standard deviation. Appendix G discusses more on this re-estimation scheme.
Without guidance re- re-estimation, our method still performs competitively against the strong baselines, which corroborates the benefit of our preference-gee seffrennc-grounnee guidance. Fig. 4 also verifies our intuition in Section 2 that the gain of this scheme depends on the zero-shot ability of the initial LMs. Specifi- cally, in the prompt task where the initial LM has little zero-shot ability, -errrr-funniion retraining helpful to both improve performance and reduce variance. In the summarization task where the initial LM does have some zero-shot ability (as shown in Ramamurthy et al. [61]), guidance re-estimation indeed helps results not as much, since the distribution-See shift issue in Section 2 is less significant in this case. this task, both our variants, with and without reward retraining, outperform the baselines.
(c): What if we learn the token-level guidance by a different number of text sequences?
To study how the number of sequences used to learn the reward function impacts our method's performance, we vary this number in the AVG variant in Tables and 2. Fig. shows the prompt results on SST-2 and summarization results CNN/DM and XSum. For the latter, again plot the average ROUGE scores. The scores of each ROUGE metric are Tables and 9 of Appendix A.1.
Recall that the best baseline result on SST-2 in Table is 90.5, CNN/DM and XSum in Table respectively 31.3 and 22.06. Thus, our method is generally robust the number of sequences

---

Reward-function Retraining
Rewardd-uncttin Retraining
92
87
T5-base
Figure 4: Performance of our method with and without the reward- wwarrd-fnnciion retraining scheme. "Best Baseline" refers to the same in Fig. 3. Error bars show standard deviation.
95.00
32
23.75
32
92.00
32
20.65
91.00
32.50
(a) SST-2
(b) CNN/DM
(c) XSum
Figure Varying the number sequences learn token-level guidance, showing mean over random seeds.
used learn the guidance. Compared with the classical our framework has the flexibility in using multiple sequences. As illustrated in Fig. 5, using three or more sequences to learn the reward function can be generally more beneficial than using only two.
learning (Section 2),
Due to the page limit, we defer additional ablation study to Appendix A.2, where we (1) show that our framework is generally robust to the hyperparameter / in Eq. (4) and in Eq. (5); (2) further validate the harm of the delayed-feedback issue to the relevant LM-training methods on longer text-sequence generation; (3) show that the efficacy of framework is not tied to the specific preference sources considered this section.
Conclusion
To address the granularity mismatch between the sequence-level preference and the token-level LM training losses, in this paper, we develop an internate-laamiin process, where we iterate between grounding sequence-level preference into token-level training guidance, and training the LM with the learned guidance. Our method performs competitively two distinct representative LM tasks. Future work includes combining our sefference-groundeee guidance with RL-bbased LM training, and applying method to human preference and/or other tasks such as (task-oriented dialog systems.
Acknowledgments and Disclosure of F Funding
S. Yang. S. Zhang, and M. Zhou acknowledge the support of NSF-IS 2212418, NIH-R37 CA271186. the Texas Advanced Computing Center (TACC), and the NSF AI Institute for Foundations of Machine Learning (IFML). S. Yang acknowledges the support of the University Graduate Continuing Fellowship from UT Austin.
References
[1] Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. A neural probabilistic language model. Advances in neural information processing systems, 13, 2000.
[2] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017.
[3] Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. Self- critical sequence training for image captioning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7008-7024, 2017.
[4] Alec Radford and lya Sutskever. Improving Language Understanding by Generative Pre- Training. In arxiv, 2018
10

---

[5] Jacob Devlin, Ming- Wei Chang, Kenton Lee. and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
[6] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[7] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequene pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv: /tt1://90..341,,,, 2019.
[8] Xinjie Fan, Shujian Zhang. Bo Chen, and Mingyuan Zhou. Bayesian attention modules. Advances in Neural Information Processing Systems, 33:16362-16676, 2020.
[9] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy. Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 1rXiv:1907.11060 2019.
[10] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877- 1901, 2020.
[11] Shujian Zhang, Xinjie Fan, Bo Chen, and Mingyuan Zhou. Bayesian attention belief networks. In International Conference on Machine Learning, pages 12413-12426. PMLR, 2021.
[12] Nisan Stienmon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize from human feedback. Advances in Neural Information Processing Systems, 33:3008-300 (0088-301,2200 2020.
[13] Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. Recursively summarizing books with human feedback. arXiv preprint arXiv:2109. 10862, 2021.
[14] Long Ouyang. Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models follow instructions with human feedback. arXiv preprint arXiv:2203 0215,, 2022.
[15] OpenAI. Gpt-4 technical report, 2023.
[16] Govardana Sachithanandam Ramachandran, Kazuma Hashimoto, and Caiming Xiong. Causal- aware safe policy improvement for task-oriented dialogue. arXiv preprint arXiv:2103.0633 2021.
[17] Yihao Feng*, Shentao Yang*, Shujian Zhang, Jianguo Zhang, Caiming Xiong, Mingyuan Zhou, and Huan Wang. Fantastic rewards and how to tame them: case study on reward learning for task-oriented dialogue systems. In The Eleventh International Conference on Learning Representations, 2023.
[18] Yuntao Bai. Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08077 0870773, 2022.
[19] Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, and Emily Dinan. Recipes for safety in open-domain chatbots. arXiv preprini arXiv:2010.07079, 2020.
[20] Braden Hancock, Antoine Bordes, from dialogue after deployment: Feed yourself, chatbot! arXiv preprint 19X1v:1901.05415. 2019.
Mazare, and Jason Weston. Learning
[21] Irene Solaiman and Christy Dennison. Process for adapting language models to society (PALMS) with values-targeted datasets. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL https://openrewee net/forum?id=e ghaB9VZBw.

---

[22] Jérémy Scheurer, Jon Ander Campos, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. Training language models with language feedback, 2022.
[23] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv: 08593, 2019.
[24] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint ar arXfv:220..05862. 2022.
[25] Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chad- wick, Mia Glaese, Susannah Young, Lucy ing language models to support answers with verified quotes. arXiv preprint arXiv:2203.11147, 2022.
Geoffrey Irving, et al. Teach-
[26] Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welin- der, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
[27] Hao Liu, Alexander Trott, Richard Socher, and Caiming Xiong. Competitive experience replay. In International Conference on Learning Representations, 2019.
[28] Ishan Durugkar, Mauricio Tec, Scott Niekum, and Peter Stone. Adversarial intrinsic motivation for reinforcement learning. Advances Neural Information Processing Systems, 34:8622- 8636. 2021.
[29] Ryuichi Takanobu, Hanlin Zhu, and Minlie Huang. Guided dialog policy learning: Reward estimation for multi domain task-oriented dialog. arXiv preprint arXiv: 10719, 2019.
[30] Huimin Wang, Baolin Peng, and Kam-Fai Wong. Learning efficient dialogue policy from demonstrations through shaping. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6355-6365, 2020.
[31] Han Guo, Bowen Tan, Zhengzhong Liu, Eric Xing, and Zhiting Hu. Efficient (soft) l-learning for text generation with limited good data. Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6969-6991, 2022.
[32] Charlic Snell, Ilya Kostrikov, Yi Su, Mengjiao Yang, and Sergey Levine. Offline rl for natural language generation with implicit language q learning. arXiv preprint 2:X.::2206.1187/ 2022.
[33] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[34] Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations. In Interna- tional conference on machine learning, pages 783-792. PMLR, 2019.
[35] Daniel S Brown, Wonjoon Goo, and Scott Nickum. saaeer-thha-dmmostrattto imitation learning via automatically demonstrations. In Conference robot learning, pages 330-359. PMLR, 2020.
[36] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-344 1952.
[37] Ralf Herbrich, Thore Graepel, and Klaus Obermayer. Support vector learning for ordinal regression. 1ET, 1999.
[38] Yoav Freund, Raj Iyer, Robert E Schapire, and Yoram Singer. An efficient boosting algorithm for combining preferences. Journal of machine learning research, 4(Nov):933-96 2003.
12

---

[39] Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. Learning to rank using gradient descent. In Proceedings of the 22nd international conference Machine learning, pages 89-96, 2005.
[40] Robin L Plackett. The analysis of permutations. Journal of the Royal Statistical Society: Series C' (Applied Statistics), 24(2):133-00) 1975.
[41] R Duncan Luce. Individual choice behavior: A theoretical analysis. Courier Corporation, 2012.
[42] Fen Xia, Yan Liu, Jue Wang, Wensheng Zhang, and Hang Li. Listwise approach to learning to rank: theory and algorithm. In Proceedings of the 25th international conference on Machine learning, pages 1192- 192-1199 199, 2008
[43] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 TEEE/RSJ International Conference on Intelligent Robots and Systems, pages IEEE, 2012. doi: 10.1199/ROOS.201..6886UU
[44] Benjamin Eysenbach, Alexander Khazatsky, Sergey Levine, and Ruslan Salakhutdinov. Mismatched No More: Joint Model-Policy Optimization for Hosel-Based RL. ArXiv, abs/2110.02255,, 2021.
[45] Toru Hishinuma and Kei Senda. Weighted model estimation for offline model-based reinforce- ment learning. In Advances in neural information processing systems, 2021.
[46] Shentao Yang, Yihao Feng, Shujian Zhang. and Mingyuan Zhou. Regularizing a model-based policy stationary distribution to stabilize offline reinforcement learning. In International Conference on Machine Learning, pages 24980-25006. PMLR, 2022.
[47] Shentao Yang. Shujian Zhang. Yihao Feng, and Mingyuan Zhou. unified framework for alternating offline model training and policy learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[48] Peter W Glynn. Likelihood ratio gradient estimation for stochastic systems. Communications of the ACM, 33(10):75-84) 1990.
[49] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist rein- forcement learning. Machine learning, 8(3):229-256, 1992.
[50] Michael C Fu. Gradient estimation. Handbooks in operations research and management science, 13:575-616, 2006.
[51] Tuomas Haarnoja, Haoran Tang, P. Abbeel, and Sergey Levine. Reinforcement Learning with Deep Energy-Basee Policies. In International Conference on Machine Learning, 2017.
[52] Tuomas Haarojja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic: Off- Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1861-1870. PMLR, 10-15 Jul 2018
[53] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, G. Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, P. Abbeel, and Sergey Levine. Soft Actor-Critic Algo- rithms and Applications. ArXiv, abs/18 20//1812.0 1812055 2018.
[54] Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg, Scott E Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, and Nando de Freitas. Critic Regularized Regression. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 7768-768877777 -778. Curran Associates, Inc., 2020.
13

---

[55] Seonggi Ryang and Takeshi Abekawa. Framework of automatic text summarization using reinforcement learning. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 256-265, Jeju Island, Korea, July 2012. Association for Computational Linguistics. URL. https:/ haaas//////aaahoooogoyy org/D12-1000 1024.
[56] Marc' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojeiech Zaremba. Sequence level training with recurrent neural networks. arXiv preprint arXiv: 1XXxiv1511..6722 2015.
[57] Romain Paulus, Caiming Xiong, and Richard Socher. A deep reinforced model for abstractive summarization. arXiv preprint (.Ki::/700504304 2017.
[58] Raphael Shu, Kang Min Yoo, and Jung-Wor Ha. Reward optimization for neural machine translation with learned metrics. arXiv preprint arXiv:2
2021.
[59] Ximing Lu, Sean Welleck, Liwei Jiang, Jack Hessel, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi. Quark: Controllable text generation with reinforced unlearning. arXiv preprint arXiv:2205.13630 2022.
[60] Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric P Xing, and Zhiting Hu. Riprompt: Optimizing discrete text prompts with reinforcement learning. arXiv preprint arXiv: /2548, 2022.
[61] Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannanch Hajishirzi, and Yejin Choi. Is reinforcement learning (not) for natural language processing?: Benchmarks, baselines, and building blocks for natural language policy optimization. arXiv preprint arXiv:2210.00 22212111,.1,,,.. 2022.
[62] Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, A. Lapedriza, Noah J. Jones. S. Gu, and Rosalind W. Picard. Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog. ArXiv, abs/1907,,0456, 2019.
[63] Natasha Jaques, Judy Hanwen Shen, Asma Ghandeharioun, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Shane Gu, and Rosalind Picard. Human-centric dialog training via offline reinforcement learning. arXiv preprint arXiv:2010.05848, 2020.
[64] Shujian Zhang, Chengyue Gong, and Xingchao Liu. Passage-mask: A learnable regularization strategy for retriever-reader models. arXiv preprint arXiv:2211.00915, 2022.
[65] Louis Castricato, Alexander Havrilla, Shahbuland Matiana, Michael Pieler, Anbang Ye, Ian Yang, Spencer Frazier, and Mark Riedl. Robust preference learning for storytelling via contrastive reinforcement learning. arXiv preprint 21X1v:2210.07792, 2022.
[66] Zhan Shi, Xinchi Chen, Xipeng Qiu, and Xuanjing Huang. Toward diverse text generation with inverse reinforcement learning, arXiv preprint arXiv: / 1255, 2018.
[67] Jiaxian Guo, Sidi Lu, Han Cai, Weinan Zhang, Yong Yu, and Jun Wang. Long text generation via adversarial training with leaked information. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018
[68] Zichao Yang, Zhiting Hu, Chris Dyer, Eric P Xing, and Taylor Berg-Kirkpatrick, Unsupervised text style transfer using language models as discriminators. Advances in Neural Information Processing Systems, 31, 2018.
[69] Kevin Lin, Dianqi Li, Xiaodong He, Zhengyou Zhang, and Ming-Ting Sun. Adversarial ranking for language generation. Advances in neural information processing systems, 30, 2017.
[70] Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. Seggan: Sequence generative adversarial nets with policy gradient. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
[71] Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu Hong Hoi. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. Advances in Neural Information Processing Systems, 25:21814-21328, 2022.
14

---

[72] Yuanzhe Richard Pang. Vishakh Padmakumar, Thibault Sellam, Ankur P Parikh, and He He. Reward gaming conditional text generation. arXiv e-prints, pages arXiv-2211, 2022.
[73] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface 's trans- formers: State-of-the--a natural language processing. arXiv preprint arXiv:1910.03777 2019.
[74] Shengnan An. Yifei Li, Zeqi Lin, Qian Liu, Bei Chen, Qiang Fu, Weizhu Chen, Nanning Zheng, and Jian-Guam Guang Lou. Input-tuning; Adapting unfamiliar inputs to frozen pretrained models. arXiv preprint a:Xiv:2003031331, 2022.
[75] Timo Schick and Hinrich Schütze. It's not just size that matters: Small language models are also few-shot learners. arXiv preprint arXiv:2009.07111 2020.
[76] Ethan Perez, Douwe Kiela, and Kyunghyun Cho. True few-shot learning with language models. Advances in Neural Information Processing Systems, 34:1 1054-11070, 2021.
[77] Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. arXiv preprint arXiv:2012.15723, 2020.
[78] Tianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang, and Xipeng Qiu. Black-box tuning for ianguug--moddll------sssss arXiv preprint arXiv:2201.03514, 2022.
[79] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang. Christopher D Manning, Andrew Ng. and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631-1642, 2013.
[80] Shujian Zhang, Xinjie Fan, Huangjie Zheng, Korawat Tanwisuth, and Mingyuan Zhou. Align- ment attention by matching key and query distributions. Advances in Neural Information Processing Systems, 34:13444-13457, 2021.
[81] Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28, 2015.
[82] Shujian Zhang, Chengyue Gong, Xingchao Liu, Pengcheng He, Weizhu Chen, and Mingyuan Zhou Allsh: Active learning guided by local sensitivity and hardness. arXiv preprint arXiv:2205.044000 04980, 2022.
[83] Brian Lester, Rami A1-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2 S0kiv:2204.08000 104.08691, 2021.
[84] Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. Auto- prompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010. 15980, 2020.
[85] Timo Schick and Hinrich Schütze. Exploiting cloze questions for few shot text classification and natural language inference. arXiv preprint arXiv:2001.07676 2020.
[86] Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task general- ization via natural language crowdsourcing instructions. arXiv preprint 21X14:2104.08773, 2021.
[87] Archiki Prasad, Peter Hase, Xiang Zhou, and Mohit Bansal. Grips: Gradient-free, edit-based instruction search for prompting large language models. arXiv preprint wrXiv:2203.0728/ 2022.
[88] Tianjun Zhang, Xuezhi Wang, Denny Zhou, Dale Schuurmans, and Joseph E Gonzalez. Tempera: Test-time prompting via reinforcement learning. arXiv preprint 1rX.v22211.11890, 2022.
[89] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter Liu, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 116 2020.
15

---

[90] Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metric for MT evaluation with improved correlation with human judgment.. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65-72, Ann Arbor, Michigan, June 2005. Association for Computational Linguisties. URL. https:/ fttps://acllanthology org/ arg/W05-0909.
[91] Chin- Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summariza- tion Branches Out, pages 11, Barcelona, Spain, July 2004, Association for Computational Linguistics. URL www.hhtttt///eetm htttts:://cllaaalllllaaaeo wrw.:////../.00//11111.
[92] Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. Advances in neural information processing systems, 28, 2015.
[93] Shashi Narayan, Shay B Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv: 08088880..888.. 2018.
[94] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv: 190.00679 2019.
[95] Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations, 2014.
[96] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05100. 2017.
[97] Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.08207. 2021.
[98] Guanghui Qin and Jason Eisner. Learning how to ask: Querying Ims with mixtures of soft prompts. arXiv preprint arXiv:2104.000 104x:2204.00599 2021.
[99] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190, 2021.
[100] Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. Warp: Word-level adversarial reprogramming. arXiv preprint arXiv:2101.0012/, 2021.
[101] Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. arXiv preprint arXiv:2103.10385 2021.
[102] Zexuan Zhong, Dan Friedman, and Danqi Chen. Factual probing is [mask]] Learning vs. learning to recall. arXiv preprint (04.::20000054400 2021.
[103] Shizhe Diao, Xuechun Li, Yong Lin, Zhichao Huang, and Tong Zhang. Black-box prompt learning for pre-trained language models. arXiv preprint arXiv:2201.08531, 2022.
[104] Daniel Khashabi, Shane Lyu, Sewon Min, Lianhui Qin, Kyle Richardson, Sameer Singh, Sean Welleck, Hannaneh Hajishirzi, Tushar Khot, Ashish Sabharwal, et al. Prompt wayward- ness: The curious case of discretized interpretation of continuous prompts. arXiv preprint 1xXi::2122.08348. 2021.
[105] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint 107.
2021.
[106] Yusheng Su, Xiaozhi Wang. Yujia Qin, Chi-Min Chan, Yankai Lin, Zhiyuan Liu, Peng Li. Juanzi Li, Lei Hou, Maosong Sun, et al. On transferability of prompt tuning for natural language understanding. arXiv preprint arXiv:2111.06719 2021.
[107] Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization, 2019.
16

---

[108] Mandy Guo, Joshua Ainslie, David Uthus, Santiago Ontanon, Jianmo Ni, Jun-Hsuan Sung. and Yinfei Yang. Longt5: Efficient text-to-text transformer for long sequences. arXiv preprint arXiv:2112.07911 2021.
[109] Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. Su-sum: Large-scale multilin- gual abstractive summarization for 44 languages. In Findings of the Association for Com- putational Linguistics: ACL-IJCNLP 2021, pages 4693-4703, Online, August 2021. As- sociation for Computational Linguistics. doi: (1186533112022 wet.ll...iinnnnal.... acl.413. URL https://aclantthll.g org/2021 finding.nninngaccl.c. .413.
[110] Mohammad Norouzi, Samy Bengio, Navdeep Jaitly, Mike Schuster, Yonghui Wu, Dale Schuurmans, et al. Reward augmented maximum likelihood for neural structured prediction. Advances In Neural Information Processing Systems, 29, 2016.
[111] Richard Yuanzhe Pang and He He. Text generation by learning from demonstrations. arXiv preprint Stti:2009.07839. 2020.
[112] Sayan Ghosh, Zheng Qi. Snigdha Chaturvedi, and Shashank Srivastava. How helpful inverse reinforcement learning for table-to-text generation? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing (Volume 2: Short Papers), pages 71-79, 2021.
[113] Marcin Junczys-Dowmunt, Roman Grundkiewicz, Shubha Guha, and Kenneth Heafield. Ap- proaching neural grammatical error correction as a low-resource machine translation task. arXiv preprint arXiv: wexiv::804.05000 (8040099440 2018.
[114] Shujian Zhang, Chengyue Gong, and Eunsol Choi. Knowing more about questions can help: Improving calibration in question answering. arXiv preprint arXiv:2106.0144 064444 2021.
[115] Shujian Zhang, Chengyue Gong, and Eunsol Choi. Learning with different amounts of annotation: From zero to many labels. arXiv preprint arXiv:2 109.040899 2021.
[116] Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim. Gunjan Chhablani, Nihal Nayak, Dehajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang. Matteo Manica, Sheng Shen, Zheng Xin Yong. Harshit Pandey, Rachel Bawden, Thomas Wang. Trishala Neeraj. Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations, 2022. URL htttps:://ppereeeee wttppeerrriiee.eeefoormrmm net/forum? id=9Vrb9DOWI4.
[117] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-fine finetuned language models. arXiv preprint arXiv:2210.11 14116, 2022.
[118] Muhammad Khalifa, Hady Elsahar, and Mare Dymetman. A distributional approach to controlled text generation. In International Conference Learning Representations, 2021.
URL /tttts::/opeereviiiii net/ weet//foum?iii wttf/fooommmm///daaeear id=jjkw45-99W9b
[119] Tomasz Korbak, Hady Elsahar, Germán Kruszewski, and Mare Dymetman. On reinforcement learning and distribution matching for fine-tuning language models with catastrophic forgetting. arXiv preprint arXiv:2206.00760 2022.
[120] Dongyoung Go, Tomasz Korbak, Germán Kruszewski, Jos Rozen, Nahyeon Ryu, and Marc Dymetman. Aligning language models with preferences through f-divergence minimization. arXiv preprint arXiv:2302. 2.2200...00.. 2023.
[121] Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Bhalerao, Christopher L Buckley, Jason Phang, Samuel Bowman, and Ethan Perez. Pretraining language models with human preferences. arXiv preprint arXiv:2302.08533 2023.
17

---

[122] Changyeon Kim, Jongjin Park. Jinwoo Shin. Honglak Lee, Pieter Abbeel, and Kimin Lee. Preference transformer: Modeling human preferences using transformers for RL. In The Eleventh International Conference on Learning Representations, 2023. URL https:// http://www..eeeoooo/ooooooooo..
[123] Richard S Sutton and Andrew 2018.
Barto. Reinforcement learning: An introduction. MIT press,
[124] Shentao Yang. Zhendong Wang, Huangjie Zheng, Yihao Feng, and Mingyuan Zhou. A regularized implicit policy for offline reinforcement learning, arXiv preprinl arXiv:2202.0667. 2022.
18

---

Appendix for "Preference-groundeel Token-level Guidance for Language Model Fine-tuning"
Table of Contents
1
Introduction
2 Main Method
2.1 Token-level Guidance Learning for Preference Grounding
2.2 LM Training with Preference-groundee Token-level Guidance
3
Related Work
4
Experiments
4.1 Input-agnostic piscrete-prompt Generation
4.2 Text Summarization
4.3
Ablation Study
5
Conclusion
10
Additional Experimental Results
20
A.1 Tabular Results
20
A.2 Further Ablation Study
22
B Additional Experiment Details
24
B.1 Prompt Generation
24
B.2 Text Summarization
25
C A Naïve Numeric Example for the Average Aggregation
26
Details on the Prompt Generation Task
26
E More Related Work
27
A Discussion on Applying RL Methods LM Tasks
28
F.1 LM Generation as Token-level MDP
28
F2 Delayed Feedback in RL-based LM Training
................................
29
F33 Sparse Reward with KL Penalty
................................
29
G Further Discussion on the Guidance Re-estimation Scheme
30
H Potential Negative Societal Impacts
30
I Limitations
31
Computational Resources
19

---

A
Additional Experimental Results
A.1
Tabular Results
Table 3: Examples of the generated discrete input-agnostic text-prompt and their classification accuracy the
corresponding test
SST-2
AG News
Prompt
Accuracy
Prompt
Accuracy
guys filmmaker filmmaker rated Grade
94.18
newslntroduction Comments Tags Search
85.78
MovieMooieFill rated Grade
94.18
newsTopic Blog Support Category
85.55
Rated CinemaScoreReporting Grade
94.01
news RecenfecendPPooosnnntiicccc
84.53
employment theater rated Oscars Grade
93.96
news Recent Brief Latest
84.51
scene filmmaking rated comedian Grade
93.85
newsVirtualBlogllgNe
84.33
Table 4: Detailed results on CNN/DM summarization under TS-base LM for Section 4.2. We bold the best result of each metric. Baseline results are directly cited from RL4LMs [61]. "Env. Reward" denotes the environmental reward in RL4LMs. The "ROUGE-L" GEG here refers to Lougee-Suum" in RL4LMs and in the Hugging Face interface, which discussed details Appendix B.2. Section 4.2, we plot the results our method with the average aggregation, which is the best variant in Table 2. We report the mean (standard deviation) of our method over three random seeds.
Algorithm
Env. Reward
ROUGE-1
ROUGE-2
ROUGE-L
Meteor
BertScore
Lead-3
40.1
17.5
36.3
33.3
87.4
Supervised
41.1
17.7
34.3
30.9
87.6
Rouge-1
41.0
18.2
34,9
27.6
87.6
PPO
Rouge-Avg
39.6
17.6
33.8
27.0
87.4
Metcor
40.8
17.8
34.2
30.1
87.3
Rouge-1
40.4
18.0
34.4
27.5
87.5
NLPO
Rouge-Avg
40.4
17.7
34.4
27.4
87.4
Meteor
40.5
18.0
34.3
29.2
87.2
Rouge-1
41.7
18.9
35.8
27.8
88.2
Supervised PPO
Rouge-Avg
42.5
19.4
36.3
29.6
88.2
Meteor
42.6
19.4
36.1
31.6
88.0
Rouge-1
42.1
19.3
36.1
28.7
88.2
Supervised NLPO
Rouge-Avg
42.4
19.3
36.3
29.5
88.2
Meteor
42.9
19.4
36.1
31.9
88.0
Ours (AVG)
43.09 (0.06)
20.17 (0.04)
39.99 (0.07)
35.23 (0.06)
89.61 (0.12)
Ours (SUM)
42.86 (0.08)
19.92 (0.08)
39.76 (0.11)
34.74 (0.37)
24 (0.11)
Ours (MIN)
42.92 (0.14)
20.01 (0.02)
39.84 (0.08)
34,88 (0.13)
89.33 (0.07)
Ours (MAX)
42.38 (0.17)
19.49
39.34 (0.09)
34.13 (0.32)
89.09 (0.19)
Setup and results of the human evaluation. Table below presents the results of our human evaluation on CNN/DM summarization under the T5-base LM. We generally adopt the protocol in Stiennon et al. [12] to evaluate the overall summary quality, Our model is compared with the baselines Supervised, SuPervised+PPO, and Supervised+NLPP in RL4LMS [61]. The result the reference summaries is also presented, which is intended for sanity check rather than method comparison. In conducting this evaluation, we randomly picked 100 articles in the test split of CNN/DM and showed to 20 qualified evaluators the summaries generated from each method, along with the article. The method names were anonymized. The evaluators were asked to read the article and score each summary. Summaries are scored on a 5-Point Likert Scale 21, 3, 4, 5). where score 5 is the highest and I the lowest. From Table it is clear that human evaluation supports the improvements in ROUGE, Meteor, and BertScore by our method in Table 4.
20

---

Table Average human ratings on CNN/DM summarization under the TS-base LM. bold the best result apart from the ground-truth Reference summary. A detailed description on the setup is in the above text.
Supervised
Supervise++PPO
Supervised+NLPO
Ours
Reference
Average Human Rating
2.92
3.17
3.29
3.61
3.88
Table 6: Scores each ROUGE metric for method using sequence-level and token-level preference-based guidance in summarization tasks in Section 4.3 (a). "Seq." denotes our method with sequence-level preference-basee guidance, and "Token" denotes our method with token-level preference- baseeceeebaed guidance. The reported numbers are mean (standard deviation) over three random seeds. The row "Average" shows the average of the three ROUGE scores, Le.. (ROUGE-1 ROUGE-2 ROUGE-L)
CNN/DM
XSum
CNN/DM (TS-base LM)
Seq.
Token
Seq.
Token
Seq.
Token
ROUGE-1
40.20 (0.07)
40.94 (0.02)
32.56 (0.08)
33.62 (0.03)
42.10 (0.15)
43.09 (0.06)
ROUGE-2
17.80 (0.08)
18.78 (0.03)
9.98 (0.04)
11.17 (0.02)
19.23 (0.11)
20.17 (0.04)
ROUGE-L
37.08 (0.06)
38.17 (0.03)
25.11 (0.07)
26.33 (0.05)
38.09 (0.14)
39.99 (0.07)
Average
31.69
32.63
22.55
23.71
33.14
34.42
Table Scores on each ROUGE metrie our method with without reward- frrnd-fnnniion retraining scheme the summarization tasks Section 4.3 (b). "Without Retrain" denotes our method without retraining, and "With Retrain" denotes our method with reward-function retraining. The reported numbers are mean (standard deviation) over three random seeds. The row "Average" shows the average of the three ROUGE scores, i.e. (ROUGE-1 ROUGE-2 ROUGE-L)/3.
CNN/DM
XSum
CNN/DM (T5-base LM)
Without Retrain
With Retrain
Without Retrain
With Retrain
Without Retrain
With Retrain
ROUGE-1
40.83 (0.10)
40.94 (002)
33.45 (0.11)
33.62 (003)
42.98 (u08)
43.09 (0.6)
ROUGE-2
(0.0))
18.78 (.03)
11.07 (0.0))
11.17 (002)
20.09 (0.6)
20.17 (0.09)
ROUGE-L
38.07 (0.09)
38.17 (0.8))
26.23 (0.10)
26.33 (0.05)
39.87 (0.08
39,99 (0.07)
Average
32.53
32.63
23.58
23.71
34.31
34.42
Table 8: Scores each ROUGE metric for the summarization task on CNN/DM in Section 4.3 (c), where we vary the number of sequences used learn token-level guidance. The reported numbers are mean (standard deviation) over three random seeds. The row "Average" shows the average of the three ROUGE scores, i.e. (ROUGE-1+ ROUGE-2 ROUGE-L)/3.
Number of Sequences
3
5
9
ROUGE-1 1
40.80 (0.06)
40.94 (0.02)
40.87 (0.09)
40.86 (0.08)
40.95 (0.01)
ROUGE-2
18.70 (0.04)
18.78 (0.03)
18.71 (0.02)
18.74 (0.06)
18.78 (0.01)
ROUGE-L
38.05 (0.03)
38.17 (0.03)
38.09 (0.07)
38.08 (0.08)
38. 18 (0.02)
Average
32.52
32.63
32.56
32.56
32.64
21

---

92.0
95
32.51
91.5
90
91.0
32.48
85
90.5
80
32. 45
2
2-1
2"
21
2
2-
2-1
20
21
2
0.06
0.08
0.1
0.125
0.15 0.2
(a) Temperature 3 (SST-2)
(b) Temperature / (CNN/DM)
(c) Balancing Coeff. (SST-2)
Figure Line plots comparing the performance under different values of hyperparameter in Eq. (4) and in Eq. (5). The plotted numbers are mean over three random seeds. Error bars show one standard deviation.
Table Scores on each ROUGE metric for the summarization task on XSum Section 4.3 (c), where we vary the number of sequences used to learn the token-level guidance. The reported numbers are mean (standard deviation) over three random seeds. The row "Average" shows the average of the three ROUGE scores, i.e., (ROUGE-1+ ROUGE-2 ROUGE-L)/3.
Number of Sequences
2
3
5
7
9
ROUGE-1 1
33.54 (0.06)
33.62 (0:03)
33.56 (0.08)
33.56 (0.02)
33.63 (0.02)
ROUGE-2
11.12 (0.04)
11.17 (0.02)
11.12 (0.05)
11.19 (0.05)
11.20 (0.03)
ROUGE-L
26.26 (0.06)
26.33 (0.05)
26.28 (0.06)
26.34 (0.06)
26.36 (0.03)
Average
23.64
23.71
23.65
23.70
23.73
A.2 Further Ablation Study
In learning the preference-basee sequence-level guidance in Section 4.3, the aggregation function f(-) in Section 2.1 is removed, since it is inapplicable and unnecessary to the sequence-level reward function. For the minimalist LM training objectives Eqs. (5) and (6) in Section 2.2, we change them to the corresponding versions that sequence-leve guidance. Self neff-normalization in reward-weighted MLE Eq. (6) is removed, since it is again inapplicable and unnecessary to the sequence-leve setting.
In this section, we continue our discussion in the Ablation Study (Section 4.3) by answering the following additional questions on our method.
(a): Is our method robust to the hyperparameter(ss): temperature 3 and balancing coefficient a??
To study the choice of the temperature parameter 3 in the soft-maximummmiinuus aggregation Eq. (4), we vary the value of 3 in the MIN variant in Tables I and 2 from = 2. Furthermore, to study the balancing coefficient in the REINFORCE-stye vary the a parameter in the AVG variant in Table from a = 2-3 Fig. 6 respectively shows the prompt results on the SST-2 dataset and the summarization results on the CNN/DM dataset. For summarization, we again plot the average ROUGE scores, with the breakdown scores of the three ROUGE metries in Table 10 below.
L.M-training approach Eq. (5),
Recall that the best bascline result on SST-2 Table is 90.5, and on CNN/DM in Table 2 is 31.3. We see that our method can achieve competitive results on a relatively wide range of the temperature 3. A too-small value of 3, such as 0.25 and 0.5, may incur a harder optimization problem and thus an inferior performance both prompt and summarization tasks.
For the choice of the balancing coefficient a, we see that our method provides competitive results in relatively wide range of a E [0.08, 0.15], when compared to the best baseline result of 90.5 in Table A too-small value of a may not prevent the REINFORCE-style method from pre-mature convergence. The resulting LM therefore may not sufficiently explore the sampling space or capture multiple good behavior modes, resulting an inferior and highly varying performance. A too-large value of distracts the optimization of the LM, and again leads to worse result.
22

---

Table 10: Scores each ROUGE metrie for the summarization task on CNN/DM, where we vary the temperature parameter / in the seft-mininum aggregation Eq. (4). The reported numbers over three random seeds. The row "Average" shows the average of the three ROUGE scores, i.e., (ROUGE-1 ROUGE-2 + ROUGE-L)/3.
(standard deviation)
3=2-2
B = 2-1
3
20
2
| = 22
ROUGE-1
40.77 (0.11)
40.74 (0.09)
40.79 (0.11)
40.78 (0.06)
40.80 (0.01)
ROUGE-2
18.67 (0.06)
18.68 (0.05)
18.68 (0.09)
18.67 (0.03)
18.71 (0.04)
ROUGE-L
38.00 (0.10)
37.98 (0.08)
38.03 (0.12)
38.01 (0.04)
38.02 (0.01)
Average
32.48
32.47
32.50
32.49
32.51
(b): How does our method perform in generating longer prompts compared with the baseline?
To further validate the harm of the delayed-feedbace feedback issue to the related LM- AM-training methods that learn under the sparse sequence-leve feedback, we compare our method with RLPrompt [60] on generating prompts with length increased from 5 to 10 and to 20 tokens, on the SST-2 dataset. Table 11 below shows the results.
Table 11: Test accuracy on the prompt task on the SST-2 dataset, for our method and RLPrompt on generating prompts with length of 5, 10, and 20 tokens. We report the and standard deviation over three random seeds.
RLPrompt
Ours (AVG)
Performance Gap
5 Tokens
90.5 (1.5)
92.6 (1.7)
2.1
Tokens
75.8 (7.6)
86.0 (2.9)
10.2
20 Tokens
65.2 (6.0)
80.9 (4.5)
15.7
We see that RLPrompt performs worse than our method on generating longer prompts. In particular, the performance gap increases as the prompt length (feedback delaying) increases. This comparison can further demonstrate the harm of the Selayed-feedbace issue in training lext-generation LMs, and that our framework, in particular our cefference-groundde token-level guidance for LM training, is a viable solution it.
is intrigued that the results of both methods deteriorate with the prompt length. After checking the generated prompts from our method, we find that longer prompts mostly contain many repeated tokens, as shown by the following example prompt of length 20
which is separated into two lines at the location "\" due to the page-width limit. In this prompt example, the tokens Movie and Video are each consecutively repeated seven times, and the bi- gram ImageVideo is repeated two times. Such prompts with heavy repetitions may confuse the downstream classifier. This aligns with our intuition that a clear and suceinct instruction is preferable than long but verbose one.
As side note, in generating Table 11, RLPrompt. It is possible that RIPrrompt requires careful tuning for generating longer prompts, due to the delayed-feedie long-prompt generation as a future work.
the default hyperparameters for both method and
issue that we try to address. We leave thorough tuning of RLPrompt
(c): Is the efficacy of our framework tied to the specific preference sources considered in Section 4?
To investigate whether the performance of our framework is tied to the specific preference-soure sources considered in the experiment section (Section 4), inspired by RL4LMs [61], we simulate the sequence- level preference on the summarization task by using another two automatic metrics "Rouge-avg" and "Rouge-avg2" rather than the classical Meteor score [90] in Section 4. Table 12 below presents the ROUGE scores of our method under each of the three preference sources on the CNN/DM dataset under the T5-base L.M. For a more thorough investigation, we provide the results for our method both with and without the guidance re-estimation scheme. The baseline results in Table 12 below come from the best baseline method in Table 4 of Appendix A.
A detailed description of the prompt task is deferred to Appendix D.
23

---

Table 12: Results for method on CNN/DM summarization under T5-basc LM when using different automatic metrics simulate the sequence-level preference. We provide the detailed ROUGE scores for method both with and without guidance re-estimation. "Baseline" denotes results of Appendix A.I. The reported numbers are the mean over three random seeds. The row "Average" shows the average of the three ROUGE scores, i.e. (ROUGE-1 1 + ROUGE- + ROUGE-L) /3
best baseline method in Table
With Guidance Re-estimation
Without Guidance Re-estimation
Baseline
Rouge-avg
Rouge-avg2
Meteor
Rouge-avg
Rouge-avg2
Meteor
ROUGE-1
42.9
43.14
43.07
43.09
42.96
42.98
42.98
ROUGE-2
19.4
20.18
20.12
20.17
20.07
20.05
20.09
ROUGE-L
36.1
39.93
39.89
39.99
39.80
39.77
39.87
Average
32.8
34.42
34.36
34.42
34.28
34.27
34.31
Concretely, these two automatic metrics "Rouge-avg" and "Rouge-avg2" are constructed as Rouge-avg = 0.5 ROUGE-I + 0.5 x ROUGE-2 + 0.5 ROUGE-L,
Rouge-avg2 0.5 x ROUGE-1 + 0.5 x 2 ROUGE-2 + 0.5 x ROUGE-L,
where the "Rouge-avg" metric is exactly the same as that in the RL4LMs [61]. The "Rouge-avg2" metric is constructed by multiplying ROUGE-2 by 2 to make its numerical value similar to the others.
It is clear that changing the preference source from Meteor to these two alternative metrics does significantly alter the performance of our method, especially when compared to the performance improvement of our method over the best baseline method in Table 4 of Appendix A.1. This set of comparisons confirms that the efficacy of our framework is generally not tied to a specific preference source. could also further corroborate the effectiveness of our cnfferencc--roundiiig perspective on guiding the LM training.
B
Additional Experiment Details
B.1 Prompt Generation
Implementation Details. based on the official codebase of RLPrompt available at https://github. http::/gidlbb.ccm/miigaaiii proomt,, and the Hugging Face library [73]. We have provided some implementation details in Section 4.1. Here we continue the discussion.
To ensure fair comparison, the implementation of our framework is
The LM Ty is parametrized as a frozen distilGPT-2 model with parameter 0 being one MLP-layer of size 2048 inserted right before the output head. The token-level reward function ra implemented as a distilGPT-22 with a two-layer projection-ML.. MLP of sizes 2048 and on top. The LM Ty is trained by maximum of 2000 steps, with early stopping based the validation set. The reward training is reconducted every 1000 steps during the first 6000 steps of the LM training process and is (and almost always) early stopped. RoBERTa-large is used [9] as the pre-trained downstream LM TDLM-
Datasets. We use the standard datasets provided in the RLPrompt codebase [60]. We test on three popular few-shot classification datasets in prior work [e.g.. 77, 78], i.e., two sentiment binary- classification datasets SST-2 [79] and Yelp Polarity [81]. and the topic
dataset AG News [81] 1]. In keeping with the standard few-shot setting [76], both the training and the validation sets have 16 examples per class. To mitigate the randomness in the few-shot setting, each dataset is subsampled into five few-shot training and validation sets, while the test set is standard. We train our models on each few-shot (sub- -dbaastt with three random seeds and evaluate three generated prompts in each case. For all three tested datasets, we report the average test accuracy and standard deviation across all evaluated prompts in all random seeds and all few-shot (sub-)datasets
Hyperparameters. Apart from the hyperparameters discussed in the ablation study (Section 4.3 and Appendix A.2), most other hyperparameters as well as the training and evaluation procedures of our framework follow RLPrompt. Additionally, our reward model in Table 13, and important hyperparameters for training our LM Table 14. The generated prompts have fixed length of 5. The same hyperparameters
list the important hyperparameters for training
used in all tested datasets.
Baselines. For the baseline results in Table 1, we rerun the codebase RL Promrppp under the same random seeds and evaluation script our method. Other bascline results are from the literature
24

---

160, 88]. We note that our reported RLPrompt results have some small discrepancies compared to the original paper's results. We have confirmed our reproduced results with RLPrompt's authors and with Table of the recent TEMPERA paper [88].
Table 13:
Hyperparametees for training our reward model the compp--eneration task.
Table 14: Hyperparameterrs for training LM in the task.
Hyperparameter
Value
Hyperparameter
Value
Gradient clipping
5.0
Gradient clipping norm
5.0
Max train steps
10000
Max train steps
12000
Steps per epoch
100
Steps per epoch
500
Number of epochs
100
Number of epochs
24
Learning rate
5e-5
Learning rate
5e-5
Batch size
64
Batch size
32
Learning-rate decay
0.8
Learning-ratt decay
0.8
Learning-rate rate scheduler
ReduceLROnPlateeuu
Learning-rate scheduler
ReduceLROnPlateau
Scheduler patience
2
Scheduler patience
2
Early-stop count
7
Early-stop count
7
Optimizer
Adam [95]
Optimizer
Adam
Backbone
distilGPT-2
Backbone
distilGPT-2
Reward retrain period
1000 steps
B.2 Text Summarization
Implementation Details and Hyperparameters. The implementation of framework is based on the Hugging Face library [73]. We have provided some implementation details Section 4.2. The discussion is continued here.
Due to our limited computational resources, unless explicitly mentioned, we use the standard Tm-small model [89] for the LM. Similar to the prompt tasks, the token-level reward function is implemented also as T5-small model, with a two-layer projection-MLPP on top with sizes 2048 and 1. The LM my trained for standard 5 epochs. Apart from the hyperparameters discussed in the ablation study (Section 4.3 and Appendix A.2), most other hyperparameters as well as the training and evaluation procedure of our framework follow the standard setting of using T5 model for text summarization on the Hugging Face library. Additionally, we list the important hyperparameters for training our reward model in Table 15, and important hyperparameters for training our LM in Table 16. The same hyperparameters are used both the CNN/Daily and the XSum datasets.
We note that the ROUGE-L metric report is technically the rougeLsum metric from the Hugging Face interface and in the RL4LMs' codebase [61]. This one matches the result scales in prior work especially texts with newlines ("\n"), ). as reported in this GitHub issue.
Baselines. For the baseline methods' results Table we rerun the codebase RL4LMs [61] with a T5-small model as our method. We have carefully tuned the (supervised+) PPO/NLPO in RLALMs on several hyperparameters, such as learning_raas rate, kl. al_ coeff, k1 _l__ivi: target_ _ll, and so on. Furthermore, ran these baseline methods on the same random seeds as our method and we provide error bars. Since we use the T5-small model and the same random seeds for both our method and the baselines, reported results are therefore (more) fair comparisons.
25

---

Table 15: Hyperparameters training our reward
Table 16: Hyperparameters for training our LM
model in the semmaarmmriitiionn task.
in the eext-summarization task.
Hyperparameter
Value
Hyperparameter
Value
Gradient clipping norm
5.0
Gradient clipping norm Number of epochs
5.0
Number of epochs
5
Amount of training data
10% of training set
Learning rate
Se-5
Learning rate
5e-5
Batch size
32
Batch size
32
Optimizer
Adam [96]
Optimizer
Adam T5-small
Weight decay
0.0
Backbone
Backbone
T5-small
Reward retrain period
0.5 epoch
C A Naïve Numeric Example for the Average Aggregation
This section provides a naïve numeric comparison that the average aggregation in Section 2.1 will not automatically favor longer sequences, while the classical summation will.
Suppose we have = sequences r' and 7² for preference learning, respectively having length T' 5 and T 15. For simplicity, assume that all tokens in 5 and 72 are the same and all have reward 1, i.e., (s at) = 1, V k; L. The average sequence length C is then C' = (1/2) x (5 15)
10. For the first sequence r", the average-aggrogated sequence-level evaluation 2 (T) = (10/5) x
E (10/5) x 5 10. And for the second sequence 72, ea (7²) (10/15) E 1 (10/15) x 15 = 10. Therefore, no sequence will be automatically preferred based only on the length.
By contrast, when using the classical summation as the aggregation function, 71 will be evaluated as E 1 = 5 while 7² will be evaluated as Σi 1 = 15. So, indeed, the longer sequence 72 will be automatically preferred.
D Details on the Prompt Generation Task
Task Description. In discrete text-prompt generation [e.g., 10, 74), we input a discrete text-promp and an observation sequence to a large pre-trained downstream LM TDLM (UDLM a, to directly classify text o, without finetuning TDLM- Here, JDLM denotes the output of the large downstream LM MDLM on the observation text prompted by text a. We follow the classical prompt setting [e.g., 10, 75, 60] that solves the classification problem by an encoder-only downstream LM via token infilling. Classification is reduced to selecting tokens corresponding some predefined class labels, known as verbalizers, such as "happy" for positive and "sad" for negative. The set of verbalizers is denoted as C. As an example, to classify downstream LM TDLM. we input a template such as "[o] [a] [MASK]* " to TDLM, and select the most probable verbalizer token that fills into [MASK].
observation text by prompt using an encoder-only
our angut-ag setting, the generated prompt independent of the observation text o. During inference time, only the learned prompts are used and the LM πy is discarded. The initial input x to na is a dummy, and the target y is the class label the mask position. We also adopt the few-shot setting, where the training set consists of small number of samples per class. There is larger standard test set for evaluation. With a fixed length T, the goal is to find discrete text-prompts (ao. 1T- that have high test accuracy.
Setting.
Source of the Preference. For learning the token-level guidance, we simulate the sequence-level preference by the recently proposed stepwise metric Rstep in Deng et al. [60]. i.e., the higher the metric value the better prompt. This choice ensures fair comparison with RLPrompt [60] and avoids a potential overfitting that we train and evaluate the LM on the same evaluation metric "accuracy".
Given prompt a, observation text o, and the true class label y E C, Rstep measures the gap between the true class's probability and the highest probability in other classes. The gap is defined as
Gap,(a, 3) = TDLM(y a, o)
max TDLM(y' aa,o,,
y' y'EC.y'F #y
26

---

where Gap, y) > 0 when the prediction YDLM (a, o) for text is correct and < 0 otherwise.
Define the indicator for correct prediction for o, Corr,, Corr, 1|Gap, (a.3) y) 0}. The stepwise metric Rxep for prompt on observation text and true class label y is define as
Sttep (VDLM (a, (a, o), y) = l -l-C.
X2 Curt. x Gapo(a.v)
where \1 = 180 and 22 = 200. In the experiments (Section and Appendix A.2), we report test accuracy as in prior works.
LM Training. Since the prompt-generation task does not assume the availability of supervised data the ground-truth prompts, the LM wo is trained by the REINFORCE.style update in Section 2.2 to automatically discover highly-accurate prompts.
E
More Related Work
Prior works [e.g.. 6, 10, 85, 97] have shown that manual prompts can steer LMs to perform NLP tasks in the few/zero-shot setting. In general, prompts can be discrete, consisting of real token-strings; or can be continuous, where the prompts are entirely free word-embeddings that do not map to real tokens. Several works [e.g.. 98-102, 83] tune continuous soft prompts using gradient descent, which typically requires some expensive gradient information [78, 103]. In this work, we apply our framework to the task of input-agnostic its challenging setting. better human understandability of the learned prompts [104, 105], potential transferability across LMs [106, 76, 60], and more robustness in the low-data regime [99]. Recent works propose some new settings such as input-dependent prompt-tuning [88], which are potential further applications of our framework and are left for future work.
Prompt Generation.
prompt optimization due to
Text Summarization. Apart from using RL techniques discussed in Sections 3. prior works on text summarization [e.g.. 7, 89, 107-109] mainly focus on structural designs of the LMs and improvements on the source of the (pre-)training data, where the LMs are typically trained by vanilla MLE on the supervised data. In this paper. we apply our preferenced-grounded token-level guidance to this task by considering a weighted-MLE objective for LM training. The weights given by the learned reward function reflect some sequence-level preference among multiple candidate summaries. Our framework thus has the potential to learn and improve from lower-quality data, and generate summaries fulfilling more general evaluation metrics, such as human preference.
Weighted MLE in NLP. Though not very common techinique, the approach of weighted MLE has been adopted in prior NLP research. For example, RAML [110] samples outputs proportionally to its exponentiated scaled "reward" (negative edit/Hamming distance) using stratified sampling. GOLD [111] frames text generation as an offline RL problem with expert demos and learns from the demos by importance weighting. where training examples with higher probability under the model are weighted higher. Besides, Ghosh et al. [112] apply the weighted MLE technique to table-to-text generation and Junczys-Dowman Dowmunt et al. [113] apply this technique to grammatical error correction MLE in Section 2.2 adds to this research thread by emphasizing the important tokens in the supervised sequences and downweighting the unimportant tokens. This design may better utilize the LM capacity and the optimization budget. The efficacy of our reward-weit weighted MLE is experimentally verified in Section 4.2.
for machine translation. Our token-level reward-weighte
Align LMs with Preference. Similar to our paper, prior works on aligning LMs with preference typically focus on adjusting the pretrained LMs, where preference comes from human feedback or from some automatic metrics. classical strategy is to add external filters on top of the pretrained LMs for the generated text sequences or for the training sequences [e.g.. 19]. where the LMs are trained using MLE on abundant supervised data. Another classical approach finetunes LMs using supervised learning (vanilla MLE) on some curated/improved datasets [20-22], or on massive highly- curated collections of tasks phrased as instructions for supervised finetuning the LMs [1 14-1.. Apart from supervised learning, reinforcement learning techniques have also been applied to learn from human feedback (RLHF). Similar to the discussion in Section 3, these works typically learn a sequence-level classifier that predicts human (pairwise) preferences and during LM training add a general- purpose KI penalty that is less-targeted to the specific LM task and feedback (preference, metric scores, etc.) [e.g., 14, 23-25], such as token-level KI. penalty towards the initial LM prior to training.
27

---

Alternatiely, the divergence of the LMs from target distribution can also be used as the finetuming objectives. This line of research [e.g.. 118-120] formalizes controlled text generation as a constraint satisfaction problem over LM's probability distribution, with an additional divergence-minimizze objective that the L.Ms should have a minimal KI.- or f-divergence from the original pretrained model. These approaches, however, require explicit functional specification on the constraints or on the human preference, rather a more vague form of (binary) comparison between LM samples. For example, Go et al. [120] consider human preference the generated text-sequence satisfies the preference. Apart from this more demanding requirement, these approaches further require special methods to sample from the resulting LM.
probability distribution measuring how well
To sum up. prior works on aligning LMs with preference mostly focus on an ungrounded sequence- level guidance, which can suffer from the dela--fee issue in LM training, as discussed in Sections and 3. By contrast, our preferencegrouudde task-specific token-level guidance on LM training, and can potentially improve on the vanilla MIL,, especially when the quality of the supervised data cannot be guaranteed. We experimentally validate this intuition in Section 4 and Appendix A.2.
perspective can provide stable, data-driven,
Apart from fine-tuning the pretrained LMs, Korbak et al. [121] recently apply preference alignment to the pre-training stage of the LMs. As with prior works, the sparse sequence-level evaluation (without KL penalty/stabilizee) directly used, learn token-level value function, to condition the LM generation on, or for reward- weighted regression objective. The pre-traie pre-training stage in Korbak al. [121] is a potential further application of our framework since we make no assumption on the zero-shot ability of the initialized L.ss, as discussed in Sections 2.2 and 4.3.
We also notice that recent robotics paper [122] proposes to learn a weighted-sum aggregation together with the per-step reward, to form the sequence-level evaluation in learning the reward function, based on pairwise preference over two trajectories of equal length. Compared with this recent work, our aggregation functions in Section 2.1 do not require additional modeling and training, and therefore can be more efficient and more stable for the reward-function learning. Additionally, we do not assume that trajectory lengths are equal, as this may be infeasible for LM tasks such as text summarization. Furthermore, our framework allows utilizing the preference among more than two trajectories, rather than the classical pairwise preference. In this particular aspect, our framework can be more general than this recent work of Kim et al. [122].
F A Discussion on Applying RL Methods to LM Tasks
F1
LM Generation as a Token-level MDP
In most LM generation tasks, there is dataset D = {(x', y') of N supervised examples, where is the input to the LM that can be a dummy, and E 3/ is the target text sequence. Viewing the LM as teken-level RL policy, LM generation can be formulated as a sequential decision-making problem. specified by the Markov Decision Process (MDP) .M- = (S, A, P, R. 7; Po) [123]. Specifically, S the state space, where the state timestep L, 8t. consists of the LM input x and the previously generated tokens azt = (oo..... 1t- >> 0, i.e. so = x and Vt > 0, st (z, aet). A is the action space, which is the vocabulary V. and an action a at timestep 2 is a token from V. P(st, a.) S x A + S is the transition function that deterministically appends the newly sampled RR
7:P0
token to the end of the current state, i.e., Vt 2 0, 8t+1 (st, az) (x. agt). R(sT,y) S is the environmental reward (task-specific evaluation metric) that depends on the final state sT of the LM- -e-gereration trajectory and the target sequence y. Here T the ending time of the trajectory, i.e., the length of the full generated text sequence; and sT (æx, ao..... aT-- is the final state of the generation trajectory consisting of the LM input x and the full generated text sequence (wo,... ... -T- 1). [0. 1] is the discount factor. And (o( S + [0. 1] is the distribution of the initial input x.
We denote the LM as Ne (az st), parametrized by 0. At each timestep t, Te(a; st) generates the next token at given the current state s; = (z, a<e). The ultimate goal of policy learning (LM training) is to maximize the expected environmental reward R, which can be expressed as
maxg B( E B nola. (x) [R(sT= (z, a), v)]
where (x. 2) is drawn from the corresponding sampling distribution.
28

---

F.2 Delayed Feedback in RL-based LM Training
As discussed in Appendix F.1, the environmental reward R ((Sr.3 2) is only defined on the full generated text sequence a. The token-level MDP formulation of LM generation thus meets the problem of sparse reward-signal or the delayed feedback issue discussed in Section 1. Hereafter, we will use "sparse reward (signal)" and "delayed feedback" interchangeably depending on the context, as they are used synonymously in the RI. literature.
Specifically, prior works [e.g., 31, 60, 32] often manually interpolate the intermediate rewards by some non-informatiie values such
-1, 4t
fror 1, << R1 u) = (R(ST.)), v), t=t
It is clear that the reward signal is sparse. In other words, the feedback to intermediate actions/tokens delayed until the full text-sequence has been generated.
We note that this ppanse-rwardddellay--ffeeddtt actor-critic or Q-learning methods in RI. With only sparse reward-signals, it can be difficult to estimate the token-level value functions in these RL methods.
problem will not be addressed by the standard
Specifically, the standard Monte Carlo estimate of the value functions is known to have high variance due to the large sampling space [123]. This problem is even severe in the LM tasks where there are exponentially many text sequences that can follow partial sequence.
foeeddaac problem can also hurt the bootstrapping-sty. method for learning the value functions, since the standard value-function learning can suffer from "the unstable per-step bootstrapping-ste can subsequently harm the LM training since many actor-critic or Q-learning methods rely heavily on how accurately the learned value function(s) can assess the quality of intermediate text sequences [31, 123, 124].
Further, as discussed Guo et al. [31], the
training with sparse reward signals." " This
F.3 Sparse Reward with KL Penalty
With the sears--rwward/dllyeeddd-e feedbback issue Appendix F.2, prior works typically add a token- level KL-penalty to the sparse sequence-level environmental rewards Eq. (7). For simplicity, assume that in Eq. (7) the intermediate rewards are interpolated by O. The KL-stabilized reward signal R( 3) is
R(a,, .c. v) = T----KL KL(TT(aa (8)) (o(r st)),
<TT-1 T-1-1
(8)
(R(sr (L( st) no(au) sc)),
where e is a [31, 60], the initial LM prior to training [23, 61], the supervised-ine-tune model [62, 63, 12, 14], or the base momentum model [65]. For concrete example, see Line 224-235 of the popular trix package's implementation.
and is some prior distribution, such as the uniform distribution
With this KL-stabilized reward signal R(st, az, v), the action-value function for the policy/LM ma is
((s6))1))) v) = 15 (ly
"(sss, ar, v) sc
(9)
T-1- T-1 p (KIII(TT(aaa (se) Ro(ay [sr)) 1 st
= =B
is clear from Eq. (9) that the environmental reward R(ST, 8) is multiplied by a factor exponentially decayed with respect to the length of the remaining horizon T - 1 Without the KL penalty, the action-value Q(5t, at y) could be tiny when is small, i.e., at the beginning of the text-sequence generation. This could make it hard to accurately model and learn the action values, echoing the previously-stated harm of the rewds--rwarr/dellyeedd.e feedback problem mentioned by Guo et al. [31]
Recall that the standard actor-critic and Q-learning methods in RL use the action-value function Qst: Gt, 3) as the token-level guidance (per-step critic) for policy/L.M training. Due to the expo- nentially decaying factor x T-1-t, when the discount factor y in Eq. (9) is not sufficiently large, this
29

---

token-level guidance Q(s². Qz, y) in RL-based LM training mainly reflects the (discounted) sum of future KL-penalty, rather than the actual goal of LM training the environmental reward R(TT. y). This phenomenon can be more evident at the beginning of the text-sequence generation, i.e, when the length of the remaining horizon 7 1 - is long. On the other hand, learning the action-vall value function (st, Gt: y) under a large discount factor is known to be challenging [123], since the highly varying (late) future can significantly affect the current action value (st,, oz. y). The selection the discount factor y, therefore, becomes a tradeoff and challenge. Note that (sTT, y) here is generic and can represent automatic evaluation metrics or (human) preference, and that the beginning text
generation affect all subsequent token selections. Intuitively, using Eq. (9) the token-level guidance for policy/LMM training can thus be less successful in the concrete LM task, especially when generating longer sequences, as we verified in Appendix A.2.
In the experiments (Section 4 and Appendix A.2), we compare our preference-groudiing approach with RL- baselines that estimate standard value function similar to Eq. (9) from sparse environmental reward with KI. penalty, such as the RIPrroppt method [60] and the (supervised+) PPO/NLPO methods in RL4LMs [61]. We leave as future work the potential combination of our preference-grounded guidance with actor-critic and Q-learning methods in RL-based LM training.
G Further Discussion on the Guidance Re-estimation Scheme
As discussed in Section 2.2, in this paper, we deal with the most general setting where the LM training directly starts from a raw pre-trained LM, rather than an initial LM that has been fine-tuned via supervised learning on the desired dataset, such in Stiennon et al. [12]. We also make no assumptions about the zero-shot ability of the raw pre-trained LMM We choose this setting because is more general and naturally fits into the task of text-prompi generation, where supervised datasets of good prompts are not available and the initial LM cannot generate good prompts.
As discussed before, under this general setting, the LM тa can evolve from less- pess-preferre distribution one, over the training process. Since our reward function ra is trained by text to sampled from To, there is distribution shift between the sequences used to train r during sequences
ceraardffnntionn learning. and the sequences evaluated by ra during LM training, especially after my has been sufficiently improved. To keep r. as accurate guidance for LM training, natural idea is to refine Te periodically on the text generations from the latest LM, leading to our reward-function retraining scheme.
We emphasize that the reward-fuuccton retraining scheme does not give our method an unfair advantage over the baseline methods. In particular, RLPrompt [60] and RL4LMs' methods [61] retrain their falue-functions in every optimization step. and thus, they query the environmental reward in every optimization step. Specifically, in Algorithm I of the RL4LMs paper, the penalized reward R, is calculated in each optimization step, whose calculation requires the true environmental reward (Eq. (1) of the RLALMS paper). Besides, in the codebase of Proopp, this environmental interaction is implemented in this line, which is queried in every optimization step, seen in this line. In the notion Reinforcement Learning from Human Feedback (RLHF), this every-step interaction similar asking humans to score the LM generations in every training step, which can be infcasible. By contrast, in our paper, we reduce the frequency of these environmental interactions by retraining the guidance model only periodically and only during the first half of the LM- LM-training process.
Though the motivation of this (Section 2.2), we notice that some prior RLHF works do implement similar ideas. For example, Page 2 of Ziegler et al. [23] mentions that "..., we continue to collect additional data and retrain our reward model as the policy improves (online data collection)." Page 2 of Stiennon et al. [12] mentions that "We can then gather more human data using samples from the resulting policy, and repeat the process." Page 5 of Menick et al. [25] and Page 20 of Bai et al. [24] also have similar discussions. Based on these, our reward-funcee function retraining scheme is both mollvvotivated and practical, even with human rankings in RLHF.
retraining scheme comes from model-based RL
H Potential Negative Societal Impacts
Since our framework can ground the sequence-level preference into token-level guidance for LM training and can be not tied to specific preference source, it is possible that this framework may be
30

---

used to train ill- in-inteendd LMs by grounding some malicious unethical preferences. This potential negative impact may be mitigated by closer monitoring the datasets on which our framework operates.
I Limitations
Since token-level guidance is learned by grounding sequence-level preference, potential failure case of our framework will when the preference orderings are very noisy. this situation, the learned guidance may not be meaningful and hence could even deteriorate the subsequent utilization of it in LM training.
Even though we have shown in Section 4.3 that it can be beneficial to use more than two sequences learn the token-level guidance, can be practically challenging to obtain a high-quality ranking among many candidate text sequences, e.g., when the number of sequences is more than seven.
Besides, the rewato-function retraining scheme may incur additional computational complexity, compared with training the reward function only once and fixing it throughout the LM-training process.
Computational Resources
The experiments are conducted on NVIDIA GeForce RTX 3090 and NVIDIA A 100 GPUs. Depending on the specific task and setting, several models could be trained concurrently on single GPU.
31


Reviews:
[{"review_id": "2X9Mfv9pMx", "reviewer": "Reviewer_XL7b", "summary": "The paper presents a new approach to training language models that address the mismatch between coarse-grained sequence-level preferences and fine-grained token-level rewards. With more fine-grained rewards, the proposed framework reduces the reliance on supervised data. Specifically, given the preference of a set of sequences, it first trains a reward function such that the aggregated rewards over tokens can satisfy the preference. Then, it applies REINFORCE to update the language model. The framework is evaluated on two text generation tasks: generating text prompts for few-shot text classification and text summarization. Compared with baselines, the proposed method improves the performance on both tasks.", "questions": "1. Are there any generated samples for the discrete prompts for zero-shot classification? Qualitative analysis may make the method more convincing.\n2. How does the choice of reward function impact performance? Does using different pre-trained models as reward functions significantly impact the performance?\n3. Why the summarization results are significantly below SOTA results? The paper mentions it uses the full training set for finetuning the model, so I expect the performance should at least not be far from SOTA results.\n4. How to obtain the preference for discrete prompts of zero-shot classification?", "limitations": "Please refer to weaknesses and questions.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Revisiting the mismatch between sequence-level and token-level feedback in fine-tuning language models (LLMs) is of particular interest to me. In scenarios where supervised data for a target domain downstream task is scarce, and fine-tuning LLMs is computationally expensive, exploring data-efficient methods becomes important. The proposed method is straightforward and reasonable to me.", "weaknesses": "1. Using discrete-prompt generation to evaluate the proposed method is not convincing. The chosen task itself lacks meaningfulness, and a better way to achieve good performance may be self-training as introduced in [1]. I can understand the goal of the experiment is to evaluate the quality of the generated sequence, but there are other more reasonable generation tasks such as personalized chatbot.\n2. The performance of the proposed method of text summarization falls significantly short of the SOTA, which leaves me unconvinced by the results. For instance, the ROUGE-L score for the current SOTA on CNN/DailyMail has surpassed 44, and for XSum it stands at 40.4 [2]. In contrast, the proposed method achieves scores of 38.17 and 26.33 respectively. For CNN/DailyMail, the results are only slightly better than the trivial lead-3 baseline. Considering that the proposed method is complex, I expect more improvement. I can understand the low performance may be partially due to model size. However, without using the model size of a state-of-the-art model, it’s hard to evaluate the effectiveness of the proposed method because It’s easier to boost the performance on a small model. In [3], they challenge the effectiveness of RL and argue that using a few human-curated prompts and responses is more effective for finetuning LLMs. \n3. Using ROUGE as the only evaluation metric for summarization is not convincing. At least human evaluation or language model scores such as BERT-score/BART-score should be used. Also, is it possible to obtain preference via language model scores?\n4. The proposed method lacks novelty. Previous studies have extensively discussed the mismatch between token-level and sequence-level rewards. Using a token-level reward function to estimate the reward for each token is not a new idea. Several related works are missing [4][5].  \n\n[1]ExploitingClozeQuestionsforFewShotTextClassificationandNatural LanguageInference, Schick et. al.  \n[2]BRIO:Bringing Order to Abstractive Summarization, Liu et. al.  \n[3]LIMA: Less Is More for Alignment, Zhou et. al.  \n[4]SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient, Yu et. al, 2016.  \n[5]Unsupervised Text Style Transfer using Language Models as Discriminators, Yang et. al.  \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate the reviewer for the careful comments.\nWe would like to draw your attention to **General Response** for additional results and common responses. The other questions are answered in detail below.\n\n> **Q1.** About the meaningfulness of discrete-prompt generation task. May test on other tasks.\n\n**A.** We respectfully disagree your comment that *“the chosen task (discrete-prompt generation) itself lacks meaningfulness.”* We believe that while the self-training in [1] may give good results, it doesn’t preclude other methods to this problem, such as the recent RL-style method [6] and the recent baselines therein.\n\nWe kindly argue that this task is meaningful at least in better human understandability of the generated prompts, compared to soft prompts. We discuss related work in Appendix E (L857-864).\n\nIn this paper, we use discrete-prompt generation to test the applicability of our method to the challenging setting of no supervised examples, i.e.,  no ground-truth prompts.\nAs discussed in L191-199, this setting makes it infeasible to apply many prior works on learning token-level guidance for LM training, since they typically require abundant supervised examples.\n\nIn Section 4.1, we show that our preference-grounded token-level guidance is not only applicable to this task, but can be more effective than many strong recent methods, such as RLPrompt [6].\n\nAs discussed in L342, we agree with you that dialog systems can be a further application of our method. But we believe that discrete-prompt generation is an equally reasonable and important task.\n\n[6] Deng, Mingkai, et al. \"Rlprompt: Optimizing discrete text prompts with reinforcement learning.\" arXiv preprint arXiv:2205.12548 (2022).\n\n> **Q2.** LIMA [3] challenges the effectiveness of RL for finetuning LLMs.\n\n**A.** We want to clarify that our method is not a RL method, though some related works and baselines are.\n\nWe thank the reviewer for raising LIMA [3]. However, LIMA was submitted to arXiv on 18 May 2023, which is after the NeurIPS submission deadline (17 May 2023). Thus, a discussion on LIMA is out of the scope.\n\n> **Q3.** Is it possible to obtain preference via language model scores?\n\n**A.** Yes, it is possible, as long as LM scores can desirably reflect sequence quality in the specific task.\nAs discussed in L101-102, we make no assumption on the preference source. \n\nIn our summarization task, we obtain preference by Meteor, to have a fair comparison with the benchmarks in Table 17 of RL4LMs [7] and to avoid overfitting the ROUGE evaluation metrics (L258-259). \n\n[7] Ramamurthy, Rajkumar, et al. \"Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization.\" arXiv preprint arXiv:2210.01241 (2022).\n\n> **Q4.** About lacking novelty and prior works on using token-level rewards, such as [4, 5].\n\n**A.** We thank the reviewer for suggesting [4, 5]. We will certainly include them in our revised draft.\n\nWe agree that some prior works have attempted to learn token-level guidance for LM training. But as we discussed in Section 3 (L191-205), those prior works typically require abundant expert data, making them infeasible for the few/zero-shot settings, such as discrete-prompt generation, where there are no ground-truth prompts.\n\nThe suggested related works SeqGAN [4] and Unsupervised Text Style Transfer [5] both fall into this category. Specifically, [4] requires a set of “real sequence data” to pre-train the generator by MLE at the beginning, and [5] needs real target-domain sentences to train LM discriminators. Thus, both [4, 5] are infeasible to our few/zero-shot prompt task.\nFurther, similar to our related work “Lin et al. [64]” (L195), the intermediate guidance in [4] comes from Monte Carlo search, which can have high variance and be compute-demanding. \nIn [5], the token-level feedback is essentially the token-level probability from the target-domain LM, which can be less general/flexible than our preference-grounded guidance.\nApart from these, both [4, 5] do not consider the preference relation among multiple generated sequences.\n\nBy contrast, our method grounds the sequence-level preference into task-specific token-level guidance for LM training; and is suitable for both standard LM tasks and the low-data regime.\n\n> **Q5.** Samples of the generated discrete prompts? \n\n**A.** Some samples of the generated discrete prompt are in Table 3 (Appendix A.1). A brief discussion is on L250-254.\n\n> **Q6.** How does the choice of reward function impact performance? \n\n**A.** In Appendix A.2 (c) (L740-757), we simulate the preference on the summarization task by two other automatic metrics “Rouge-avg” and “Rouge-avg2”, rather than the Meteor in Section 4.2. We show that the efficacy of our method is not tied to using Meteor as the preference source.\n \nWe appreciate it if you could further elaborate this question: *“Does using different pre-trained models as reward functions significantly impact the performance?”* Overall, we think that the answer depends on how different pre-trained models are used as reward functions, and how the performance is measured.\n\nWe note that in summarization, our preference is from Meteor or Rouge-style scores. Both we and the results in RL4LMs [7] do not use pre-trained models as reward functions. Thus, a discussion of that may be beyond scope. \nIn the prompt task, we obtain the preference by the stepwise metric in RLPrompt [6] (L226-227), to ensure a fair comparison with RLPrompt and the baselines in its paper. We are unsure how different pre-trained models can be used or if changing this preference-obtaining metric will still make fair comparisons.\n\n> **Q7.** How do you obtain the preference for discrete prompts?\n\n**A.** As discussed in L224-225, it’s obtained by the stepwise metric in RLPrompt [6], i.e., the higher the metric value the better. Details of this metric are in our Appendix D (L842-851).\n"}, {"review_id": "l5EyBG93tS", "reviewer": "Reviewer_wRKD", "summary": "This paper proposed break the pairwise sequence preference into token-level guidance signal by iterating between learning a token-level reward from sequence level preference and improving LM with the learned token level guidance. Experiments are conducted on two language generation tasks and competitive results are reported.", "questions": "NA", "limitations": "NA", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The paper propose to address a problem of how to effectively ground sequence-level preference into LM finetuning.\n- studies different aggregation function to break sequence level preference into token level reward function\n- the setup with supervised data to weight token level MLE by token level reward is interesting", "weaknesses": "- in general the paper is difficult to follow, e.g. Figure 1 is not self explanatory, experimental setups and datasets could be use more details.\n\n- a core motivation/hypothesis of the paper between sequence and token level losses is questionable. E.g. abstract mentions \"a granularity mismatch between the preference and the LM training losses\". It is wrong to categorize LM loss as token-level loss, because it is MLE over target sequence on both token level and sequence level. there are many unsupported assumptions around the mismatch of sequence and token level losses in the introduction section.\n\n- in PPO, even though the reward function is sequence level, there is a token-level value network. Discussion and ablation of how that is trained is very relevant to choosing the aggregation of your token level reward model.\n\n- the dataset to test your method is not proper. \"simulate the sequence-level preference by the classical Meteor score and report the standard ROUGE scores\" is questionable because it is not real sequence level feedback. On the other hand, there are datasets readily available, e.g. \"Learning to summarize from human feedback\" which contains real human preference data on summarization tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for raising several important questions. Please would you first check our **General Response**, which should address your concern on testing against the RLHF summarization paper. We answer your remaining questions in details below.\n\n> **Q1.** The clarity of our paper. \n\n**A.**\nWe thank the reviewer for raising this issue. We will carefully revise our paper and include more details into Figure 1 in the next version of our manuscript. Meanwhile, we would like to refer to **Reviewer gmdH**’s Summary section for a good overview of our paper.\n\nWe provided details on experimental setups and datasets in Sections 4.1 and 4.2. Due to the page limit, we deferred additional experimental details to Appendix B and additional details about the prompt task to Appendix D.\n\nFinally, we notice that **Reviewer xDED** praised our paper as *“well written … and the experimental setup is well documented”*; while **Reviewer gmdH** thinks that our Figure 1 *”is a good illustration”*. Therefore, we will highly appreciate it if you can point out explicitly the confusing parts of our paper.\n\n> **Q2.** About the validity of our core motivation and categorizing LM loss as a token-level loss.\n\n**A.** We respectfully disagree with the reviewer that the core motivation of our paper is questionable and that it is wrong to categorize LM loss as token-level loss.\n\nAs discussed in Line 17, we consider LM (cross-entropy) loss as token-level because each token position has a corresponding term in the overall training loss.\nBy contrast, the preference/feedback is sequence-level because there is only one feedback for the entire text-sequence, rather than densely at each intermediate timestep.\n\nIntuitively, the granularity mismatch comes from the fact that the LM needs to decide each token, while the preference/feedback is available only after the entire sequence has been generated and is only at the sequence level. \n\nAs discussed in Section 3 (Line 191-199), some prior studies have attempted similar/related problems, but are under more restricted/ideal settings.\n\nTherefore, we believe that our core motivation is valid and sound, which is also supported by the other reviewers.\n\n> **Q3.**  In the introduction section, there are many unsupported assumptions on the mismatch of sequence- and token-level losses.\n\n**A.** We will be deeply appreciated if you can explicitly point out the unsupported assumptions in our introduction section, apart from the above **Q2** that we have clarified.\n\n\n> **Q4.** Despite the sequence-level feedback, there is a token-level value network in PPO. And how that is trained is very relevant to choosing the aggregation of your token-level reward model. \n\n**A.** We agree with the reviewer that there is a token-level value network in PPO. However, the main problem lies in learning this value function. \n\nSpecifically, the mismatch between sequence-level feedback and token-level LM training/generation leads to the problem of delayed feedback (sparse reward-signal) when applying RL methods to LM tasks. It is known in the literature [1, 2] that with only sparse rewards, it can be difficult to estimate the token-level value functions in RL methods, including PPO. In particular, [1] points out that the standard value-function learning can suffer from *“the unstable per-step bootstrapping-style training with sparse reward signals.”*\n\nIn Appendix F.2, we provide a detailed discussion on this delayed feedback problem in RL-based LM training, which we believe applies to PPO. In Section 4.2 and Table B in the added PDF, we show that our method outperforms the benchmarking results of (Supervised+) PPO cited from RL4LMs [3].\n\nWe are confused about the second part of your comment. We are unaware of other ways to train the value network in PPO, apart from the standard one implemented in our baseline RL4LMs [3]. We are also confused about why that will be relevant to choosing the aggregation of our token-level reward model. As a gentle reminder, in our paper, the aggregation function is used to train the token-level reward function $r_\\phi$ (Line 104-109).\n\nWe will highly appreciate it if you can provide more details on your comment so that we can respond better, rather than only taking an educated guess.\n\n[1] Guo, Han, et al. \"Efficient (soft) q-learning for text generation with limited good data.\" arXiv preprint arXiv:2106.07704 (2021).\n\n[2] Sutton, Richard S., and Andrew G. Barto. Reinforcement learning: An introduction. MIT press, 2018.\n\n[3] Ramamurthy, Rajkumar, et al. \"Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization.\" arXiv preprint arXiv:2210.01241 (2022).\n\n> **Q5.** Simulating the sequence-level preference by the Meteor score is questionable because it is not real sequence-level feedback.\n\n**A.** We respectfully disagree with the reviewer. Since the Meteor metric scores the entire generated text-sequence, it provides a sequence-level feedback on LM generation.\nTherefore, we believe that it is valid to use Meteor to simulate the sequence-level preference, i.e., the higher score the better generated sequence, as discussed in Line 101-103. Hence, our summarization experiments on datasets CNN/DM and XSum are proper.\n\nIn general, we believe that automatic evaluation metrics can be a valid and rich source of sequence-level preference, apart from real humans. \n"}, {"review_id": "tAuZmGDNQm", "reviewer": "Reviewer_xDED", "summary": "The paper proposes to solve the issue of granularity match in preference-based tuning of LMs (RLHF for e.g.,), that is task-based preference is defined at the sequence level (via pairwise preference learning) while the reward model training and policy optimization is done at the token level. The paper proposes an extra step to ground the token-level reward into the sequence-level preference. Evaluation is done on two tasks (prompt generation and summarization) and some improvement is shown compared standard RL-based method such as PPO and NLP as well as vanilla supervised models \n\n", "questions": "- See last two points in Weaknesses\n- In the ablation in section 4.3,  (b), How is this done? you just replace $f(\\lbrace r_\\phi(s_t^k, a_t^k)\\rbrace)$  with $\\mathcal{R(s^k_T)}$? \n", "limitations": "I think the authors addressed some of the limitations of their work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well written and explains the proposed method well and the experimental setup is well documented.\n- The studied problem is interesting and relevant especially since RLHF methods are becoming more common nowadays. \n- Evaluation is done against strong baselines. \n- The proposed method seems to bring some improvements on the two studied tasks ", "weaknesses": "- **Limited tasks**: Evaluation is only two tasks and it's not clear why these two particular tasks were selected. I imagine other tasks should be used, where preference-based learning is relevant. Such tasks include toxicity avoidance and controllable/constrained generation.\n- **No human evaluation**: No human evaluation is done on summarization: We know that RL methods are good at hacking metrics and I would expect human evaluation on the summarization task to support the improvements in ROUGE. \n- **Some choices are not justified**: For example, in Algorithm 2, why is the reward model retrained only for the first half of the iterations? Why not for the entire LM training? Why is a max-entropy gradient added to REINFORCE in Eq 5? Did you add the same for the baselines? \n- **Not clear where improvement compared to the baselines comes from**: For example in Algorithm 2, the reward model is trained alternatively with the  LM. Are all baselines trained the same way? If not, then there is no way to know if improvement comes from this method of training or from the sequence-level grounding.  ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and careful review. We first want to bring your attention to our **General Response** for our human evaluation results on CNN/DM summarization. \nBelow are our detailed responses to your other concerns. \n\n\n> **Q1.** Evaluation is only on two tasks. It's not clear why these two tasks were chosen. Some other tasks can be used.\n\n**A.** Due to the page limit and our limited computational resources, in this paper, we test our method on two carefully-chosen, distinct, and representative LM tasks. \n\nSpecifically, the prompt-generation task represents few/zero-shot learning since there are no ground-truth prompts. Text summarization represents a standard LM task, where there is a set of supervised samples and the generated sequences are of variable-length and are relatively long. Furthermore, on these two tasks, baseline methods/results within our computational budget are easily accessible.\n\nMoreover, we conduct extensive ablation studies in Section 4.3 and Appendix A.2 to comprehensively demonstrate the efficacy and benefit of our method. Therefore, we believe that our experiments are sufficient to validate our method.\n\nWe agree with the reviewer that there are many exciting further applications of our method, such as toxicity avoidance and controllable generation that you suggest. We believe that the (potentially) wide applicability of our method is a merit and we will certainly test on these tasks in our future work.\n\n\n> **Q2.** Why is the reward retraining only conducted in the first half of the iterations? Why not over the entire LM-training process? \n\n**A.** We retrain the reward model only during the first half of the iterations in order to save computational cost. Our preliminary study suggests that retraining the model throughout the entire LM-training process doesn't yield substantial performance gains, making it less computationally worthwhile.\n\n\n> **Q3.** Why do you add a max-entropy gradient to the REINFORCE-style update Eq. (5)? Did you add the same for the baselines?\n\n\n**A.** As discussed in Line 161-163, since we want multiple generated texts in typical LM tasks, we add the max-entropy gradient so as to capture multiple good behavior-modes (good generated texts). \n\nThe REINFORCE-style update Eq. (5) is adopted in the prompt-generation task. The baseline method RLPrompt [1] is based on soft Q-learning [2], which is a maximum-entropy Q-learning method for text generation. By its design, it naturally contains a max-entropy gradient. We kindly note that the max-entropy gradient may not be applicable to other baselines due to their specific nature/designs. Their results are directly cited from the literature. \n\nTherefore, we believe that the comparisons between our method and the baselines are fair. \n\n[1] Deng, Mingkai, et al. \"Rlprompt: Optimizing discrete text prompts with reinforcement learning.\" arXiv preprint arXiv:2205.12548 (2022).\n\n[2] Guo, Han, et al. \"Efficient (soft) q-learning for text generation with limited good data.\" arXiv preprint arXiv:2106.07704 (2021).\n\n\n> **Q4.** Not clear if improvement comes from the reward-retraining scheme or from the main contribution: the preference-grounded token-level guidance.\n\n**A.** We respectfully refer the reviewer to our ablation study in Section 4.3 (b), where we remove the reward-function retraining scheme. It is clear that ​​without the reward-retraining scheme, our method still performs competitively against the strong baselines, which confirms the benefit of our preference-grounded guidance and is acknowledged by **Reviewer gmdH**.\n\nAs discussed in Appendix G (Line 1007-1017), the reward-retraining scheme does not give our method an unfair advantage over the baseline methods. In particular, the baselines RLPrompt [1] and RL4LMs’ (supervised+) PPO/NLPO [3] retrain their value-functions in every optimization step, which can be more demanding than our method. \n\nTherefore, we believe that our improvement over the baselines should come from our main algorithmic contribution: the preference-grounded token-level guidance, rather than an unfair training scheme.\n\n\n[3] Ramamurthy, Rajkumar, et al. \"Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization.\" arXiv preprint arXiv:2210.01241 (2022).\n\n\n> **Q5.** How is the ablation in Section 4.3 (b) done?\n\n**A.** In our ablation study in Section 4.3 (b), we remove the reward-function retraining scheme. This can be seen from Algorithm 2 as removing the “if” block containing “Re-train $r_\\phi$ by Algo. 1 without re-initialization.” In other words, we train the reward function $r_\\phi$ only at the beginning of the LM-training process and fix it thereafter (never retrain it).\n"}, {"review_id": "6HTcnHgKya", "reviewer": "Reviewer_gmdH", "summary": "To fine-tune an LM, the paper proposes “token-level guidance” by leveraging sequence-level preference. The algorithm alternates between two stages: (1) learning token-level “guidance” (aka reward function) and (2) fine-tuning LM using the “guidance”/reward.\n\nTo aggregate token-level rewards, the authors propose aggregation functions that are different from the classical summation: average, soft maximum, and soft minimum. \n\nFigure 1 is a good illustration of the learning algorithm. It’s similar to RLHF, except the preference is based on multiple generated sequences, the reward is token-level, and the aggregation function can be different.\n\nTwo “minimalist” LM training objectives are proposed:\nWhen there’s no supervised data, the authors use a REINFORCE-style update (with a max-entropy regularizer).\nWhen we have enough data, the authors use a reward-weighted MLE objective – the weight depends the importance of the token, which is essentially the normalized token-level reward. \n\nExperiments are done on prompt generation (generating discrete prompts so that the accuracy of the corresponding task is good) and text summarization. The performance is competitive with respect to baselines shown in Table 1 and Table 2. \n\n\n\n", "questions": "A few questions are in the \"weaknesses\" section. ", "limitations": "I don't see significant discussion on potential limitations. Please let me know if I missed those paragraphs.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "\nIt’s great that the authors have done ablation that removes reward retraining, as shown in Section 4.3 (b). Good to see that the performance, after removing retraining, is still competitive with respect to baselines. \n\nThe token-level reward idea is worth pursuing, and I’m glad to see work in this direction. The experimental results should benefit the community. \n\nIn addition, I agree with the intuition that summation is not necessarily the only / optimal approach for aggregating different token-level rewards, and it’s great to see that the authors have attempted other aggregation functions.\n\n\n", "weaknesses": "\nReward function retraining seems expensive. Is there an analysis on the compute cost? \n\nI just want to make sure that the authors have ensured that their ROUGE computation is fair.\nBART paper and the “Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization” paper have different ROUGE scores for lead-3 baseline, which is a bit weird. BART’s lead-3 result is higher. I just want to make sure in Figure 2, for example, the ROUGE scores are all comparable.\n\nRelated: Is there a reason why Table 2 results are run using T5-small instead of a larger model? Would the trend still hold when T5 gets a lot larger?\n\nMy understanding is that the reward is Meteor in summarization experiments. Although the summaries are evaluated by ROUGE, what are the Meteor scores (the actual rewards being optimized) on dev/test set? This detail seems to be missing, but apologies if I missed it. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful reading of our paper and insightful comments.\nWe appreciate it if you can also consider our additional results and clarifications in the **General Response**.\nBelow are our detailed responses to your questions.\n\n> **Q1.** The reward-retraining scheme seems expensive.\n\n**A.** We thank the reviewer for raising this important concern. \n\nIn our algorithmic design, we reduce the computational cost of the reward-retraining scheme by  only periodically retraining our reward model during the first half of the LM-training process, rather than over the entire training process.\n\nBy contrast, as discussed in Appendix G (Line 1007-1017), the baselines RLPrompt [1] and RL4LMs’ (supervised+) PPO/NLPO [2] retrain their value-functions in every optimization step, throughout the whole LM-training process. This can be more demanding and expensive than our method.\n\nTherefore, compared with the baselines, we believe that our method is efficient and thrifty.\n\n[1] Deng, Mingkai, et al. \"Rlprompt: Optimizing discrete text prompts with reinforcement learning.\" arXiv preprint arXiv:2205.12548 (2022).\n\n[2] Ramamurthy, Rajkumar, et al. \"Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization.\" arXiv preprint arXiv:2210.01241 (2022).\n\n> **Q2.** Make sure that the ROUGE computation is fair.\n\n\n**A.** We thank the reviewers for this careful reminder.\n\nWe ensure that our ROUGE computation is fair by following the codebase of RL4LMs [2], to have a fair comparison with our baselines. \n\n\n> **Q3.** Why results in Table 2 are run using T5-small instead of a larger model? Would the performance gain still hold when T5 gets larger?\n\n**A.** As discussed in Line 263, results in Table 2 are run using the standard T5-small model because of our limited computing resources. Furthermore, using T5-small facilitates our comprehensive ablation study in Section 4.3 and Appendix A.2. \n\nWe verify the performance of our method under a larger LM by scaling up from T5-small to T5-base in Section 4.2 (Line 285-290), where the model is enlarged by about 3.5 times. The results are shown in Figure 2, with detailed numbers available at Table 4 in Appendix A.1 and Table B in the added PDF. It is clear that our method still performs favorably against the strong baseline results directly cited from RL4LMs [2], when T5 gets a lot larger. \n\nTherefore, we believe that our experiments are sufficient to demonstrate the effectiveness and benefits of our method, based on our main results, ablation studies, and the newly uploaded PDF.\n\n\n> **Q4.** My understanding is that the reward is Meteor in the summarization task. What are the Meteor scores on this task?\n\n\n**A.** The reviewer is correct that in the summarization task, we use the Meteor score to obtain the sequence-level preference, which we ground into token-level guidance for LM training by our proposed method.\n\nThe Meteor scores on CNN/DM summarization under T5-base LM can be found at Figure 2 (Section 4.2), with detailed numbers at Table 4 (Appendix A.1) and Table B in the added PDF. By standard, these results are on the test set. The baseline results are directly cited from RL4LMs [2].\n\nIt is clear that our method outperforms these strong baselines in the Meteor metric as well.\n\n> **Q5.** No significant discussion on limitations.\n\n**A.** Due to the page limit, we defer the discussion on limitations to Appendix I (Line 1031-1041).\n"}, {"review_id": "I8imzsl273", "reviewer": "Reviewer_9EYo", "summary": "This paper aims to tackle the misalignment between sequence-level preferences and token-level language model in NLG. The authors design an iterative training framework that integrates the sequence-level preference into token-level training guidance, mitigating the granularity mismatch. Experiments are conducted on two different LM tasks - discrete-prompt generation and text summarization, indicating its effectiveness.", "questions": "Can the performance be enhanced by integrating the token-level feedback with RL-based baseline models? These baseline models appear to show promising performance as depicted in Figure 2.", "limitations": "yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The methodology of decomposing sentence-level preference into token-level preference presented in the paper is both intuitive and rational, providing a feasible approach to address the granularity mismatch in language model training.\n\n2. Through comparative performance with baseline models, comprehensive ablation studies and discussions, the paper demonstrates the proficiency and potential benefits of the proposed model.", "weaknesses": "1. The paper lacks a precise definition of 'preference,' appearing to suggest it is human bias towards superior text quality. However, the method's experimental use of the METEOR score as a sequence-level reward contradicts this assumed understanding. This ambiguity may lead to confusion among readers and raises questions about the evaluation methodology. Why using METEOR as rewards can improve the evaluation in terms of ROUGE scores? Human evaluations should be conducted for a more comprehensive assessment.\n\n2. The reported performance on CNNDM and XSum tasks, measured in ROUGE scores, is below benchmark levels. A basic BART model reportedly outperforms the proposed method (44 vs 40 ROUGE-1 score), which questions the effectiveness of the new training process. This raises concerns that the method may not improve upon existing architectures as suggested.\n\n3. The absence of an accompanying code with the paper is a significant drawback. Without this, it is challenging to reproduce the results, especially given the complexity of applying reinforcement learning. This could limit the wider verification and applicability of the presented method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate the reviewer for your thoughtful review. We would like to draw your attention to our **General Response** on human evaluation results on CNN/DM summarization and a discussion on why our summarization results are below the SOTA.\nBelow, we address your remaining concerns in detail. \n\n> **Q1.** About the definition of “preference” in our paper.\n\n**A.** As discussed in Line 101-103, in this paper we make no assumption about the source of the preference --- it may come from human ranking or task-specific evaluation metrics. Our notion of preference is essentially an ordering of the text-sequences based on the evaluations of full text-sequences, where the evaluations come from automatic evaluation metrics or humans.\n\nTherefore, we believe that using the Meteor score to obtain the sequence-level preference in our experiments aligns with our notion of preference.\n\nWe apologize for the confusion and will make this more clear in the next version of our draft. \n\n> **Q2.** Why using METEOR as rewards can improve the evaluation in terms of ROUGE scores? \n\n**A.** We are a bit confused about this question and will appreciate it if any further explanation can be provided. We provide our current best answer below.\n\nMeteor measures the matching between the model-generated string and the reference string, so we believe that it is a valid preference source for the summarization task.\n\nAs shown in Section 4.2 and Table B in the added PDF, our method improves over the baselines in terms of several metrics: the ROUGE scores, Meteor score, and BertScore. As discussed in Section 4.2, we attribute our performance gain to our main algorithmic contribution: learning and utilizing the preference-grounded token-level guidance for LM training.\n\nBesides, we note that the baseline results from RL4LMs [1] suggest that using Meteor as the environmental reward for training the RL methods, (Supervised+) PPO/NLPO, can lead to ROUGE scores competitive or even stronger than using ROUGE-based environmental rewards.\nThis further validates our use of Meteor as the preference source in the summarization task.\n\n[1] Ramamurthy, Rajkumar, et al. \"Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization.\" arXiv preprint arXiv:2210.01241 (2022). \n\n\n> **Q3.** There is no accompanying code.\n\n**A.** As stated in L​​ine 214, to facilitate the reviewing process, our source code has been anonymously released. The clickable link in the submitted paper is under the red word “released”. Since links are not allowed in the rebuttal text, we respectfully refer the reviewer to our paper for the link.\n\nWe apologize for the confusion and will make the link more apparent in the next version of our draft.\n\n\n> **Q4.** Can integrating our token-level feedback with RL-based methods enhance the performance?\n\n**A.** Thank you for the great suggestion!\n\nSince our LM training objectives in Section 2.2 are only minimalist, we believe that more sophisticated training approaches, such as RL-based methods, can further improve our performance.\nOn the other hand, as discussed in Appendix F.2, due to the granularity mismatch between the native sequence-level feedback and token-level LM training/generation, RL-based LM training can suffer from the delayed feedback issue. As a potential mitigation of this issue, it is promising to integrate our preference-grounded token-level guidance into RL-based methods for LM training. \n\nNevertheless, we kindly note that since our paper is not aimed at RL-based LM training, due to the page limit, the combination between our token-level feedback and RL-based methods is out of this paper’s scope. As discussed in Line 341, we will certainly pursue this direction in our future work.\n"}]

Author rebuttal:
## General Response
We thank all reviewers for the valuable comments. Below are our additional results and responses to common concerns. 

> **Q1.** Human evaluation results on the CNN/DM summarization.

**A.**  We conduct human evaluation on the quality of the generated summaries on the CNN/DM dataset under the T5-base LM. We generally adopt the protocol in [1] to evaluate the overall summary quality.
Our model is compared with the baselines Supervised, Supervised+PPO, and Supervised+NLPO in RL4LMs [2]. We also present the result of the reference summaries, which is intended for sanity check rather than method comparison.

Specifically, we randomly picked 100 articles in the test split of CNN/DM and showed to 20 qualified evaluators the summaries generated from each method, along with the article. The method names were anonymized. The evaluators were asked to read the article and score each summary.

Table A in the added PDF shows the average human ratings. We see that our method outperforms all baseline methods Supervised and Supervised+PPO/NLPO. This aligns with our results in Section 4.2 and **Reviewer xDED**’s expectation that human evaluation on the summarization task supports the improvements in ROUGE scores by our method.

[1] Stiennon, Nisan, et al. "Learning to summarize with human feedback." Advances in Neural Information Processing Systems 33 (2020): 3008-3021.

[2] Ramamurthy, Rajkumar, et al. "Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization." arXiv preprint arXiv:2210.01241 (2022).

> **Q2.** Results on an additional evaluation metric: BERTScore [3].

**A.** As suggested by **Reviewer XL7b**, we add an additional metric BERTScore, which can *“correlates better with human judgments and provides stronger model selection performance”* [3]. 

Table B in the added PDF shows the results expanded from Figure 2 (Section 4.2) and Table 4 (Appendix A.1). 
It is clear that on BERTScore, our method again relatively significantly outperforms the strong baselines from RL4LMs [2]. 

[3] Zhang, Tianyi, et al. "Bertscore: Evaluating text generation with bert." arXiv preprint arXiv:1904.09675 (2019).

> **Q3.** Test against the task/models/results in the paper “Learning to summarize with human feedback” [1].

**A.** In the review, **Reviewer wRKD** suggests us to test our method against the RLHF summarization paper [1]. 

The main reason for not using this task/dataset is *evaluation*. Specifically, [1] mostly uses large-scale human study to evaluate the models. This requires hiring a large number of human evaluators, which is beyond our budget and scope.

As shown in Section 4 of [1], the (baseline) models in [1] are of sizes ranging from 1.3 billion parameters to 12.9 billion, which are beyond our computational budget for making a fair comparison. In fact, these model sizes are even much larger than the T5-base model used in the RL4LMs paper [2] (220 million parameters). 

We are unaware of benchmarking results on this task/dataset on standard automatic metrics, evaluating a wider variety of algorithms and (smaller) LMs. The limited benchmarks further complicates testing our method on this task.

We agree that it is important to test our method using human feedback. We will certainly conduct this study once resources are ready.

> **Q4.** Why are our summarization results below the SOTA?

**A.** We clarify that the goal of this paper is not chasing the SOTA ROUGE-scores on the summarization task. Rather, our goal is to ground sequence-level feedback into token-level guidance for LM training (Section 1). Therefore, we believe that our direct baselines are methods using ungrounded feedback but the same LM backbone, such as RLPrompt and RL4LMs, which we have carefully compared in our experiments.

Further, apart from summarization, we also validate our method on the task of discrete-prompt generation. We believe that our experiments can clearly show the improvement of our method on existing architectures.
  
**Reviewer 9EYo** mentions that a BART-Large model can give a higher ROUGE score. However, BART-Large is ~7 times the size of our T5-small backbone, and ~2 times of our T5-base. We kindly argue that it is unfair to compare our results with other methods that use a much larger backbone.

In Section 4.2 (Line 285-290), we provide the results on CNN/DM when scaling up our LM from T5-small to T5-base. It is clear that scaling up the LM size further improves our results, e.g., ROUGE-1 increases from 40.9 to 43.1 while ROUGE-L from 38.2 to 40.0. Detailed results of our method under T5-base LM are in Table 4 (Appendix A.1) or Table B in the added PDF. It is also clear that under a T5-base backbone, our method performs more-significantly better than the strong Lead-3 baseline on CNN/DM. 

We respectfully argue that apart from using a much larger LM backbone, SOTA summarization methods can contain some specifically-designed techniques, which may interfere with our method and make it hard to separate out the gain of our method.
For example, the BRIO [4] mentioned by **Reviewer XL7b** proposes a training paradigm that assumes a non-deterministic target distribution, which may not easily integrate into the general paradigm of preference-based LM training that our paper considers.

We will appreciate it if our results can be compared against summarization baselines that are more fair and relevant, such as those in the RL4LMs paper. Meanwhile, we appreciate that **Reviewer xDED** considers our evaluation as *“done against strong baselines;”* and **Reviewer gmdH** thinks our experimental results *“should benefit the community.”*

Finally, we are unsure about **Reviewer XL7b**’s comment that “it’s easier to boost the performance on a small model.” We would love to know details/references.

[4] Liu, Yixin, et al. "BRIO: Bringing order to abstractive summarization." arXiv preprint arXiv:2203.16804 (2022).

Meta-review: The paper presents a novel approach to training language models by addressing the granularity mismatch between sequence-level preferences and token-level rewards. Through iterative training, the method integrates sequence-level preference into token-level guidance. The approach is evaluated on two tasks, discrete-prompt generation and text summarization.

Although experiments are not done at a larger scale with bigger models, reviewers agree that the proposed approach is reasonable and most of the experiments are well executed and compared. The exploration of different aggregation methods for token-level rewards is interesting.
Decision: Accept (poster)