Fine-Grained Human Feedback Gives Better Rewards for Language Model Training
Authors: Zeqiu Wu
Yushi Hu
Weijia Shi
Nouha Dziri
Alane Suhr
Prithviraj Ammanabrolu
Noah A. Smith
Mari Ostendorf
Hannaneh Hajishirzi
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Language Model
Reinforcement Learning with Human Feedback
Long-Form Text Generation
TL;DR: We introduce Fine-Grained RLHF, a framework that enables training and learning from fine-grained rewards: (1) density, providing a reward after each granular segment; and (2) mixing multiple reward models associated with different feedback types.

Abstract: Language models (LMs) often exhibit undesirable text generation behaviors, including generating false, toxic, or irrelevant outputs. 
Reinforcement learning from human feedback (RLHF)---where human preference judgments on LM outputs are transformed into a learning signal---has recently shown promise in addressing these issues. However, such holistic feedback conveys limited information on long text outputs; it does not indicate which aspects of the outputs influenced user preference; e.g., which parts contain what type(s) of errors. In this paper, we use fine-grained human feedback (e.g., which sentence is false, which sub-sentence is irrelevant) as an explicit training signal. We introduce Fine-Grained RLHF, a framework that enables training and learning from reward functions that are fine-grained in two respects: (1) density, providing a reward after every segment (e.g., a sentence) is generated; and (2) incorporating multiple reward models associated with different feedback types (e.g., factual incorrectness, irrelevance, and information incompleteness). We conduct experiments on detoxification and long-form question answering to illustrate how learning with this reward function leads to improved performance, supported by both automatic and human evaluation. Additionally, we show that LM behaviors can be customized using different combinations of fine-grained reward models. We release all data, collected human feedback, and codes at https://FineGrainedRLHF.github.io.

Fine-Grained Human Feedback Gives Better Rewards for Language Model Training
Zeqiu Wul Yushi Hul"
Weijia Shi
Nouha Dziri²
Alane Suhr
Prithviraj 4 Noah A. Smith Mari Ostendorf
Hannaneh Hajishirzi
Univeesityy of Washington
AAlle Institute for Artificial Intelligence
University of California, Berkeley
University of California, San Diego MosaicML.
Abstract
Language models (LMs) often exhibit undesirable text generation behaviors, in- cluding generating false, toxic, irrelevant outputs. Reinforcement learning from human feedback (RLHF)-Wel whhee human preference judgments on LM outputs are transformed into learning signal- has recently shown promise in addressing these issues. However, such holistic feedback conveys limited information on long text outputs; it does not indicate which aspects of the outputs influenced user preference; e.g., which parts contain what type(s) errors. In this paper, we use fine-gu greigga human feedback (e.g., which sentence is false, which sub-sentence is irrelevant) an explicit training signal. We introduce FINE-GRAING enables training and learning from reward functions that are fine-grained in two respects: (1) density, providing a reward after every segment (e.g., a sentence) is generated; and (2) incorporating multiple reward models associated with different feedback types (c.g., factual incorrectness, irrelevance. and information incom- pleteness). We conduct experiments on detoxification and long-form question answering to illustrate how learning with such reward functions leads to improved performance, supported by both automatic and human evaluation. Additionally, we show that LM behaviors can be customized using different combinations of fine- one-grained reward models. We release all data, collected human feedback, and codes at http:://FnerraindRRtte .gw..gg.........................
RLHF, a framework that
Introduction
State-of-the-art Al is built on pre-trained language models that are then trained through interaction
with humans [29 28 9. with combination of supervised learning and reinforcement learning.
Incorporating human feedback into the process of language model (LM) training has been shown
effective reduce false, toxic and other undesired model generation outputs (29
3 33 10).
Many of these studies adopt reinforcement learning from human feedback (RLHF) 46 a framework that converts human feedback into effective LM training signal reach these goals. Specifically, humans are presented with two or more outputs and asked to select one rank them, and this signal then used train reward model, which computes single scalar reward for each -M-ggnnrraed sequence. The LM is then trained with RL to optimize the reward it receives (from the reward model).
Such a reward provides relatively sparse training signal, especially for tasks that require the generation of long-form text making RLHF in such domains unreliable (33] Furthermore, previous research 88, 18
into automated evaluation of generated text shows that it can be challenging for human annotators to reliably compare the overall quality of two or more model outputs when the outputs contain a mixture of diverse undesired behaviors. They demonstrate how categorizing and localizing model errors (i.e., fine-grained evaluation) provides explicit insights about which part of the model output has what type of problem. We thus ask the question: how can we improve rewards for LM training via RLHF by using more fine- grained human feedback?
25 4
Equal contribution. Correspondence <Zeqiu Wu: zeqiuwul @ @uw.eduz, <Yushi Hu: yushihu @uw.edu>
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

(a) Preference based RLHF
(b) Ours: Fine-Grained RLHF
Step 1: Collect human feedback and train the reward models
Prompt:
Prompt:
What are the most common gasses in earth's stmosphere?
What are the 3 most common gasses in earth's 's atmosphere?
LM output:
LM outputs:
atmosphere Earth is a layer gases refained by Earth's gravity The most common gas, by dry air volume, nitrogen. The second most is oxygen. The third most is carbon dioxide;
The atmosphere of Earth a laper of gasea retained by Earth's gravity...
The atmoophere is commonty top gumes by volume dry
The that sumounds planet Earth contains various groes. Nirogen...
atmosphere Earth the layer gasvss, generally known as air...
Fine-Grained Human Feedback
Relevance RM
Irelevant// / Redundant
Human Feedback
Factuality RM
Unverifiable / Untruthful
A
Preference
Information Completeness RM
Missing
The third most Aigon
Step 2: Fine-tune the policy LM against the reward models using RL
Sampled Prompt: Does water bail quicker at high altitudes?
Sampled Prompt: Does water bail quicker at high altitudes?
Relevant: + 0.3 Factual: 0.5
It takes longer water to at high altitudes. The reason is that water boils at
takes yinger water to
high
eltiuudee. Ihe reason (e that water bois at a lower temperature at higher altitudes
a lower temperature at higher altiuuees
PPO
Preference
Relevant: 0.3 Factual:
completer
Update policy with rewards
Update policy with rewards
Figure 1: Comparison of (a) RL with human preference and (b) our FINE-GRAINED RLHF on long-form QA. Different from (a), which collects human preferences on the overall quality of LM outputs, we ask annotators to mark which part of an output contains what type(s) of errors. We train a fine- grained reward model for each type of error and optimize LM against these reward models. In this example, we provide a relevance reward and a factuality reward after each sentence is generated. There is also a holistic information completeness reward after the whole text is generated.
In this paper, we propose that humans give fine-grained feedback to LM output, associating categories of undesired behavior (e.g., false or irrelevant generations) and text span at a density (e.g., sentence or sub-sentence-level) To enable LMs to learn from such fine- enaiinri feedback, we introduce the FINE-GRAINER RLHF framework. As shown in Figure 1 to train fine-grained reward models such that each of them focuses on one category and provides rewards at the density associated with that category. We then integrate these reward models into Proximal Policy Optimization (PPO) a commonly used RL algorithm for training LMs with preference-bast based human feedback ($2)
we first use collected human feedback
We conduct experiments on two language generation tasks detoxiditoiiiicattion 12 and long-form
question answering (QA) [39 ( For detoxification, toxicity is the only error category and we explore learning with dense reward. We adopt PERSPECTIVE [I. widely used language toxicity detection model trained on millions of human annotations, as our reward model. We use it to calculate a fine-graine -naigrai reward after the generation of every sentence. Our experimental results show the efficacy and data efficiency of training models with dense reward compared to a holistic sequence-level reward, supported by automatic evaluation results.
With experiments on long-form QA, we aim to examine training models with fine-grained rewards the two granularity dimensions (density and error category), for which we construct long-form QA dataset, QA-FEEDBACK along with our collected human feedback. We carefully develop a pipeline to collect fine-grained human feedback on three error categories different density levels: i) irrelevance, repetition, or incoherence (sub-sentence), ii) incorrect or unverifiable facts (sentence), and iii) incomplete information (whole sequence; see Figure [T]. Our experimental results show improved results in each error category by learning with such fine ine-ggained feedback, supported by both automatic and human evaluation results. In a scenario with multiple reward models representing different error types, we also show different weights and thus control the model training process towards customized combination of desired behaviors.
RLHF allows us to combine reward models with
2

---

2 FINE-GRAINED RLHF
We introduce SINE-GRAINED RLII,, a framework that enables us train fine-grained reward func- tions for generation outputs across different feedback types. We first define the RL environment and learning algorithm. Then we define the fine-gre -eeignadddd reward models and describe how incorporate the fine-grained reward model(s) into an RL algorithm, in contrast to previous RLHF studies that only consider single reward.
Environment: language generation as a MDP. We focus on language generation tasks. For each task, given set of task input prompts D {-"" 1==1 1- We follow [33 define language generation as a Markov Decision Process (MDP) (S. R, n. maxx with a finite vocabulary V. Each MDP episode starts with a sampled prompt
(01, ................................ .,))) with x; V, and ends when the current time step exceeds Tmar or an end of sequence token is generated. S is the state space and 50 (I), .,,,..,, 22) ) S is the initial state. An action in the environment a, AA is a generated token (by the policy LM model Pe) at time from V (ag is the begin sequence token). The transition function P: S x AA- + AS appends a at the end of the state 8t = (I1, U2,..2..... 21, au, a1:... ... a(-1). This process continues until the end time step T S Tan is reached, which gives generated sequence y (o)..... .r.). A reward function R: : S A + R, which comes from the reward model(s) in FINE-GRAINED RI.F,, provides dense rewards before and when 7 is reached. Po can be initialized with pre-trained language model, and sometimes also with supervised fine-tuning on task specific demonstrations. The reward function is defined later.
an actor-critic RL algorithm that is widely used in previous RLHF work to optimize the policy model against a reward model of human feedback. It uses a value model (st(st) to estimate the value of state 8t. and optimizes the policy model with a PPO clipped surrogate training objective. The advantage A, at timestep "Tre VV (1) - (e((s)) )). with y as a hyperparameter and as the discounting factor for rewards. r; is the reward assigned to ae, which in our case is acquired using one or multiple learned reward models. The value model Vi (st) is optimized with an expected squared-erros loss with the value target as (s,) = > 2r'=t yd "ry + +TT-VV tou (ST). where Volus is the lagging value model. Finally, PPO is trained to optimize both policy (Pa) and value (V₂) models with their respective objectives. No reward model is being optimized during PPO training. See Appendix B for more details.
Learning algorithm:
proximal policy optimization (PPO). PPO [37
t is estimated by a generalized advantage estimation function [36]: A. =
Fine-grained reward models. Previous RLHF work adopts holistic reward model Rg that maps input prompt x and generated output y to a single scalar reward representing its overall quality (Figure I [)). This single scalar reward only assigned to the final token in the generated sequence. ar. Formally, r. = R5(2, y) if = T and 0 otherwise.
In contrast, we consider a reward function that is derived from or multiple moultipl.fine-eeee fine-grained reward models that (1) provide rewards densely (i.e., for subsequences of the generated output), and (2) compute rewards on distinet categories of undesired behaviors (e.g.. false or repetitive generation). where each category is associated with an individual reward model.
For fine- grained reward model Røs that gives feedback error category Ck, we first segment y into La segments (vf' vž. v......... vi. corresponding to the density (c.g.. sentence-level) of R where each segment S/ ends timestep TyK. Ros outputs a reward Rşa (x, v. for each segment ッ² given x and y as the input, which is assigned to the final token in y² Additionally, to ensure the fluency of generated outputs,
follow [41] to add approximate KL divergence penalty each token a with a weight 8, that is not backpropagated through during training. Formally, assuming that we have K fine-grained reward models that represent different error categories, we will have combined reward function for each token ar as:
K
Tike
(1(t= T5) we Ros (2, (a.,,.)) - 3331 log (a (at st) Po (a, 8t)
k=1==1
where Wi E R is a weight assigned to reward model Røg- Then we follow the same PPO training algorithm to optimize the policy model. We discuss how we define and train fine-grained reward models for the detoxification and long-form QA task in our experiments in $ and respeetivvely
3

---

3
Task 1: Detoxification
The task of detoxification aims to reduce the toxicity in the model generation y when given a prompt x. Toxicity is the only undesired behavior in this task, and we aim explore learning with a dense reward in comparison to a single holistic reward. We conduct our experiments on REALTOXICITPROMPTT
dataset of 100K sentence-level prompts derived from the web that known to easily elicit problematic generations in GPT-2 [3]. Using a dense sentence-level fine- grained reward, we demonstrate that our fine-grained reward exhibits greater sample efficiency compared to a holistic reward, achieving lower toxicity with fewer training steps while maintaining better fluency ($3.T).
Holistic reward for (non-)Toxicity, We use the PERSPECTIVE API [T] as our reward model, which is widely used for language toxicity detection and is trained with millions of examples gathered from several online platforms and annotated by human annotators for toxicity. That means we use an off-policy reward model that is not trained outputs from Po....- The API outputs score between 0 (non-toxic) and (toxic). Given the entire model output y, the holistic reward for RI. is 1- --PERPPECTIVEE))
Sentence-level (fine-grained) reward for (non-)Toxicity. To calculate the fine-grained reward, we query the API after the model generates each sentence instead of generating the full sequence. For each generated sentence yj. we assign PERSPECTIVE(111.... as the sentence reward (i.e., how much toxicity is changed from generating fy). Since there is only one error category, we omit the category superscript, using yj to denote the jth segment (c.g., sentence) in y.
y--1)) PERSPECTIVE((31.
yjD)
3.1 Experiments
Implementation details. We follow previous work [1 21 and use GPT-2 large model as the initial policy model Pa.... During both the exploration stage RL training and inference, we use nucleus sampling decoding with p 0.9 and temperature 1.0. The generation length limit is set to 48. The value model used during RL training is initialized with GPT-2-base due to GPU memory constraint. We report RL training parameters in Appendix B All scores averaged over 3 independent runs.
Compared systems and evaluation. We report the performance of GRAN-ED RLHF, RLHF with holistic reward (Hol. RLHF), and the state-of-the-arr controlled generation approaches GeDi 17 and DEXPERTS 21 We follow previous work [17 21 to report the toxicity score calculated on each full generation sequence from the PERPLEXITY API, as well as other commonly used metrics for REALTOXICITY PROMPTS, including n-gram diversity and GPT-2 XL perplexity (PPL) as a proxy for fluency. The lower the perplexity, the more fluent the generated text. The toxicity score is reported as the maximum score among 4 sampled model outputs, averaged over test input prompts. Other metrics reported the average score of the same 4 samples.
Texicity avg max (4)
Fluency PPI. (1)
Diversity
max toxicity
PPL
dist (1)
dist-3 (?)
FG. RLJ
GPT-2
0.192
9.58
0.947
0.931
11.5
Hot. KLEF
Controlled Generation
11.0
GeDi
DEXPERTS
0.136
0.154
24.78
22.83
0.938
0.932
0.922
0.3
10.5
Hol. RLHF
0.130
11.75
0.943
0.926
10.0
F.G. RLHHE
0.081
9.77
0.949
0.932
0.0
200
400
600
2000
600
Figure 2: Curves of toxicity and perplexity on the dev set vs. training steps.
Table 1:
Results on the REALTOXICI-
TYPROMPTS test set.
Main results. Table I shows the experimental results on the REALTOXICITY test set. FINE- GRAINED RLHF with sentence-level fine-ge -ne-grained reward attains the lowest toxicity and perplexity among all methods, while maintaining similar level of diversity.
Sample efficiency analysis. Figure Shows
shows the max toxicity and average perplexity on the devel- opment set during training. FINE-GRAINER RLHF has the toxicity drop much faster while keeping a low-level perplexity. This shows that learning from denser fine-grained reward more sample efficient than holistic reward. One explanation is that fine- sine-grained reward locates where the toxic

---

content is, which is a stronger training signal compared with scalar reward for the whole text. The cost that we have to query the reward model more times per example.
4 Task 2: Long-Form Question Answering (QA)
Long-form QA requires an LM to generate textual response question with comprehensive answer and explanation. To examine learning with fine-grained rewards at the two granularity dimensions (error category and density), we collect QA-FEEDBACK ($4.1), a long-form QA dataset annotated with human feedback on LM-generated responses. We define three error categories at different density levels and train reward model for each ($4.)). We describe the experimental setup in $4.3 Both human and automatic evaluation show that
RLHF outperforms preference- RLHF and supervised fine-tuning models on all error categories ($44.4) We then show that adjusting the weights of fine-grained reward models during RL training leads to distinct behaviors in LM generation, allowing us to customize the LM for users with different needs ($4.5). Finally, we conduct an in-depth analysis of the fine-graieed reward models, revealing that they compete against each other, and provide an analysis of their impact on the resulting policy model.
4.1 QA-FEEDBACK: Long Form QA with Human Feedback
QA- QA-FEEDBACK is based on ASQA [3), a dataset that focuses on answering ambiguous factoid questions (26) in open-domain setting. We use their provided oracle knowledge contexts to reformulate the task into a reading comprehension setting: given the input æ that contains a question q and a set of knowledge passages P {pI, . P.rih generate long-form response y. On average, there are roughly 65 words in each gold response. Since ASQA does not release the test set, we create own train/developmennttest data split from the original train and development sets. We name our newly constructed data, along with collected human feedback (discussed next). QA- DA-FEEDBACK. Overall, we have 3,853 training, 500 development, and 948 test examples (details in Appendix (
Initial policy and fine-grained human feedback. Before collecting human feedback, we follow 33 to initialize the policy model with supervised fine-tuning on a small set of examples. Specifically, we use IK training examples to supervise fine- aie-ttning of T5-large (the original baseline for ASQA) 32] to get Po We name this initial policy model SFT. We then sample outputs from SFT for the remaining training and development examples and collect fine-grained human feedback in three error categories- C1: irrelevance, repetition, or incoherence; ; Cz: incorrect or unverifiable facts based on knowledge passages; and C3: incomplete information The collected feedback instances are then used as the training and development examples for training reward models. For each task prompt x, we only collect -raiigga feedback for one model output. Our data collection has IRB approval and is deemed exempt.
We instruct workers to identify any error in each model output y = (aI..... (rr), marking the span of text associated with each identified error type. Formally, we define the set of user-annotated feedback for a task prompt and model output y as F {fi} where each f: (c,, b,, e4) represents the user-identified span (ab....... de;) ) of the error category Ca., where e; E {1, 2, 3). Importantly, we impose three restrictions in the annotation: (1) error spans of category Ci or C2 should not overlap with each other; (2) only spans that do not have error C1 need to be assessed as containing error C2 not; (3) C33 can only apply to whole output sequences. Additionally, we ask workers to mark passage sentences that contain missing information if a C33 error is annotated. We also ask workers to rewrite y into corrected version y/ that addresses all annotated feedback F. Details about the feedback collection interface, instructions, and quality control are in Appendix
To analyze human-human agreement, subset of 300 examples receive annotations from two distinct workers. We observe that while exact agreement in error span boundaries is low, workers achieve reasonably high agreement
whether a sub-sentence contains C₁ and whether sentence contains C2 C Therefore, we decide to have the density for error type C1, C2, and C3 as sub-sentence, sentence and full sequence. We provide more data analysis including human agreement in Appendix C
Preference-based human feedback. For comparison purposes, we follow 29 to separately collect pairwise human preferences from the same group of workers. We sample 4 model outputs for each prompt x, which gives pairs of model outputs. We ask the workers to indicate pairwise preferences
We use spaCy 15 to segment generated model outputs into sentences. We then split sentences into sub-sentences using a comma or semicolon.
5

---

(ties are allowed) based on all errors they can find in each model output. They are not asked to explicitly annotate these errors.
Annotation details. On average, both annotation tasks of fine-grained and preference feedback for one question take a worker about 6 minutes finish. In contrast, (39] report that they spend about 15 minutes to label a human-written response for each question, which is much more time-consuming than our feedback annotation. On average, we pay $1.65 per example for both tasks, leading to $16.50 hourly pay for our workers. We include details of the pay structure in Appendix C We observe that human annotators can reach higher agreement in each aspect of fine-gre mne-grained feedback compared to pairwise comparisons because the feedback definitions are more concrete.
4.2 Fine-Grained Reward Models
We train three separate reward models Røi, Røg. and Røg for C₁ C2, and C3 error categories respectively with a density of sub-sentence, sentence, and full sequence, respectively. Since reward models provide scalar reward scores and do not perform generation, we use the encoder-only Longformer-bar 4 as our backbone model to handle long input sequences (more details of each reward model are in Appendix D)
C Irrelevance, repetition, or incoherence. Rs, targets to predict whether each sub-sentence in y contains C1 type error. We denote y (v1- ......../.. ). where yf is the jth segment at Re, 's density (i.e., sub-sentence), with 1 segments in total. We add 2-class token-level classification layer (a single feed-forward layer) on the top of the Longformer encoder. The model input has the format of "question: answer: [sep] yl [sep] 1/2 ...". and we take the classification output at each [sep] token to indicate whether the following 3/ contains a C₁ error. We do not add passages in the model input because, intuitively, the detection of C1 errors does not depend on them. To train Rø,, we apply a token-level classification loss to each [sep] token before w). where its gold label gj is "has error" if there is a Si E F that has (un..... ... (e.)) overlapped with y and c; = 1, and "no error" otherwise. When R₂ provides reward during RL training as in Eq. [ we read reward Rø, (ar, j) for every vj given and y. We define R2, (x. +1 if Rø, predicts "no error" for y and 1 otherwise.
C2: Incorrect or unveritiable facts. R2g is developed for detecting a C2 error at the sentence level in similar way. The model input has the format of "question: q context: P1 P2 answer: [sep] yी [sep] y2 ...". where p's denotes the grounding passages and yf represents the jth sentence. We train Rø,, similarly to Rø,, with one exception: as we instruct the workers not to annotate a C2 error for a span that is already labeled containing C₁ error, we do not calculate loss on sentences that are labeled as containing Ci but not C2 during Røz training.
Cz: Incomplete information. Røg is trained to measure the information completeness of y, the full sequence level. Motivated by [19], Ros predicts a single scalar reward and is trained with pairwise comparison loss [29]:
(+) (.. log (o Bp) Ross (x, Bii))
(2)
where Rig, (I. 3) is the scalar output of the reward model for input and output y; Fp and ÿt are sampled from the same input x, and Up has less missed information compared with ūi: Dr contains the pairwise comparisons bootstraped from human feedback on C3 errors (see details Appendix
Preference-based reward model. The preference-bare based reward model is trained in a similar way to Røg. with Up representing the human preferred response against कt in the loss function Eq. 2 It outputs scalar score for the given r and y that represents the overall response quality.
4.3 Experimental Setup
Compared systems. We compare our proposed method, FINE-GRAINED RLHF with the initial T5 policy model trained with IK examples (SFT) and RLHF with holistic preference-bante (Preference RLHF). The reward models used in RLHF experiments are trained on 2.8K examples with annotated feedback (but no gold human response). For analysis, we also use the human gold responses of all training examples to finetune a fully supervised T5 model (SFT-Full). Notice that SFT-Full requires much higher annotation cost because it takes longer (15 minutes per example (39 for annotators to draft long- form responses.
rewards

---

Implementation details. Our policy model is based on T5-Jarge [2] and is supervised finetuned on IK training examples, as explained in $ During RL exploration, we use top-k (k 20) sampling decoding with temperature 0.7, which is set based on previous RLHF work 33 The value model used during RL training is initialized with T5-base due to GPU memory constraint. The reward model weights we used in FINE-GRAINER RLHF are W1 0.3, W2 = 0.5, W3 0.3, unless otherwise specified. Although we use three reward models during RL training, we only observe very small relative additional cost (roughly 1 1 training time) compared to preference RLHF, During inference, we use greedy decoding to generate responses. We report more details including RL training parameters in Appendix B. All scores reported are averaged over 3 independent runs.
Evaluation. We conduct both human and automatic evaluation. Human evaluation is run on 200 randomly sampled test set examples of QA QA-FEEDBACK to compare Fine-Grained RLHF with all baselines. Each model output sampled from inference results of 3 training runs. We use the same protocol of feedback collection to have the same set of workers annotate spans in each model output that contain (1) irrelevance, repetition, or incoherence error (rel.) and (2) incorrect or unverifiable facts (fact.) They are also asked compare the information completeness (comp.) for each output pair. To report evaluation scores for rel. and fact. error spans, we first map them to their corresponding error type density (sub-sentence and sentence). Then we report the error rate for each error type, measured as the percentage of sub-sentences that contains this type of error. Since spans with rel. error are not checked for fact. error (discussed in $4.1). we exclude sub-sentences with only rel. error when report the eπror rate of fact. error. For automatic evaluation, report RougeLSum [20] as used for the original ASQA data, as well as the score from each fine-grained reward model (Roi Røg. and Røg). Specifically, we report the percentage of all sub-sentences (or sentences) in the test set predicted as "no error" by Rø, (or R$z). For Røg. we report the averaged output score for all test examples.
4.4 Main Results
Figure 3 shows the human evaluation results for rel. and fact. error types. Table shows the human pairwise comparison results for information completeness (comp.).
Ours vs.
Win
Tic
Lose
0.98
ErorRate
0.543
0.557
0.541
0.18
0.185
SFT-Full
SFT 23.0%
22.0%
65.5%
61.0%
11.5%
17.0%
0.82
0.16
Pref. RLHF
19.5%
71.0%
9.5%
0.14
Table 2: Human pairwise compari- on information completeness (comp.) where win/lose refers
8FT-Ful
Pruf. RLHF
Figure 3: Human evaluation on rel. (left) and fact. (right) error, measured by % of sub-sentencee that contain the error type (L). FINE- GRAINED RLHF.
FINE-GRAINES RLHF outperforms SFT and Preference RLHF all error types. Figure and Table show that our CONE-GRAINED RLHF lcads to generation that is much more factually correct and contains more complete information, compared to all other systems. It generates fewer irrelevance, repetition, andincoherence errors, compared with SFT and Preference RL.F. In the meantime, Preference RLHF, despite greatly reducing factual errors compared to the initial policy model SFT, generates even more irrelevance, repetition, and incoherence errors than SFT. FINE- GRAINED RLHF outperforms Preference RLHF potentially due to more specific and localized training signals. In addition, we ask annotators to compare the overall generation quality of FINE-GR AINED RLHF and preference RLHF. Although Preference RLHF is trained directly with such preference feedback, FINE-GRAINED RL.FF was rated better than Preference RLHF 30.5% of all examples and worse 24.5% of examples. The annotators indicate a tie the remaining 45% of cases. Surprisingly. FINE-GRAINED RLHF outperforms SFT-Full with more factual and complete generation, despite much lower annotation cost.
shows that both FINE- RLHF is particularly effective in reducing factual errors. Figure 3
GRAINED RLHF and Preference RLHF are effective in reducing factual errors in model generation. Meanwhile, we see little or no improvement in reducing irrelevance, repetition, or incoherence errors. We provide more in-depth analysis for this observation in $4.5)
Table 3 shows automatic scores on the QA-FEEDBACK test set, which show similar trends as human evaluation in terms of system comparisons, while all four systems achieve similar Rouge scores.
7

---

rel.
fact.
comp.
rel.
Fact.
comp
uvg.
Ray(1) (1)
(1)
(1,(1)
Rouge(1)
Conlig
Ho,
Ro,(1)
Noy(1) (f)
Rouge(t)
len
SFT-Fall
0.508
0.756
0.044
49.63
Short
0.637
0.760
-0.231
48.99
74.92
-0.053
Medium
0.513
0.816
0.139
49.93
98.66 66
+ Prof. RLHF
0.482
0.781
0.101
49.84
Long
0.425
0.860
0.241
48.72
109.63
F.G. RIHF
0.513
0.816
0.139
49.93
Table Automatic evaluation results (test set) of FINE-GRAINED GRAINED RLHF trained with different reward model weight configurations.
Table 3: Automatic evaluation the QA-
FEEDBACK test set.
4.5 LM Customization with FINE-GRAINED RLHF
RLHF, adjusting their weights (see Eq. T during RL may lead to different LM behaviors. For example, adding more weight to a reward model associated with one specific desired behavior type (e.g., information completeness) may lead the generation more towards that behavior type compared to others (e.g., information relevance). This flexibility can potentially fit users with diverse needs. Therefore, in this section, we explore FINE-GRAINED RLHF's ability to customize the LM behavior.
Since multiple reward models in
LM customization. As in T we explore three configurations of reward model weights (W1, W2- and wy for Rø., Ræg. and Røg) and name them 'short', 'medium', and 'long' according to the LM's average generation length. For simplicity, for W1, which leads to 'short', 'medium', and 'long' generation outputs respectively. We manually inspect 30 random examples and observe that (1) 'short' generates more relevant content, but is less factual and complete; (2) 'long', in contrast, gives the most factual and complete generation. This reflects that the LM is referencing large amount of content from passages; (3) The 'medium' configuration balances the three rewards and has the highest Rouge score. 24/30 examples follow the above rule. Qualitative analysis and examples of LM customization are in Appendix(A]
fix w2 0.5 and Wt 0.3, and use 0.4, 0.3, and 0.2
Trade-off between error types. We observe that a higher wi leads to a bigger rel. reward, smaller fact. and comp. rewards, and shorter generated outputs. One interpretation is that Rş, penalizes text spans that are irrelevant to the questions. As such, it encourages answering the question directly and penalizes referencing passages and generating auxiliary information. This reduces the model generation length and information completeness, and induces more factual errors.
4.6 Analysis
rel. reward
fact, reward
reward
rel.
fact.
comp.
avg.
Re,
Rs,(1)
Rag(T)
Rouge(t)
len
SFT
0.514
0.735
0.065
43.13
96.69
FG. RLHF
0.516
0.825
0.266
44.29
101.76
0.249
0.771
0.742
38.52
179.31
w/o.
Ray 0.716
0.640
0.177
43.18
78.08
w/o. 0.565
0.799
Figure 4: Dynamics of each type of re- ward during training (reward vs. training steps). All rewards are z-normalized.
Table 5: Ablation of reward models on the development set. Rig, Ręz and R&s, correspond to the reward model for relevance, factuality, and information completeness.
Reward models are competing against each other. In the prior section, we find that there is trade-off between error types. To further look into this phenomenon, we explore the dynamics of each reward model during training. set during training. All rewards are z-normalized for visualization. We see that the fact. reward consistently inereasing, The rel. reward increases rapidly in the first 250 steps and then starts decreasing, while the comp. reward exhibits an opposite trend, decreasing at first and then starting to increase. As discussed earlier, one interpretation is that relevance (precision) and information completeness (recall) can be adversarial objectives, so the rewards are competing. The three rewards reach equilibrium point in later steps.
[quuee/showe each reward model's rewards on the development
Ablation: Does the LM learn from all reward models? What if we remove one reward model? Table 5 explores the policy LM behavior when one of the three reward models is removed during training. Qualitative examples are in Appendix A First, we observe that the corresponding reward decreases dramatically when the model is removed. When the rel. reward model (Ra, is removed,

---

the outputs become extremely long and the comp. reward is extremely high. We observe the outputs and find the model copying a lot of content from the passages. When the fact. reward model ( RRg)) ) is removed, the rel. reward becomes the highest. We observe that the LM tends to answer the question directly and not reference the passages, which causes a lot of hallucinations. When the comp. reward model Rggg ) is removed, the outputs are concise and factual but not providing all relevant information to the question. Thus, has lower information completeness and Rouge score compared with the LM trained with all reward models.
Reward model performance. We report and analyze the performance of each reward model in predicting its corresponding error category. The rel. reward model Rg, has binary classification accuracy of 69.6, and an Fl score (for the "has error" class) of 68.5 on model-generatie sentences from the development set. We sample 20 sub-sentences where RS, predicts the opposite of the human label, and observe that all of them either 1) contain relevant auxiliary information and are marked as "no error" by humans, or 2) are marked as irrelevant by humans but provide closely related background information to the question. In other words, Rø, is mostly struggling with predicting the relevance of auxiliary information, and it rarely fails to predict direct answer as "no error".
sub-
The fact. reward model Ray has an accuracy of 77.8 and an Fl score of 67.5. We sample 20 sentences where RØ, makes a prediction mistake and we observe that the mistakes often happen when the generated sentence is highly abstractive instead of directly copying information from the passage. We also observe that more than 80% of human labeled factual errors occur when the model generates a direct answer (not auxiliary information) that contains hallucinated information or a random entity from a passage. We notice that Rø, correctly captures more than 80% of such errors.
The comp. reward model Rog has an accuracy of 70.9 in pairwise comparison. In contrast, the preference-based reward model only reaches an accuracy of 57.2. This helps confirm our intuition that assessing long-form generation outputs holistically can be more ambiguous and subjective than evaluating the outputs with a focus on a specific undesired behavior type.
Comparison with ChatGPT responses. We experiment with answering the questions with ChatGPT. To familiarize ChatGPT with the style of our LFQA task, we prompt it with the task instruction and a single random QA example (due to length limitation). ChatGPT achieves a Rougel .Sum score of 40.92 on the test set, which is much lower than our models. We do not use our trained reward models to evaluate ChatGPT outputs because reward models trained on T5-large may not generalize well to ChatGPT. We instead manually inspect the ChatGPT responses, and observe that they are mostly concise and factual, yet lack the auxiliary information necessary to clarify ambiguous questions. Qualitative examples are in Appendix A This shows the difficulty for ChatGPT in learning user-desired behaviors through simple prompting.
5 Related Work
aims to optimize the policy language model to generate content that desired by human. This framework has been explored to improve the model performance on variety of natural language processing tasks such
Reinforcement learning from human feedback (RLHF). RLHF 46 42 29
as text summarization [40), instruction following (29). question answering [24] 27 and reducing
harmfulness [3] 2 22 10] Most of these studies collect human preferences over pairs of model
outputs on one or a set of desired attributes, in order to train a reward model to assign a holistic score for generation output during RL training.
[ trains separate reward models that assign scores for different desired attributes, but still uses a single reward that combines scores from all reward models. In contrast, we explore RLHF with fine-graine reward models trained on human feedback where each reward model provides dense reward after every small text segment for a specific type of desired behavior. (30 explores using intermediate rewards improves LM performance on reasoning tasks. Learning from human feedback in NLP. There also exists work that explores non-RL methods to learn from human feedback. 44 trains a reward model that predicts a single score for each model
output and selects samples with the highest reward scores for supervised fine-tume fine-tuning. [38] 14 42
train a conversational model to predict both the response and a binary user satisfaction score in order to improve the response generation. Besides such numerical human feedback, natural language (NL) human feedback has also been explored. 23 6 collect and store NL human feedback in feedback memory for the model to retrieve and then perform the end task conditioning on the retrieved feedback. use refinement model to refine model outputs conditioning on NL human feedback and
35 34

---

then use a reward model to select the best refined outputs for supervised fine-tuning. Methods for using a reward model guide LM generation towards desired behaviors at inference time 22 can complement our work that aims to improve the LM during training. [16] also explores incorporating human feedback into LM pre-training.
Discussion
Annotation Costs. It is important to note that the fine- tne-grained human feedback used for training fine-grained reward models does not incur a greater cost than holistic human preference. As outlined in 4.2, our observations reveal that annotators require substantial amount of time compare two lengthy text outputs. For the long-fom QA task, both fine-grained feedback and preference- baased feedback takes approximately minutes per sample for an annotator.
6.1 Broader Impacts
We propose the FINE-GRAINED RLIIF framework that can incorporate multiple reward models to provide dense rewards for RL training, which leads to LM outputs that are optimized towards such rewards. Our framework can be applicd to any text generation task, thereby enhancing LM performance by offering more nuanced guidance than holistic feedback. The key advantages of the FINE-GRAINED RLHF framework are two-fold:
Flexibility, Our framework significantly expands the versatility of reward models for RLHE For example, future work involving fact-checking, sentiment classification, toxicity detection, among others, can all be incorporated within this framework. LMs can be trained against all these reward models via FINE-GRAINED RLIIF.
Controllablility. Having multiple reward models that stand for different feedback types allows the end user to exert greater control over RL training (e.g., through different combinations of reward model weights; see details in $ $1..5. This leads to customized model behaviors, a benefit particularly valuable for applications like educational tools where model personalization is crucial.
6.2 Limitations and Future Work
One major limitation of our framework comes from the additional compute cost of getting fine- grained rewards, compared to RLHF with holistic reward. For instance, the detoxification task, we need to make multiple PERSPECTIVE API calls for each model output depending on how many sentences generated, while RLHF with holistic reward only requires one. In the long-form QA task, we need to calculate a dense reward from multiple reward models, which takes more compute time and GPU memory than single reward model.
Another limitation is that different tasks may have different definitions of fine-grained feedback in terms of the feedback types and the density level of each type. Therefore, defining feedback that is well-suited for a task and training reward models accordingly requires non-trivial manual effort.
Finally, in this work, we carefully control the quality of annotated feedback, which is then used to train reward models for RL. In practice, when a deployed model is released to the public, end users don't 'on't always give clean feedback. Therefore, how to obtain effective learning signals from noisy human feedback the wild still needs further investigation.
Some other interesting questions to explore in the future include: 1) Can we obtain fine-graid graiira feedback from LMs like GPT-4 instead of humans to improve model performance and reduce annotation costs? 2) How can other non-RL approaches of using human feedback such as controlled generation during inference time complement PINE-GRAINED RLHF? 3) How would fine-grained reward and value model sizes affect policy model performance during RL training?
7 Conclusion
In this work, we introduce FINE-GRAINED RLHF, a framework that enables LMs to learn from multiple fine-grained reward models trained from human feedback, where each reward model detects a specific error category and provides dense rewards. We conduct experimental analysis on two text generation tasks to illustrate the performance gain of FINE-GRAIE DINE-GRANNED RLHF than RLHF over holistic rewards, supported by both automatic and human evaluation. Furthermore, we show that an LM can be customized for specific needs using different combinations of fine-gr inn-grained reward models.
10

---

Acknowledgments
We thank Jiacheng Liu for sharing the standard PPO training code, and Yizhong Wang for providing insights during early discussions of the project. We also thank UW TIAL members for participating in our pilot feedback annotation. We extend our thanks to UW NLP members who provided insights or feedback to our project. Lastly, we especially thank all our AMT workers for helping us annotate the high quality feedback data. This research was developed with funding from the Defense Advanced This work was also
Research Projects Agency (DARPA) under Contract No. FA8650-23-C-7735 funded in part by the DARPA MCS program through NIWC Pacific 186001-19-2-4033 IIS-2044660, and ONR N00014-18 1-1---11112666 The views, opinions and/or findings expressed are those of the author and should not be interpreted as representing the official views or policies of the Department of Defense or the U.S. Govermentt
NSF
References
[1] Perspective htp.l/pee/gggttooo..coo//rrrrraa
[2] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 2022.
[3] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08077: 2022.
[4] Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint SEEV:220040015555 2020.
[5] Angelica Chen, Jérémy Scheurer, Tomasz Korbak, Jon Ander Campos, Jun Shern Chan, Samuel R Bowman, Kyunghyun Cho, and Ethan Perez. Improving code generation by training with natural language feedback. arXiv preprint arXiv:2303. 16749, 2023.
[6] Bhavana Dalvi Mishra, Oyvind Tafjord, and Peter Clark. Towards teachable reasoning systems: Using a dynamic memory of user feedback for continual system improvement. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9465-9480, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
[7] Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text generation. In International Conference on Learning Representations, 2020.
[8] Yao Dou, Maxwell Forbes, Rik Koncel-Kedziorse text indistinguishable from human text? scarecrow: A framework for scrutinizing machine text. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7250-7274, Dublin, Ireland, May 2022. Association for Computational Linguistics.
Kedziorski, Noah Smith, and Yejin Choi. Is GPT-3
[9] Yao Fu, Litu Ou, Mingyu Chen, Yuhao Wan, Hao Peng, and Tushar Khot. Chain-of-thought hub: A continuous effort to measure large language models' reasoning performance. arXiv preprint arXiv:2305. 2023.
[10] Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas Liao, Kamilè Lukošiūté, Anna Chen, Anna Goldie, Azalia Mirhoseini, Catherine Olsson, Danny Hernandez, et al. The capacity for moral self-correction in large language models. arXiv preprint 2302.02000 2023.
[11] Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. Enabling large language models to generate text with citations. arXiv preprint vrXiv:2305.14627 2023.
[12] Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. Real- ToxicityPrompts: Evaluating neural toxic degeneration in language models. In Findings of the A Association for Computational Linguistics: EMNLP 2020, pages 3356-3369, Online, November 2020. Association for Computational Linguistics.

---

[13] Amelia Glaese, Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209. 14375, 2022.
[14] Braden Hancock, Antoine Bordes, Pierre-Emmanuel Mazare, and Jason Weston. Learning from dialogue after deployment: Feed yourself, chatbot! In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3667-3684, Florence, Italy, July 2019. Association for Computational Linguistics.
[15] Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. spaCy: Industrial- strength Natural Language Processing in Python. 2020.
[16] Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Bhalerao, Christopher Buckley, Ja- son Phang, Samuel R Bowman, and Ethan Perez. Pretraining language models with human preferences. arXiv preprint arXiv:2302. 085822, 2023.
[17] Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. GeDi: Generative discriminator guided sequence generation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4929-4952, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[18] Kalpesh Krishna, Erin Bransom, Bailey Kuehl, Mohit lyyer, Pradeep Dasigi, Arman Cohan, and Kyle Lo. LongEval: Guidelines for human evaluation of faithfulness in ong--fomm summa- rization. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1650-1669, Dubrovnik, Croatia, May 2023. Association for Computational Linguisties.
[19] Margaret Li, Jason Weston, and Stephen Roller. Acute-eval: Improved dialogue evaluation with optimized questions and multi-turn comparisons. arXiv preprint arXiv: 1909, 119999000877 2019.
[20] Chin-Yeu Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summariza- fion Branches Out, pages 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics.
[21] Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A. Smith, and Yejin Choi. DExperts: Decoding-time controlled text generation with experts and anti-experts. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Vol- ume 1: Long Papers), pages 6691-6706, Online, August 2021. Association for Computational Linguistics.
[22] Ximing Lu, Sean Welleck, Liwei Jiang, Jack Hessel, Lianhui Qin. Peter West, Prithviraj Ammanabrolu, and Yejin Choi. Quark: Controllable text generation with reinforced unlearning. In Thirty-sixth Conference on Neural Information Processing Systems (NeurIPS), 2022.
[23] Aman Madaan, Niket Tandon, Peter Clark, and Yiming Yang. Memory-assisted prompt editing improve GPT-3 after deployment. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages (833-2861) Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
[24] Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chad- wick, Mia Glaese, Susannah Young, Lucy Campbell-Gillinghannl Geoffrey Irving, et al. Teaching language models to support answers with verified quotes. arXiv preprint arXiv:2203. 11177 2022.
[25] Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannanch Hajishirzi. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. arXiv preprint arXiv:2305. 2023.
[26] Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. AmbigQA: Answering ambiguous open-domain questions. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 5783-5797, Online, November 2020. Association for Computational Linguistics.
12

---

[27]
Reiichiro Nakano, Jacob Hilton, S. Arun Balaji, Jeff Wu, Long Ouyang. Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Wehgpt: Browser-assisted question-answering with human feedback. ArXiv, abs/2112.09332. 2021.
[28] OpenAl. Gpt-4 technical report, 2023.
[29]
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022.
[30] Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beatriz Borges, Antoine Bosselut, Robert West, and Boi Faltings. Refiner: Reasoning feedback on intermediate representations, 2023.
[31] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodci, and Ilya Sutskever. Language models unsupervised multitask learners. 2019.
[32] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, (1(1::545555555 2020.
[33] Rajkumar Ramamurthy*, Prithviraj Ammanabrolu Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization. In International Conference on Learning Representations (ICLR), 2023.
[34] Jérémy Scheurer, Jon Ander Campos, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. Training language models with natural language feedback. arXiv preprint arXiv:2204. 14146, 2022.
[35] Jérémy Scheurer, Jon Ander Campos, Tomasz Korbak, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. Training language models with language feedback at scale, 2023.
[36] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. In Proceedings of the International Conference on Learning Representations (ICLR), 2016.
[37] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 17Xiv:1707.06347, 2017.
[38] Weiyan Shi, Emily Dinan, Kurt Shuster, Jason Weston, and Jing Xu. When life gives you lemons, make cherryade: Converting feedback from bad responses into good labels. arXiv preprint arXiv:22100 15893, 2022.
[39] Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming- Chang. ASQA: Factoid questions meet forg-form answers. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8273-8288, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
[40] Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan J. Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. ArXiv, abs/2009.01322 2020.
[41] Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. Recursively summarizing books with human feedback. arXiv preprint arXiv:2109. 10862, 2021.
[42] Jing Xu, Megan Ung, Mojtaba Komeili, Kushal Arora, Y-Lan Boureau, and Jason Weston. Learning new skills after deployment: Improving open-domain internet-driven dialogue with human feedback. arXiv preprint mrXiv:2208.032700 2022.
13

---

[43] Wenda Xu, Danqing Wang, Liangming Pan, Zhenqiao Song, Markus Freitag, William Yang Wang, and Lei Li. Instructscore: Towards explainable generation evaluation with automatic feedback. arXiv preprint arXiv:2305. 14282, 2023.
[44] Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback without tears, 2023.
[45] Xiang Yue, Boshi Wang, Kai Zhang, Ziru Chen, Yu Su, and Huan Sun. Automatic evaluation of attribution by large language models. arXiv preprint arXiv:2305.0631/ 2023.
[46] Daniel M. Ziegler, Nisan Stiennon, Jeff Wu, Tom B. Brown, Alec Radford. Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. ArXiv, abs/1909.08593, 2019.
14

---

Appendices
Qualitative Examples for Long-Form QA
16
A.I
Examples on LM Customization
16
A.2 Examples on LM Errors
17
A.3 Examples on Reward Model Ablation
19
A.4
Comparison with ChatGPT responses
19
B
Algorithm and Training Details of FINE-GRAING RLHF
21
PPO with Fine-Grained Rewards
B.2
Implementation Details
21
Long-Form QA Data and Human Feedback Annotation
22
Data Construction
22
Human Feedback Annotation
22
Analysis of Collected Fine-Grrined Feedback
23
Long-Form QA Reward Model Training Details
23
15

---

A Qualitative Examples for Long-Form QA
A.1 Examples on LM Customization
As discussed in $ 4.5 we can modify the weight of each fine-grained reward model during RL training to get LM with different behaviors. Here, we explore three configurations of reward model weights and name them "short', 'medium', and 'long' based on the LM's average generation length. The 'short' configuration generates concise and short responses, while the 'long' configuration generates detailed and long responses. Table djemonstrates the different behaviors of our customized LMs. Given the same question, each LM generates different amount of auxiliary information in the response.
Question:
When did the French join revolution on colonists' side?
Passages:
Article Title: France in the American Revolutionary War
French involvement in the American Revolutionary War began in 1775, when France, rival of the British Empire. secretly shipped supplies to the Continental Army. A 'Treaty of Alliance in 1778 soon followed, which led to shipments of money and material to the United States, Subsequently. the Spanish Empire and the Dutch Republic also began send assistance, leaving the British Empise with no allies. Frunce's help is considered vital and decisive contribation to the United States' victory against the British. in deht. After its defeat in the Seven Years' War in 1763, France lost its vast holdings in North America. Meanwhile, the American colonists and the British government began to fight over whether Parliament in London or the colonial assemblies had primary responsibility for taxation. As part of that conflict, the colonists organized the Boston Tea Party in response to a tax on lea. The British government responded by passing the Intolerable Acts, which included the closing of Boston Harbor and the revocation Massachusett'ss 's colonial charter. This conflict exacerbated tensions further. The ideological conffict escalated inlo upen warfare in 1775, al which point the American patriols revolted against British rule. France, who had been rehuilding their Navy and other forces, saw this as an opportunity to seriously weaken her perennial enemy.
cost participation in war, France accumulated over billion livres
Article Title: France in the American Revolutionary War France bitterly resented its loss the Seven Years' War and sought revenge. It also wanted to strategically weaken Britain. Follewing the Declaration Independence, the American Revolution was well received by both the general population and the aristocracy in France. The Revolation was perceived as the incarnation the Enlightenment Spirit against the "English tyranny." Benjamin Franklin traveled to France in December 1776 in order to rally the nation's 's support, and he was military aid (predominantly gunpowder) through a company called 'Rodrigue Hortalez et Compagnic' beginning in the spring of 1776. Estimates place the percentage of French supplied arms to the Americans in the Saratoga campaign at up 90%. By 1777, glory in battle or animated by the sincere ideals of liberty and republicanism, volunteers like Pierre Charles 1'Fnfann joined the American army. The most famous was Lafayetle, a charming young aristucral defied the king's 's order and enlisted in 1777 at age 20. He became an aide tn Genrge Washington and a combat general. More importantly, he solidified a favorable American view of France.
with great enthusiasm. At first, French support was covert: French agents sent Patriots
five million "livres" of aid had been sent the American rebels. Motivated by the prospect of
Article Title: Treaty of Alliance (1778)
On March 17. 1778, four days after a French ambassadur informed the British government that France had officially recognized the United States
independent nation with the signing of the Treaty of Amity and Commerce and the Treaty of Alliance, Great Britain declared war on France, thereby engaging that nation in the American Revolutionary War. French entry into the war would lead further escalation of the war when Spain entered the fight against Britain as France's ally, after the signing of the Treaty of Aranjuez on April 12, 1779, and again in December 1780 when Britain doclared war on the Dutch Republic after scizing a Dutch merchant ship they claimed was carrying contrabund France during the Affair of Fielding and Baland After the signing of the treaty French supplies of arms, ammunition, and uniforms proved vital for the Continental Army while their actions in the Anglo French War 1778 83 the West Indies and elsewhere forced Britain to redeploy troops and naval units away from the North American colonies in their boldings.
L.M outputs with different reward model configurations:
Short:
French involvement in the American Revolutionary War began in 1 1775. when France, a rival of the British Empire. secretly shipped supplies to the Continental Army. Theaty Alliance in 1778 soon followed, which shipments of money and matériel to the United States. Subsequently, the Spanish Empire and the Dutch Republic also began send assislance, leaving the British Empize with no allies.
Medium:
French involvement in the American Revolutionary War began in 1775, when France, a rival of the British Empire, socretly shipped supplics to the Continental Army. Treaty of Alliance in 1778 soon followed, which led shipments of money and matériel to the United States, Subsequently. the Spanish Empire and the Dutch Republic also began to send assistance, leaving the British Empire with allies. March 17, 1778, four days informed the British government that France had officially recognized the United States as an independent nation, Great Britain declared war on France, thereby engaging that nation in the American Revolutionary War.
French amhassador
Long:
French involvement in the American Revolutionary War began in 1775. when France, a rival of the British Empire, secretly shipped supplies to the Continental Army. Theaty of Alliance in 1778 soon followed, which of money and matériel to the United States, Subsequently. the Spanish Empire and the Dutch Republic also began to March 17, 1778, four days
shipments
assistance, leaving the British Empire with no allies. informed the British guvernment that France had officially necognized the United States independent nation with the signing of the Treaty of Amity and Commerce and the Treaty of Alliance, Great Britain declured war on France, thereby engaging that nation in the American Revolutionary War.
French amhassador
Table 6: Qualitative examples on LM customization.
16

---

A.2 Examples on LM Errors
Table 7 and Table 8 show examples of LM outputs from all the compared systems (SFT, Pref. RLHF, and
RLHF). We mark the fine ane-grained errors on the model outputs. Overall, our TINE-GRAINEL RLHF outperforms SFT and Pref. RLHF in all three error types.
Question:
Who was the prime minister of Canada in 1920?
Passages:
Article Title: Robert Borden
Not only did Borden's is persistence allow him to represent Canada Paris as nation, it also ensured that each of the dominions could sign the Treaty of Versailles its own right, and receive sepurale membership the League of Nations. During the conference Borden tried to act as an intermediary hetween the United States and other members of the British Empire delegation, particularly Australia and New Zealand discussed with Lloyd George, the possibility of Canada taking over the administration of Belize and the West Indies, but no agreement was reached. Borden's insistence, the treaty was ratified by the Canadian Parliament. Borden was last Prime Minister to be knighted after the House of Commons indicated its desire for the discontinuation of the granting of any future titles to Canadians 1919 with the adoption of the Nickle Resolution. In 1919 Borden approved the use of troogs pul down the Winnipeg general strike, which Soviet Union, Sir Robent Borien retired from office in 1920. He was the Chancellor of Queen's 's University from 1924 to 1930 and also Chancellor of McGill University from 1918 1920 while still Prime Minister. Borden also served as Vice-President of The Champlain Society between 1923 and 1925. He was the Society's 's first Honorary President between 1925 and 1938.
the issue of Mandates, Borden also
feeured to be the result of Bolshevik
from the
Article Title: Robert Borden
Sir Robert Laird Borden. (June 26, 1854 - June 10, 1937) was a Canadian lawyer and politician who served as the eighth Prime Minister of Canada, in office from 1911 to 1920. He best known for his leadership of Canada during World War I. Borden was born in Grand-Pré Pré, Nova Scotin. He worked as schoolteacher for period and then served his articles of elerkship at a Halifax law firm, He called barristers. Borden was elected to the House of Commons of Canada in 1896, representing the Conservative Party. He replaced Charles Tupper as party leader in 1901, and became prime minister after the party's 's victory at the 1911 foderal election. As prime minister. Borden led Canada through World War I and its immediate aftermath. His government passed the "War Measures Act", created the Canadian Expeditionary Force, and eventually introduced compulsory military service, which sparked the 1917 conscription crisis, On the home front, it dealt with the consequences of the Halifax Explosion. introduced women's 's suffrage for federal elections, and used the North-West Mounted Police to break up the 1919 Winnipeg general strike.
bar in 1878, and soon became one of Nova Scotia's most prominent
Article Title: Arthur Meighen
Arthur Meighen Arthur Meighen June 1874 5 August 1960) was Canadian lawyer and politician who served the ninth Prime Minister Canada, in office from July 1920 December 1921 and again from June to September 1926. He led the Conservative Party from 1920 to 1926 and from 1941 to 1942. Meighen was born in rural Perth County, Ontario. He studied mathematics the University of Toronto, and then After qualifying to practice law, he moved to Portage la Prairic. Manitoba. Meighen entered the House of Commons Canada 1908, aged and in 1913 was appointed to the Cabinet of Prime Minister Sir Robert Berden. Under Borden, Meighen served as Solicitor General (1913 1917). Secretary of State for Canada (1917), Minister of Mines (1917: 1919-1920), Minister of the Interior (1917-1920; and Superintendent--eene 1920, Meighen succeeded Borden Conservative leader and Prime Minister the first born after Confederation, and the only one from a Manitohu riding. He suffered heavy defeat in the 1921 election to Mackenzie King and the Liberal Party, but re-entered Parliament through 1922 by-election and remained as Opposition Lender.
to Osgoode Hall Law School.
Indian Affairs (1917-1920). In
I.M outputs
SFT:
Sir Robert Borden was eighth Prime Minister of Canada from 1911 to 1920. Arthur Meighen the ninth Prime Minister of Canada from 1920 to 1926.
Pref. RLHF:
Sir Robert Laird Borden was Canadian lawyer and politician who served eight Prime Minister of Canada, office from 191 to 1920. Anthur Meighen was Canadian lawyer and politician who served as the ninth Prime Minister Canada, office from July to December 1921 and again from June September 1926. He led Conservative Party from 1920 to 1926 and from 1941 to 1942.
FINE-
Sir Robert Laird Borden was Canadian lawyer and politician who served eighth Prime Minister of Canada, office from 1911 to 1920, Anthur Meighen of Canada, in office from July 1920 to December 1921 and again from June to September 1926.
GRAINED RL.F
Canadian lawyer and politician who served the ninth Prime Minister
Table 7: Examples of errors in LM outputs of SFT, Pref. RLHE and our mark the C₁ (irrelevance, repetition, incoherence) and C2 (incorrect or unverifiable facts) types of errors in the model outputs.
RLIIE, We
17

---

Question:
infraspinatus museles of beef, and may this displayed roast. Anatomically, the musele forms the dorsal part of the rotator cuff of the stoer. This cut is anatomically distinct steaks usually have significant Where does flat
from?
Passages:
Article Title: Flat iron steak
Flat iron steak (US), butlers' steak (UK), oyster blade steak (Australia and New Zealand) is a cut of steak cut with the grain from the shoulder
animal. This produces flavorful fascia membrane unless removed. Some restaurants offer it on their menu, aften at lower price than the more popular rib-eye and strip steaks the same animal, for example Kobe beef. This of steak is from the shoulder of a beef animal. It is located adjacent to the heart of the shoulder clod, under the seven or paddle bone (shoulder blade or scapula). The steak encompasee the 'top blade'
tough because contains gristly
same grade. This used, some places, as means selling
expensive cut from
butcher shops and
markets
the shoulder lender, amount of marhling.
directly below and the mujor.
LM outputs
SFT:
Flat iron steak is cut of steak cut with the grain from the shoulder of the animal. This produces flavorful cut that is bit tough because contains gristly fascia membrane unless removed, Some restaurants offer it on their menu, often lower price than the more popular rib-eye and strip steaks of the same grade. This used, in some places, as a means of selling less expensive cut from same animal, example Kobe beef.
Pref. RLHE:
Flat iron steak is cut of steak cut with the grain from the shoulder of the animal. This produces flavorful cut that is bit lough because contains gristly fascia membrane unless removed. Some restaurants ofler on their menu, often lower price than the more popular rib-eye and strip steaks selling expensive cut from
same grade. This used, in some places, as a means of animal, example Kobe beef.
FINE-
Flat iron steak cut of steak cut with the grain from the shoulder of the animal. This is located adjacent the heart the shoulder clod, under the seven paddle bone (shoulder blade scapuls). This steak encompasses the infraspinatus muscles heef, and one may see this di displayed in some butcher shops and meat markets as a "top hlade roast.
GRAINED RE.F
Table 8: Qualitative examples on another question. Here we highlight the information in blue that is covered by the LMM trained with FINE-GRAINED RLHF but missed C3: incomplete information eиror) by SFT and Pref.RLHE
18

---

A.3 Examples on Reward Model Ablation
As discussed in $ 4.6 reward models are competing against each other, and we experiment with removing one of the three reward models during RL training. Table shows an example of how LMs behave such scenarios. See 4.6 for our observations.
Question:
How much does it cost to raise the Costa Concordia?
Passages:
Article Title: Costa Pacifica
"Concordia" -llas cruise ship for Costa Crociere. She was banded over Costa Crociere on 29 May 2009. Her sister ships, "Costa Concortia"" and "Custa Serena", were launched in 2006 and in 2007, with "Cosia Favolosa" and "Costa Fascinosa" launched and PlayStation World, aren boasting PS3 consoles, and the first recording studio at sea. "Costa Pacifica" is the third ship of the "Concordia" class, preceded by sister ships "Costa Concordia" and "Costa Serema" and was followed by "Costa Favolosa" and "Costa Fascinosa" investment of 2.4 billion Euro, and is currently the largest flect expansion program in the world. "Costa Pacifica" was December 2005, by Carnival Corporation. the parent company of Costa Crociere. The order was four ships, which includes; "Camival Dream", for Carnival Cruise Lines; "Nieuw Amstendam" for Holland America Line; and 'Ruby Princess" Princess Cruises, Article Title: Costa Concordia disaster
Costa Pacifica
and 2012 respectively. Onboard facilities inchude the Samsara Spa
2011 and 2012, all part of the expansion program of Costa which entailed an
ordered
total cost
$3 billion.
On 13 Lanuary 2012, the Italian cruise ship "Costa Concordia" ran aground and overturned after striking an underwater rock off Isola del Giglio, Tuscany, resulting 32 deaths. The eight year old Costa Cruises vessel cruise around the Mediterancaan Sea when she deviated from her planned route at the Isola del Giglio, sailed closer to the island, and struck a ruck formation brought ashore. "Costa Concordia" was officially declared a 'constructive total loss" by the insurance company, and her salvage was "one of biggest maritime salvage operations". On September 2013, the parbuckle salvage began, and by the early hours of 17 September 2013, the ship was upright on its underwater cradie. In July 2014, the ship was refloated by large sponsons (metal lanks) welded to its sides and was towed 320 kilometres (200 miles) to its home port of Genoa for scrapping which finished in July 2017. The total of the disaster, including victims" compensation. reiloating, towing and scrapping costs, is estimated at approximately $2 billion, more than three times the $612 million construction cost the ship.
the first leg of
sea floor. six-hour rescue effort resulted most of the passengers being
ship
LM outputs:
removing one fine-grained reward model during training
All reward models:
Costa Concordia is a cruise ship for Costa Crociere. 13 January 2012, the Italian cruise ship Costa Concordia ran aground and overturned after striking an underwater rock off Isola del Giglio, Tuscuny, resulting in 32 deaths. The total cost of the disaster, including victims' compensation, refloating, towing and scrapping costs, is estimated at approximately $2 billion, more than three times the $612 million constraction cost of the
Without
On 13 January 2012, the Italian cruise ship Costa Concordia ran aground and overturned after striking an under mek off Isola del Giglio, Tuscany, resulting in 32 deaths. OO 16 September 2013, the parbuckle salvage of the ship began, and by the early hours of 17 Seprember 2013, the ship was set upright on its underwater cradle. In July 2014, the ship was relloated by large sponsons (metal lanks welded to its sides and was lowed 320 kilometres (200 miles) its home port of Genon for scrapping which was finished in July 2017.] The total cost of the disaster. including victims' is estimated approximately $2 billion, more than
relevance
RM R
compensation, refloating, reflosting, towing and scrapping three times the 5612 million construction cod of the ship. The total cost of the disuster, including victims' comperssation, refloating, towing and scrapping costs, estimated at approximately billion. Comment: Too much irrelevant information repetition
Without factuality Rsg
The cost raise the Costa Concordia was estimated approximately $2 billion, , more than three times the $612 million construction cost of the ship.
Comment: Factuality emor. Direct answer without any supporting evidence from the passages.
Without
Costa Concordia "Concordia" cllass cruise ship for Costa Crociere. She was handed over to Costa Crociere on 29 May 2009. The total cost of the disaster, including victims' compensation, refloating, lowing and scrapping costs, estimated at approximately $2 billion, more than three times the 5612 million construction cost of the ship.
info. com-
pleteness
RM Roa
Comment: Miss auxiliary information about the disaster..
Table 9: Qualitative examples on the reward model ablation experiment, We show the LM outputs when we use all the reward models and when one of the reward models is removed during training. We also mark the errors in each output.
A.4 Comparison with ChatGPT responses
We compare the responses generated by ChatGPT (one-shot F and our system in Table 10 As discussed in $ $ We find that ChatGPT responses are relevant and factual, yet lack the auxiliary information to answer the ambiguous questions. This shows that it is challenging for ChatGPT to leam user-desired behaviors through prompting and in-context learning.
$ 'Since the input for each example is very long, we cannot fit more than one in-context example into the model.
19

---

Question:
When did the Rams go to St Louis?
Passages:
Article Title: History of the Los Angeles Rams The Los Angeles Rams
professional American football team that play in the National Football League (NFL). The Rams franchise was founded in 1936 as the Cleveland Rams in the short lived secomd American Foothall League before joining the NEL next year. In 1946, metro area until 1994, when they moved to St. Louis, and were known as the St. Louis Rans from 1995 to 2015. The Rams frrnchhie returned Los Angeles in 2016. This article chronicles the franchise's history during their time Angeles, from playing at the Los Angeles Memorial Coliscum between 1946 and 1979, to playing at Anaheim Stadium (now known as Angel Stadium of Anaheim) in Anaheim from 1980 1994, and its return to Southern California begianing with the season. On January 12, 1946, Dan Reeves was denied request by the other National Football League (NFL. owners to move his team, the Cleveland Rams to Los Angeles and the then 103,000 seat Los Angeles Memorial Coliseum. Reeves threatened to end his relationship with the NEL and get of the professional football business altogether unless the Rams transfer to Los Angeles was permilted. A settlement was reached and, as result, Reeves was allowed to move team Los Angeles.
franchise moved to Los Angeles. The Rams franchise remained in the
Article Title: History of the St. Louis Rams
The professional American football franchise now known as the Los Angeles Rams played in St. Louis, Missouri, as the St. Louis Rams from the 1995 through the 2015 scasons before relocaling back to Los Angeles where the leam had played from the 1946 season in the 1994 season. The Rams franchise relocated from Los Angeles to St. Louis in 1995, which had been without National Fuotball League (NFL) leam since the Cardinals moved to Phoenix, Arizona in 1988. The Rams" first home game in St. Louis was at Busch Memorial Stadium against the New Orleans Saints on September 10, 1995, before the Trans World Dome (later the Edward Jones Dome, and now known as The Dome at America's Center) was completed for their November 12 game against the Carolina Ponthers. Their last game played the Edward Jones Dome in St. Louis was against the Tampa Bay Buccaneers on December 17, 2015, which they won, 31-23, The St. Louis based club was January 3, 2016, against the San Francisco 49ers Levi's 's Stadium, where they lost in overtime 19-16.
Rams' last game
Comparison between our output and ChatGPT output
ChatGPT:
The Rams moved to St. Louis in 1995 and were known as the St. Louis Rams from 1995 to 2015.
FINE- GRAINED RLHF
The Los Angeles Rams are professional American football team that play in the National Football League (NFL). The Rams franchise was founded in 1936 as the Cleveland Rams in the short-lived second American Football League before juining the NFL the next year. In 1946, the franchise moved to Los Angeles, The Rams franchise remained in the metro area until 1994, when they moved to St. Louis, and were known as the St. Louis Rams from 1995 to 2015. The Rams frunchise returned Los Angeles in 2016.
Question:
Who sang the song I'm Coming Out?
Passages:
Article Title: I'm Coming Out
There's ' also been cover by Marcia Hines, It was featured as a cover for the video game "Karnoke Revolution" as well. In more recent times, the song was sampled in Keyshia Cole's remake of 'I'm Coming Out' (featuring Iggy Azalea) for the original soundtrack of the 2014 movie The Other Woman, starring Cameron Diaz, Kate Upton and Leslie Mann. The song is sampled in "Break Your Heart Right Back" from Ariana Grande's sophomore album "My Everything" (2014). This song was also covered for the soundirack to the 2016 movie "Trolls.. In 1997, German comedian Hella Sinmen sampled "I'm Coming Out" for her single "Mein Cornngg Out", using the term "coming out" in the context of coming out of the closet, revealing one's homosexullity In 2003, American singer Amerie covered 'I 'I'm Coming Out' for the soundtrack to the film 'Maid in Manhattan", Her version was released as a single in select European countrics and Australia.
Loren Dawson remix. The original single version (which is similar to the original) can be found on both CD singles and the original motion picture soundtrack to "Maid in Manhattan", Ross' version appears on the film's is soundtrack alhum well. Article Title: I'm Coming Out
final single version
released
"I'm Coming Out" is a song recorded hy American singer Diana Ross. It was written and produced hy Chic members Bernard Edwards and Nile Rodgers, and released in August 22, 1980 the second single from Ross" self-titled tenth album "Diana" (1980). In 1979. Ross commissioned Chic founders Nile Rodgers and Bemard Edwards to create material for new album after taking daughttes see the band concert. following & Simpson composed and produced I.P.' The Boss, "Rudgers got the idea for I'm Coming Out" after noticing three different drag queens dressed as Diana Ross at New York club called the GG Barnum Room. The lyrics hold an adkitional meaning to Ross, as she was leaving Molown Records and "cominn out" from under Berry Gordy" 's thumb. According to the BBC4 documentary "How to Make It in the Music Business", Ross loved the record until she put it front Frankic Crocker, who pointed out that 'm'm Coming Out' what homosexuals use announce their homosexuality; she ran back to the studio in tears and demanded to know why Rodgers wanted to ruin her career. The song was amother hit from album, peaking at number five on U.S.
success Ross's final Ashford
Comparison between our output and ChatGPT output
ChatGPT:
The song "I'm Coming Out" was recorded by American singer Diana Ross. was written and produced by Chic members Beraard Edwards and Nile Rodgers, and released in August 22, 1980 as the second single from Ross' self-titled tenth album "Diana" (1980).
FINE- GRAINED RL.F
"I'm Coming Out" is song recorded by American singer Diana Ross, It was written and produced by Chic members Berard Edwards and Nile Rudgers, and released in August 22, 1980 second single from Ross' self-titled tenth album Diana. In 2003, American singer Amerie covered "I'm Coming Out" for the soundtrack to the film Maid in Manhattan. Her version was released as single select European countries and Australia, The final single version to be released was the Loren Dawson remix.
Table 10: Comparison between ChatGPT outputs and
RLHF outputs.
20

---

B Algorithm and Training Details of FINE-GRAINED RLHF
B.1 PPO with Fine-Grained Rewards
The algorithm below shows in detail how PPO updates the policy LM Pe and the value model Vi with K fine-grained reward models Rsc.
Algorithm 1 Grainer Reinforcement Learning from Human Feedback GRINNED RLHF)
Input initial policy model P initial value model Verai K reward models Rø, trained from human feedback; task prompts D; hyperparameters 7, A, 6,
S
1: policy model Po Pa... value model Vy Vihn
2: for step 1, M do
3: Sample batch D, from D
4: Sample output sequence y" ~ Po x²)) for each prompt x² E 'D,
|w"| for each sampled output y" by running Rø, 5: Compute rewards {r7 Jt=1
Eq.
6: Compute advantages {A,
yn" and value targets { (s:)) for each y" with V
7:
for PPO iteration ..... f Update the policy model by maximizing the PPO clipped surrogate objective:
8:
1
|2
Σ
Σ min
lv'l
( (ac st) As, clip(u,, 1- 1 E)A,) Po.. (ar s,)
< arg max
[D&
9: Update the value model by minimizing square-error objective:
Del
1v"1 (Vo(s)) - (s)))
1
s -
arg min
Dol
Output Pg
B.2
Implementation Details
Model architectures. For the detoxification experiments, the policy model is initialized with GPT2- large [31], and the value model is initialized with GPT2-base. For the long-form QA experiments, the policy model is initialized with a supervised fine-tuned T5-large (32), and the value model initialized with T5-base. This design follows InstructGPT 29 which uses larger (175B) policy model, and smaller value and reward (6B) models.
Training details on detoxification. For both the holistic reward baseline and the sentence-level (fine-grained) reward, we do hyper-parameter search with the same set of hyper-parametert For training, we run 200K episodes. The batch size (number of episodes per card during training) is 64. We use Adam optimizer with a linear learning rate scheduler and 10 warmup steps. We perform a hyper-paramett grid-search for peak learning rate E {5e 6, 1e 5, 2e - 5}, KL coefficient E {0.1,, 0.3). discounting factor À E (0.95, 0.97, 0.99). and the frequency of exploration (number of sampled outputs) E {2, 4, 8J. We find that the higher the KL coefficient, the lower the perplexity, and the higher toxicity. This is consistent with findings from previous RLHF studies ([2))
33 ). For a fair comparison, we eventually choose set of parameters that achieve similar level of perplexity for both reward models. The optimal set of hyper- parerreraeeters for holistic reward is = 0.3, À 0.99. For sentence-level reward B 0.1, 0.95. The learning rate is 1e 5, and the exploration frequency is for both experiments. We choose the checkpoint with the lowest validation set toxicity for evaluation. Regarding computation time, we use 2x 80G NVIDIA A100 GPU for training, and the run time is about 22 hours.
Training details on long-form QA. We conduct a similar hyper- parametrt grid search as our detoxification experiments. For long-Form QA, the input length limit 1024, and the output length limit is 200. Notice that this is much longer than detoxification, so we use smaller batch size and fewer training episodes. We experiment with multiple combinations of reward model weights. Fixing W2 0.5 (factuality reward weight), we perform grid search W1, W3 e [0.0, 0.5].
21

---

We eventually choose w1 =0.3, = 0.5, u3 = 0.3, which reaches a balance between three reward models and allows all three rewards to increase during training. For training, the batch size (number of episodes per card during training) is 32. We use Adam optimizer with a linear learning rate scheduler and 100 warmup steps. We perform a hyper-parameter grid-search for peak learning rate {5e 6, l 5, 2e 5). KL coefficient 3 E {0.1, 0.2, 0.3), discounting factor E {0.95, 0.97, 0.99), and the frequency of exploration E (2,4,8) The optimal set of hyper- parameters for Pref. RLHF is 3 0.2, A 0.99. For FINE-GRAIN
RLIIF, 3 0.3, À = 0.95. The learning rate is le - 5, and the exploration frequency is for both experiments. we run 80K episodes, which is approximately 5 epochs. We choose the checkpoint with the highest validation reward for evaluation. Regarding computation time, we use 2x 80G NVIDIA A100 GPU for training, and the run time is about 15 hours.
note on the error bars. All results we report in the paper are from independent runs. The scores reported are all averaged across all runs. The error bars are represented as the shades behind each training curve in our figures. It shows the standard error across three runs.
C Long-Form QA Data and Human Feedback Annotation
C.1 Data Construction
ASQA 39 a long-form QA dataset that focuses on answering ambiguous factoid questions in an open-domain setting that requires passage retrieval from a given Wikipedia passage corpus. We reformulate it into a reading comprehension setting: given the input æ that contains a question g and set of knowledge passages P = {p1...., P|r(), J. generate a long-form response y. To construct P for each input æ, we use the oracle knowledge contexts provided by ASQA for each æ, that are text snippets from the passage corpus. We BM2 to map each knowledge context (text snippet) to the closest passage from the passage corpus and use the resulting passages as P. Our train and dev examples come from the original ASQA train set and our test examples are the original ASQA dev examples.
C.2 Human Feedback Annotation
Fine-grained feedback. As discussed in $ 4.1 we first use IK randomly sampled training examples to train T5-large based supervised model SFT as the initial policy model Pa..... Then we collect feedback on sampled outputs from SFT for the remaining 2.853 training examples and the 500 development examples, using the Amazon Machanical Turk platform
Figure shows the fine-grained human feedback annotation interface with an example from QA- FEEDBACK. In addition to the task upput-question q and oracle passages P, we also provide human-written response from ASQA to the worker as reference. However, it is important to note that, in practice, the annotation of our fine-grained feedback should not require the human-written response. The only purpose for us to provide the gold response is to have our workers follow the same question interpretation and expected response of the workers who annotate for ASQA, such that our experimental comparison with supervised models (SFT and SFT-Full; details in & 4.3 is fair. However, we still instruct our workers to strictly use the given passages for checking factual errors. For each span error, we ask the worker to select one out of 5 categories shown in Figure 6 (left) b However, we collapse these 5 categories into two categories (C₁ and C2 mentioned in $ based on whether the eтror detection depends on the passages or not. When workers mark passage sentences as containing missing information, we instruct them to categorize each sentence as missing "answer", "major auxiliary information" or "minor auxiliary information," as shown in Figure (right). Our instruction to the worker is provided in Figure
Quality control. Before feedback collection, we design a qualification task to select qualified workers for this feedback annotation task. The qualification task consists of 5 questions with their corresponding passages and model outputs for the workers to annotate. We manually review about
https:// www.k. com/ We see very few "incoherence" errors (1%), so the majority of labeled errors are from the other four categories during annotation.
22

---

70 submissions of the qualification task and select 15 workers whose annotation is marked by us as of high quality. Throughout the actual feedback annotation process, we constantly monitor the annotated data and send feedback to workers.
Preference-based feedback. For comparison purposes, follow (29 to collect pairwise human preferences from the same group of workers we select from the qualification task. We sample four model outputs for each prompt v, which gives pairs of model outputs. Similarly, we provide the worker with the human-written response and ask the workers to indicate pairwise preferences (ties are allowed) based on all errors they can find each model output. Figure 7 shows the preference- based human feedback annotation interface with an example from QA DA-FEEDBACK.
preference feedback. If the example consists of 2 3 passages to read, we assign an additional $0.3 bonus to the example. On average, we pay roughly $1.65 per example for both tasks, which gives an $16.5 hourly pay for our workers.
Pay structure. We pay base rate of $1.5 per example for annotating fine-grained
C.3 Analysis of Collected Fine-Grained Feedback
Overall, among all error spans we collect, 76% of them are C1 errors and the remaining 24% are C2 errors. However, it is important to note that we instruct workers to label C2 errors only at places that don't have C1 error. 75% examples are labeled as being incomplete; i.e., containing missing information that can be found in the given passages (Cz). Among all marked passage sentences that contain missing information, 31 %,, 42% and 27% are missing answer, major auxiliary information and minor auxiliary information respectively.
To analyze human-human agreement, subset of 300 examples receive annotations from two distinct workers. We observe that while the exact agreement in error span boundaries is low, workers achieve reasonably high agreement on whether a sub-sentence contains C1 (reach an agreement for 83% of all sub-sentences) and whether sentence contains C2 (92%). The agreement on whether a model output contains complete information or not (C3) is 85%. Therefore, we decide to have the density for error type C1, C2. and C3 as sub-sentence, sentence and full sequence.
D Long-Form QA Reward Model Training Details
We train reward models with the 2,835 training examples with feedback collected and select the best model for each error category based on the their performance on the development set. The batch size and training epochs are 24 and 50 for RS, and R Each training is on single 80G NVIDIA A100 GPU, taking and hours for training R&g, and Rs, respectively The batch size and training epochs are 12 (per GPU) and 30 for Rø, and the preference-
reward model. Each training is run on 2 80G NVIDIA A100 GPU and takes hours. We use Adam optimizer with a linear learning rate scheduler for all reward model training. For each reward model, we search the learning rate over (5e-6, 15 50- weight decay over {0.001, 0.01}, and warm-up step ratio over {0.1, 0.2) based on the dev set performance. Specifically, we use the model for Rø, and R&s, that achieve the best binary classification accuracy. For Rog and the preference- that achieves the best pairwise comparison accuracy. We also provide more training details for each reward model below.
reward model, we select the model
RS, for C₁: Irrelevance, repetition, or incoherence. To train the reward model Rs, that detects error of irrlevance, repetition, incoherence, we apply token-level classification loss to each [sep] token before w/. where its gold label gj is "has error" if there is a fr E FF that has as....... overlapped with v and c; = 1. and "no error" otherwise. We observe that most of the spans marked as eгтor type C₁ that are shorter than 5 words usually carry very little information or are annotated as a result of workers being very careful or strict. Therefore, we filter out such short spans before constructing training examples for Rø,, Overall, we get 7379 and 8059 sub-sentences with the "has error" and "no error" label respectively.
We use spaCy [15] to segment generated model outputs into sentences. We then split sentences into sub-sentences using a comma or semicolon. Note that training Ro, takes shorter time as its input does not contain passages.
23

---

RRz, for C2: Incorrect or unverifiable facts. We train Røs in a similar way as how we train Rø. Instead of predicting the error for each sub-sentence, R is trained to predict the sentence level (i.e., yf is the j sentence in y). Since workers do not annotate C2 error for spans that are already labeled as having C1 error, in order to avoid false negatives in training Røy. we do not provide gold label nor calculate loss for sentences that only contain C1 error from training. sentences that contain C2 error has the gold label "has error" and sentences that contain no enror has the gold label "no error". Overall, we get 1600 and 3411 sentences with the "has error" and "no error" label respectively.
other words, all
Rgs. for C3: Incomplete information. Instead of casting this as a classification task, RSg predicts single scalar reward given I and and is trained with a pairwise comparison loss [29]. This is motivated by early work [9 that shows the better reliability of pairwise comparison than error classification when assessing full generation sequence. To construct training data for R we bootstrap pairwise comparisons from the corrected model output y' as follows. We first map each sub-sentence in y to passage sentence in P that has sub-string with the highest token-level F1 score with sub-sentence and denote all mapped sentences as S. We then sample four responses from SFT, for each we do the same sentence mapping to get set of passages sentences S", We calculate score(y) = |S' SSS/SS] as the information completeness score for each model response y. We follow [29] to pair up sampled responses for q and denote each sampled response pair (pp- W.. where score(Up) > score (.t). We drop the pairs where ecore(ūp) = ( Then we follow [29] to train Røg with the loss function in Eq. We have total number of 6821 pair examples in training.
Preference-baeed reward model. The preference- based reward model is trained in similar way as R with Up representing the human preferred response against Yt in the loss function Eq. [ We drop the pairs where tie is indicated. We have a total number of 14981 pair examples in tramiig.
We manually review 50 mapped passage sentences and find over 90% of them are correctly mapped, which indicates frequent extractive behaviors from Po..t.-
24

---

Imput Queation:
wAs parliament formed
Passage Tible (8o) Parfinment of the United Kingllo
Purtarmenyy Thter Art fermaly amended Tarfament Urited stngdor of thrrat Noether Indledd' the ahe woreir
Irihh caperaioo
trepire, Pullianent angged pelliial potera countrion cakansa au whet perfictentayy system.
Hoverns teggroed thegpt ettteteed
fhevy, segrese iegelathv power
powers
only delaying legitationn gotwer
Xingfort neppssdaiity
Canaay merger Kingdome
praactioo.
Responte::
putikanent UK antially trrilament Unived kinglle Gámaat iidand und Marlarmentacy tilke spay fermall anandee name Partiament kingaees Goest Morthere bodand", thee year the secnviee Kinplans Great Northem crented Novenaber s928.
Parfarnent Aingtons Geest Britate
on Juuraay 1801, pailianee Senvel
STRPP i: Iomur labiling for the modei-rredictedd responsec ts
sewin the windde svergt wesii lesappille
After CARSFUT, checking, find no ernor this predieted response!
The Utiitd Kingdor indd Invladt conenated en Jancary sten, by mvrppr Kinglere Giment fritate under
autharastt you Inrreed Londie..
Uaverifable
Brnolmrnn
Nivila Iads
Wilt (fficeee achattes deent ru deabl
[esplanation]] Ppanage
STXP Your cornection the model-groiette grodieted responses Clicsts
The Repal and Pullanestury Vites usz furaaly ancoded name toe Puriaaent the Caited Kingloon Great Bettai and Nortbea Iedlad"" years alter secssian the lish Partomest lnnel Kingknn lrissen Inaaad was crusted lansory 18t..
Figure 5: Fine tine-grained feedback annotation interface.
Selected
the Slngooos Biritats and Trikard enderrtte
Missing Ansurr
Milanigg Major Ausditarr
Missing Minor Austilary
Select the factual (antualllnnggggge.....
You fin pince miissing information Ins
Ierelevant
Repetitive
Inconeletent
Unveetfluble
Pact
Incoberent
passage
sentence
Co cues, lat matogta meteaaoe exparieed with conera (raccemuy)
CONPIRM
Figure 6: Error types in the fine- tne-grained feedback annotation interface.
Your Taak Make painise comperiern felloning modld prodiction bend evenal quality
en dlat (hird pairs siferett on peshlperettisss Wirisitr vonli pevilciin sncligghiii wall color ovsresiiee
Click lade:
Cocppere Model Prediction VB O better Orgaaal es better
Pred es: The United Kegaken Grnat Britain and Inclas wa crusaa on L'maaay stos, by the reregre the Kingdoms Great Brisen and Iniandd under the Acts Union dioon On lanuary 1ños perliument was fiemal in Londie.
United
Kanglanz Irekud United Kingioon Great Norbhra
Pufiienentary
changed Pndianeett
Gotpeee
VS
Repul
Campan Pralicion
lutter
Inebow
Carppon Trudietion -
aqual Cityl Dchytteeeee
Cleck tos Cotapere Model Predccico es e4 O better
equal O belne
Click tonlow
Comperr Model Trodiatio vs 44 betee aqual lebette
Figure 7: Preference feedback annotation interface. The task input is omitted (same as Figure
25

---

each task, you given a question, set Wikipedia passages (with article provided). reference response, and model- predieted response. Each passage is presented as sequence of sentences (title is indexed sentence #o). Your goal 1) label mistakes made in the model prediction and 2) make corrective edits to the model prediction based on your labeled mistakes.
Important Definitions: An ideal response to question should provide both ansuer((s) that directly responds to the question and crucial auxitiory information better comprehension of answer(s). We consider audilary information as crucial it used in the reference response. Additionally, all information the referenes response is written by human with potentially different grounding passages, and thus, you might find ansupers that can be found in the passuges but are not included in the referenne, which are SVIL. expected to be in the madel prediction. On the other hand, answers the model prediction. To conclude, all answerss are
ideal response should factually consistent with (i.e., grounded in) the passages. Nota that
reference that cannot he found or verifiahle by the passages are NOT expected to expected the madel prediction IF AND ONLY IF can be found the passages, Crucial auxiliary information expected in the model prediction IF AND ONLY IF can be found in both the reference response and the passages.
Here the detailed anaotation steps:
STEP
Read the question and label mistakes made in preedce-eedietee response. As explained ahove, leverage the reference, BUT rely the passages. Decide the mistake type and follow detailed instructions as follows. We encourage you to use CTRL-F/CMI F/CMD-F for navigating the reference and passages using keywurds.
Erroneous Span (i.e., substring): Highliigh each span in the model prediction that contains une of the following errors. Label each span as short as possible make each labeled span only contain one single information piece. You be prompted to type und provide explamation if apply. For each span, Jabel only one type. If span contairs multiple erroes, select the erroe type based on the order below (s.g, if span contains "irrelevant information, ignore any 'Inconsistent fact" it contains).
erzor
i. [rreleeantt: The span contains "irrelevant" information (e.g, neither an answer nor crucial suxiliary information, defined in the first sentences 'Impurtant Definitions") ") detect "irrelevant" errons, you do not to consider whether the information is factually consiatent with the passages not.
i. [Repetitive]: The span repeats information in its previous text. Pruvide the previous text being repeated (as short as possible). Note that a "repetätive" span should still be marked even its previous text being repeated contains an factual or coherence issue (defined below). However, the previous text 'irrelevant, should marked as "irrelevant" too.
[Inconsistent Fact]: The span factually incomsistent with the passages. Enter the passage id and sentence id(s) evidence. Note that if you find multiple evidences in the passages, mark only one of them. The need for multiple passage ids usually indicates that you should separate the errot into multiple ones (due to multiple information pieces).
[Unverifiable Fact]: spun factually unverifiable (Le,, not mentioned in any passage), after curefielly checking all passages. Common seuse (e.g.. 'a bicyle has wheels") doesn't need to verified. However, do not count knowledge oaly commoniy known in specific region/ community as commonoense, This can be subjective, and simply follow your best judyment.
v. [Tacoherent]: The span contains major grammar error Ginoore minor typos), uninterpretable, contradicts to common aense, or coherent with cuntext.
Missing Information: Iddettffy information that expected missing the model prediction. Check 'Imprrtnnt Definitions' above how to identify such information. Classify each piece of missing information [Missing Answer] or [Missiig Major/ /Minor Auxiliary Information]. and enter the passage and sentence id(s) as evidence. Mark the missing auxiliiar information as major if you think the information indeed helpful for understanding the answer. Otherwise (e.g., a bit off-topic), mark it as minor. Simply follow your best judgment. Follow same "Note that" rule under 7/nconsistent Fact]" above.
STEP Correct the model prediction to address your labeled mistakes in STEP s. Instead of copylig and pasting the reference into the box, make minimal local edits on the original prediction. Make sure the corrected version fluent and free of typos. In rare cases when you find no usefal information passages answering question, you can leave box as blank if necessary.
Figure 8: Fine- graine feedback annotation instructions.
26


Reviews:
[{"review_id": "lHaJr2wM3T", "reviewer": "Reviewer_tgte", "summary": "This paper introduces FINE-GRAINED RLHF, a framework that enables training and learning from reward functions that are fine-grained in different error categories. Through experiments on detoxification and long-form question answering, the authors demonstrate how learning with this FINE-GRAINED reward function leads to improved performance, supported by both automatic and human evaluation. The paper's contributions include a new framework for fine-grained reward learning, a new dataset for long-form question answering, and experiments demonstrating the effectiveness of the proposed approach. Though, the proposed method still has some limitations in terms of computational cost, algorithm complexity, and task-binding. In all, it has some contributions to the RLHF field.", "questions": "- The authors say that \"the higher the KL coefﬁcient, the lower the perplexity, and the higher toxicity\". But only three KL coefficient is tested (i.e., 0.1, 0.2, 0.3) in the experiments, it is not enough to get such a conclusion. Also, what are the results on the generation fluency when the KL coefficient is 0, which means we do not employ a KL penalty? It would be better to explain the KL penalty in more detail.\n- Why do you choose different models for different tasks? I.e., for the detoxiﬁcation experiments, the policy model is initialized with GPT2Large, and the value model is initialized with GPT2-base; for the long-form QA experiments, the policy model is initialized with a supervised ﬁne-tuned T5-large, and the value model is initialized with T5-base.\n- Is the proposed method a good solution in terms of LLM's hallucination?", "limitations": "- The training dataset is not large enough (especially for the Long-Form QA task), which limits the verification of a conjecture: As the size of training data increases, the effect of the FG-RLHF proposed in this paper will weaken.\n- The proposed method is expensive, not only in the additional computational cost of the reward model, but also in the different deﬁnitions of ﬁne-grained feedback in terms of the feedback types and the density level of each feedback type.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "- The paper introduces a new framework for fine-grained reward learning, which is a novel approach to language model training. The authors also provide a new dataset for long-form question answering, which can be used for future research in this area.\n- The paper is well-written and clearly presents the proposed framework and experimental results. The authors provide detailed descriptions of their methodology and evaluation metrics, which enhances the quality of the paper.\n- The proposed framework has the potential to improve the performance of language models by providing more fine-grained feedback during training.\n- The idea is simple but reveals the potential of RLHF at finer grains.", "weaknesses": "- I know that data labeling is cumbersome and expensive, but the 3,853 training data collected in this work is still a bit small, not sure if this is enough to draw the conclusions mentioned in this work.\n- The proposed method is a bit complicated, it will introduce more computational cost in RLHF, and it will introduce more hyper-parameters for fine-tuning.\n- The design of T_i^k is not fully explored, which may have a big impact on the performance of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments! Please see our responses below.\n\n> ***1. I know that data labeling is cumbersome and expensive, but the 3,853 training data collected in this work is still a bit small, not sure if this is enough to draw the conclusions mentioned in this work.***\n\nThe amount of pairwise comparisons (6 comparisons per prompt) used here is **comparable to that used in other work**, e.g. WebGPT (20k comparisons). More importantly, we believe that it is an advantage to have a method that works well with a small amount of data.\n\n> ***2. The proposed method is a bit complicated, it will introduce more computational cost in RLHF, and it will introduce more hyper-parameters for fine-tuning***\n\nWe use the same RL algorithm (PPO) as preference RLHF, and only observe **very small relative additional cost (~1% training time)** from running multiple reward models, compared to preference RLHF during RL training. Although it introduces more hyperparameters (reward model weights), they also bring the benefit of allowing LM training customization. We will add clarification on this to the next version of the paper.\n\n> ***3. The design of T_i^k is not fully explored, which may have a big impact on the performance of the proposed method.***\n\nThe point of this work is to show that there are many options for rewards, including different types of rewards and different density levels. That space is bigger than can be fully explored in one paper. We hope the proof-of-concept offered in our work inspires more exploration with these and other tasks. \n\n> ***4. The authors say that \"the higher the KL coefﬁcient, the lower the perplexity, and the higher toxicity\". But only three KL coefficient is tested (i.e., 0.1, 0.2, 0.3) in the experiments, it is not enough to get such a conclusion. Also, what are the results on the generation fluency when the KL coefficient is 0, which means we do not employ a KL penalty? It would be better to explain the KL penalty in more detail.***\n\nOur observation **confirms findings from previous RLHF studies** like Ouyang et al., 2022 and Ramamurthy et al., 2023. As a result, we didn't expand more on it. For example, previous work provides extensive empirical evidence showing that the KL term is important for preventing reward hacking and this trend in particular. Therefore, we did not experiment with KL coefficient = 0. We will update the paper to make this point more clear.\n\n\n> ***5. Why do you choose different models for different tasks? I.e., for the detoxiﬁcation experiments, the policy model is initialized with GPT2Large, and the value model is initialized with GPT2-base; for the long-form QA experiments, the policy model is initialized with a supervised ﬁne-tuned T5-large, and the value model is initialized with T5-base.***\n\n​​In order to have **fair comparisons with baseline models** for the two tasks, we chose GPT2-large and T5-large as the policy model for detoxification and long-form QA. In specific, prompts in REALTOXICITYPROMPTS are those known to elicit toxic generation from GPT2. T5-large is the current SOTA model and also the original baseline for ASQA. We simply use the same value model with the policy model (GPT2 or T5). However, due to GPU memory constraint, we use the base instead of large value models.  We will clarify the motivation for these choices in the paper.\n\n\n> ***6. Is the proposed method a good solution in terms of LLM's hallucination?***\n\nIn Figure 3, we empirically show that fine-grained RLHF is more effective than preference-based RLHF in improving factual consistency with given evidence passages for the long-form QA task. We believe our framework has a great potential to help solve the general LLM hallucination issue, but believe further work is necessary to establish the scope of such benefits.\n\n \n> ***7. The training dataset is not large enough (especially for the Long-Form QA task), which limits the verification of a conjecture: As the size of training data increases, the effect of the FG-RLHF proposed in this paper will weaken.***\n\nPlease see our response above regarding the training data size.\n\n> ***8. The proposed method is expensive, not only in the additional computational cost of the reward model, but also in the different deﬁnitions of ﬁne-grained feedback in terms of the feedback types and the density level of each feedback type.***\n\nPlease see the above response about the computational cost of reward models. In the paper, we show that the fine-grained annotation task is easier for human annotators (more consistent in terms of agreements) because the definitions are more concrete, and annotation cost of fine-grained feedback is comparable to pairwise comparisons. We will add more discussion of this point in the next version of the paper.\n"}, {"review_id": "Cq6Jc4kqu4", "reviewer": "Reviewer_7ENw", "summary": "This work improves the reinforcement learning from human feedback (RLHF) with more fine-grained reward signals.\nFor the feedback density, the authors suggest to reward the policy model at sub-sequence level.\nThe experiments on the detoxification task shows that the fine-grained rewards can outperform the holistic sequence level ones.\nFor the feedback diversity, they set three reward categories: 1) **rel.**, irrelevance, repetition, and incoherence, 2) **fact.**, incorrect or unverifiable facts, and 3) **comp.** information completeness.\nThe first two signals, i.e., rel. and fact., are with the sub-sentence and sentence level density, and the comp. is for the full sequence.\nA new QA-FEEDBACK dataset with fine-grained error annotations and pair-wise preference ranks (for the baselien preference-based RLHF) is constucted for the training and evaluation.\nCompared with the preference-based RLHF, the suggested fine-grained one behaves better on all three error types in both human and automatic evaluation.\nThis framework also demonstrates the promising ability of customizing LM behaviors by adjusting the weights of different reward models.", "questions": "1. More different weights settings for the LM customization.\n2. Efficiency and speed evaluations of the suggest method and baselines.", "limitations": "None", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. This work suggests a new perspective to RLHF for LM, where fine-grined and denser reward signals can benefit LMs. \n2. The proposed fine-grained RLHF framework is simple and effective, validated via thorough experiments and analysis.\n3. A new dataset is contributed for the research community, which is carefully created.\n4. The paper is well-written and easy to follow.", "weaknesses": "1. The models used in this paper are relatively limited in size, only including large and base models. I'm curious how your method would scale to larger models, such as the more commonly used 13B and 30B models, and what kind of performance it would achieve. \n2. LM customization is interesting, but lacks human evaluation. I suppose that incorporating human evaluation could help to further understand this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments! Please see our responses below.\n\n> ***1. The models used in this paper are relatively limited in size, only including large and base models. I'm curious how your method would scale to larger models, such as the more commonly used 13B and 30B models, and what kind of performance it would achieve.***\n\nWe agree that analyzing the scaling of our framework to larger models would bring insights to the community, but our work focuses on analyzing the effectiveness of using fine-grained reward for training LMs using RL, compared to holistic reward. Due to limited compute resources (RL training is expensive), we leave the exploration of model scaling for future work.  We do believe that, since many research groups and practitioners face similar computational budget issues, the findings we report on smaller models will be of wide interest.\n\n> ***2. LM customization is interesting, but lacks human evaluation. I suppose that incorporating human evaluation could help to further understand this.***\n\nWe agree that human evaluation is important. Appendix A shows qualitative examples for LM customization. The anecdotal examples we’ve looked at support the findings (see line 301) from automatic evaluation results. We will expand the human assessment study in the appendix to have quantitative results.\n\n> ***3. More different weights settings for the LM customization.***\n\nWe studied different weight settings in Table 5 as well. Table 5 provides results when removing each reward model (i.e., weight=0) on the valid set. Future work can include a more comprehensive personalization of the weights given the end task.\n\n> ***4. Efficiency and speed evaluations of the suggest method and baselines.***\n\nWe use the same RL algorithm (PPO) as preference RLHF, and only observe **very small relative additional cost (~1% training time)** from running multiple reward models (not back-propagated), compared to preference RLHF during RL training. Specifically, the training time of each preference RLHF and fine-grained RLHF run takes 15.1 and 15.3 hours for long-form QA. The inference speed of preference and fine-grained RLHF is the same, roughly 0.7 seconds per example.\n"}, {"review_id": "ZbJsNiX5Bm", "reviewer": "Reviewer_Q8Ja", "summary": "This paper studies how fine-grained RLHF improve over holistic preference RLHF.\nThe study is conducted on two tasks and fine-trained reward comes from sub-sentence, sentence, and full sequence level annotations on different feedback types (toxicity, relevancy, factuality and completeness).\nStudy show that fine-grained reward improved over holistic preference and demonstrated LM behaviors can be customized using combinations of feedback types.", "questions": "See weakness section.", "limitations": "NA", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is clearly written with good visual aid, has sufficient details on datasets, reward model training, descriptions on experiment hyperparameters and frameworks.\n- The paper demonstrates benefits of fine-grained reward model on two tasks, with various reward density and reward types. It is convincing to see  it works across very different problem settings.\n- There is clear and concise analysis on reward model and human rating agreement to understand the quality of reward model.\n- Experiment sections contains relevant ablations that supports main results and analysis of how different types of reward model works together.", "weaknesses": "- (minor) in the detoxification analysis, I'm not fully convinced that the gain comes from reward, it is possible that the API has better accuracy annotating shorter sequence. Some additional analysis similar to your reward model human agreement analysis on task2 could help explain it better. \n- I didn't find the mention of reward model size which I assume would be the same as the value model of t5-base which is smaller than the policy model. In previous works there is advantages of using larger than policy network value/reward function. It might be important to show what is the effect of reward/value model size on fine-grained reward vs holistic reward.\n- (major) on task2, there lacks analysis comparing F.G.RLHF vs Pref.RLHF on holistic preference eval. It would be a lot more convincing if there is gain in that eval. Otherwise it could be argued that these different approaches are optimizing toward very different signals.\n- the comparison ChatGPT using ROUGE score: it is a reference-based score that might not correlated well with human preference.  Using ROUGE score itself to compare these systems are not sufficient or even detrimental. Your eval with rel/fact/comp and the holistic preference are much better metrics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments! Please see our responses below.\n\n> ***1. (minor) in the detoxification analysis, I'm not fully convinced that the gain comes from reward, it is possible that the API has better accuracy annotating shorter sequence. Some additional analysis similar to your reward model human agreement analysis on task2 could help explain it better.***\n\nThis is an interesting question. One possible analysis would require evaluating the API on hand-annotated data of different lengths, which is not available (to our knowledge). In our informal experiments, the toxicity is context-dependent and benefits from longer sections, which is why our sentence-level reward is calculated as API score difference of before and after the sentence generation (line 129).\n\n\n> ***2. I didn't find the mention of reward model size which I assume would be the same as the value model of t5-base which is smaller than the policy model. In previous works there is advantages of using larger than policy network value/reward function. It might be important to show what is the effect of reward/value model size on fine-grained reward vs holistic reward.***\n\nWe follow Ouyang et al., 2022 to use reward models (Longformer-base) with a similar size as value models. It would be very interesting to explore the effect of reward/value model sizes on policy model performance, but this is out of the scope of our work. Some recent studies provide insights on the scaling of reward models: Scaling Laws for Reward Model Overoptimization; Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. We will add this to our discussion section.\n\n> ***3. (major) on task2, there lacks analysis comparing F.G.RLHF vs Pref.RLHF on holistic preference eval. It would be a lot more convincing if there is gain in that eval. Otherwise it could be argued that these different approaches are optimizing toward very different signals.***\n\n**We do have the overall human preference results on fine-grained vs preference-based RLHF – 30.5% (win), 45% (tie), 24.5% (lose)**, where win/lose refers to fine-grained RLHF. We did not include the results because we believe the fine-grained evaluation would draw more insights and also due to space limits. We will try to squeeze these numbers in or put them in the Appendix.\n\n> ***4. the comparison ChatGPT using ROUGE score: it is a reference-based score that might not correlated well with human preference. Using ROUGE score itself to compare these systems are not sufficient or even detrimental. Your eval with rel/fact/comp and the holistic preference are much better metrics.***\n\nDue to the same concern with ROUGE, we conduct additional manual analysis (line 353) and provide qualitative examples (Appendix A) on ChatGPT results. We didn’t use our reward models to automatically evaluate ChatGPT outputs because reward models trained on T5-large do not generalize well to ChatGPT.  We will explain this more clearly in the paper. \n"}, {"review_id": "Grh76isQG5", "reviewer": "Reviewer_uPU2", "summary": "This paper proposes *FINE-GRAINED RLHF*: an extension to (the now popular) *reinforcement learning from human feedback* (RLHF) that allows incorporating more granular human feedback into the language model (LM) training process. Typically, RLHF leverages reward/preference models that are trained to output a *scalar* reward for the whole generated sequence, and this *reward sparsity* can hinder improvements for long-sequence generation. Furthermore, scalar rewards do not offer insights into which parts of the sequence were problematic or displayed undesired behaviors.\n\nTo address this, the using *fine-grained reward models*: for each \"category\" of feedback of interest, a separate reward model is trained. Furthermore, these models generate feedback at lower granularity (for example, at sentence-level). The policy LM can then be optimized with respect *weighted* sum of the rewards at each timestep, using standard RL algorithms like PPO.\n\nThe authors explore the impact on two language generation tasks: (1) detoxification and (2) long-form question answering.\nFor detoxification, the authors find that (with a reward based on an external API) training with rewards provided for every sentence yields better performance when compared to a holistic sequence-level reward. However this evaluation is fully automatic.\nFor long-form question answering (QA), the authors introduce a novel dataset, QA-Feedback, accompanied with fine-grained human feedback on initial policy LM. The feedback comes in three error categories at different levels of granularity: (a) irrelevance, repetition or incoherence (sub-sentence level), (b) incorrect or unverifiable facts (sentence level), and (c) incomplete information (whole sequence level). They collect this feedback through crowd-workers. They also collect *preference-based* feedback, for comparison with (hollistic) RLHF.\nBy training separate reward models for each category, the authors show that fine-grained RLHF outperforms the traditional RLHF and supervised baselines, both in automatic and human evaluation. Interestingly, by carefully adjusting the weights of different reward models, the authors show LM behaviors can be fine-tuned to optimize different combinations of desired behavior, and outperform. The authors also explore the reward model accuracy and even compare against a ChatGPT baseline.", "questions": "- The levels of granularity in the paper make sense for the three categories considered, but have you thought about the more general problem of allowing arbitrary granularities? It seems it is mostly a UX/feedback collection process, and that nothing in the algorithm would prevent it.\n- Is there plans to open-source the (fine-grained) RLHF code? Integrating this into popular packages for RLHF (like TRLX) could heavily boost the impact of this work", "limitations": "See above", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I really enjoyed reading this paper. It explores a very prominent topic recently (learning from human feedback), it is well written and it does an very extensive analysis (at least for one of the tasks it tackles). “Reward sparsity” is a well known problem in RL, and recent works in RLHF for LLMs just kinda ignores it. Their proposed fine-grained RLHF is seems like a good approach to tackle it, makes intuitive sense and seems easy to implement. Their findings surrounding controllability could also have significant implications for (RLHF’d) LLMs. They also introduce a novel dataset for QA with fine-grained feedback.", "weaknesses": "The main weakness of this paper was that only a single LLM was tried: while I believe that their findings will generalize to other LLMs, an experimental validation of this would make this paper even stronger. Even trying re-using the reward models trained for the initial GPT-2 policy on some other LLaMA would already be quite a bonus. \n\nAlso a couple of other minor ones:\n\n- I think task 1 (detoxification) in general is not a good setup: in uses closed source APIs for model-based (from a relatively weak one) PPL as evaluation. the simulated “sentence-level” reward is also not very elegant. I would ideally suggest trying to replace it with another simple, controllable generation task and preferably using human-eval. But ye most of these are covered by task 2\n- The multi-reward aspect of the algorithm makes RLHF now a sort of *multi-task* optimization problem, and bring with the problem of how to weight the different categories. The discussion on the trade-off in (4.5) is interesting and helpful, but I think more could be said on how to pick a weighting for a general case where we just want a single “quality”. Also a very simple pareto analysis (probably trying more weightings of the reward) and some plots showing the final trade-off (ideally in terms of human reward?) would add more technical insight", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments! Please see our responses below.\n\n> ***1. The main weakness of this paper was that only a single LLM was tried: while I believe that their findings will generalize to other LLMs, an experimental validation of this would make this paper even stronger. Even trying re-using the reward models trained for the initial GPT-2 policy on some other LLaMA would already be quite a bonus.***\n\nWe agree that analyzing the scaling of our framework to larger models would bring insights to the community, but **our work focuses on analyzing the effectiveness of using fine-grained reward for training LMs using RL**, compared to holistic reward. Due to limited compute resources (RL training is expensive), **we leave the exploration of model scaling for future work**. We do believe that, since many research groups and practitioners face similar computational budget issues, the findings we report on smaller models will be of wide interest.\n\n\n\n> ***2. Also a couple of other minor ones:***\n\n> I think task 1 (detoxification) in general is not a good setup: in uses closed source APIs for model-based (from a relatively weak one) PPL as evaluation. the simulated “sentence-level” reward is also not very elegant. I would ideally suggest trying to replace it with another simple, controllable generation task and preferably using human-eval. But ye most of these are covered by task 2\n\nOur setup of task 1 including the policy model (GPT2) and evaluation metrics are consistent with previous studies of our baselines. Although it’s not perfect, it leads to a fair comparison with the baselines. The use of a reward at the sentence-level provides a contrasting case of “fine-grained” compared to task 2. The specific definition for the sentence-level reward (the score change with a newly generated sentence) was carefully designed based on the definition of PERSPECTIVE scores (the likelihood that someone will perceive the text sequence as toxic) and reward in RL (the reward collected by taking the action $a$ at state $s$).\n\n\n> The multi-reward aspect of the algorithm makes RLHF now a sort of multi-task optimization problem, and bring with the problem of how to weight the different categories. The discussion on the trade-off in (4.5) is interesting and helpful, but I think more could be said on how to pick a weighting for a general case where we just want a single “quality”. Also a very simple pareto analysis (probably trying more weightings of the reward) and some plots showing the final trade-off (ideally in terms of human reward?) would add more technical insight\n\nTask 1 experiments reflect the case where only a single “quality” is used, where “quality” means reward, and the fine-grained rewards are temporal (denser). The Pareto analysis would be feasible with Rouge scores, but the human evaluation is beyond the scope of what we can explore in this study and it would be particularly important for finding a good and general weight tradeoff.\n\n\n> ***Questions:***\n\n> The levels of granularity in the paper make sense for the three categories considered, but have you thought about the more general problem of allowing arbitrary granularities? It seems it is mostly a UX/feedback collection process, and that nothing in the algorithm would prevent it.\n\nYes, our fine-grained RLHF framework can be applied to arbitrary granularities if the corresponding reward model is available. In fact, we collected our feedback data at the phrase level. However, we finally decided to use sub-sentence and sentence density levels, due to the fact that human annotators have low agreements on phrase boundaries (explained on line 196). Future work can explore more fine-grained feedback.\n\n\n> Is there plans to open-source the (fine-grained) RLHF code? Integrating this into popular packages for RLHF (like TRLX) could heavily boost the impact of this work\n\nYes, we will add a link to our data and code in the final version.\n"}]

Author rebuttal:
We thank all the reviewers for your valuable feedback, and acknowledging our contributions in terms of novelty, effectiveness, and controllability of our fine-grained RLHF, as well as the data quality of QA-Feedback.

Please see our response to each reviewer’s comments separately below. We will add all corresponding clarifications and discussions to the next version of the paper. As we notice reviewers’ shared interest in our analysis on LM customization, we plan to expand the human assessment study on it to include more insights in the next version of the paper.


Meta-review: The paper presents FINE-GRAINED RLHF, a framework that allows for training and learning from reward functions that are fine-grained in different error categories. The authors demonstrate improved performance on detoxification and long-form question answering tasks through learning with this FINE-GRAINED reward function. 

The idea of  fine-grained reward learning is interesting and has  has the potential to improve language model performance by providing more detailed feedback during training. The framework demonstrates the promising ability of customizing LM behaviors by adjusting the weights of different reward models.

While there are some open questions about choice of base models, evaluation and generalization to other tasks, the paper presents a valuable contribution  and has potential for impacting future research in this area.
Decision: Accept (spotlight)