Title: Fine-Grained Human Feedback Gives Better Rewards for Language Model Training
Venue: NeurIPS 2023
Accepted: yes
Keywords: Language Model

Reinforcement Learning with Human Feedback

Long-Form Text Generation
Abstract: Language models (LMs) often exhibit undesirable text generation behaviors, including generating false, toxic, or irrelevant outputs. 
Reinforcement learning from human feedback (RLHF)---where human preference judgments on LM outputs are transformed into a learning signal---has recently shown promise in addressing these issues. However, such holistic feedback conveys limited information on long text outputs; it does not indicate which aspects of the outputs influenced user preference; e.g., which parts contain what type(s) of errors. In this paper, we use fine-grained human feedback (e.g., which sentence is false, which sub-sentence is irrelevant) as an explicit training signal. We introduce Fine-Grained RLHF, a framework that enables training and learning from reward functions that are fine-grained in two respects: (1) density, providing a reward after every segment (e.g., a sentence) is generated; and (2) incorporating multiple reward models associated with different feedback types (e.g., factual incorrectness, irrelevance, and information incompleteness). We conduct experiments on detoxification and long-form question answering to illustrate how learning with this reward function leads to improved performance, supported by both automatic and human evaluation. Additionally, we show that LM behaviors can be customized using different combinations of fine-grained reward models. We release all data, collected human feedback, and codes at https://FineGrainedRLHF.github.io.

Fine-Grained Human Feedback Gives Better Rewards for Language Model Training

Abstract

Language models (LMs) often exhibit undesirable text generation behaviors, including generating false, toxic, or irrelevant outputs. Reinforcement learning from human feedback (RLHF)-where human preference judgments on LM outputs are transformed into a learning signal-has recently shown promise in addressing these issues. However, such holistic feedback conveys limited information on long text outputs; it does not indicate which aspects of the outputs influenced user preference; e.g., which parts contain what type(s) of errors. In this paper, we use fine-grained human feedback (e.g., which sentence is false, which sub-sentence is irrelevant) as an explicit training signal. We introduce FINE-GRAINED RLHF , a framework that enables training and learning from reward functions that are fine-grained in two respects: (1) density, providing a reward after every segment (e.g., a sentence) is generated; and (2) incorporating multiple reward models associated with different feedback types (e.g., factual incorrectness, irrelevance, and information incompleteness). We conduct experiments on detoxification and long-form question answering to illustrate how learning with such reward functions leads to improved performance, supported by both automatic and human evaluation. Additionally, we show that LM behaviors can be customized using different combinations of fine-grained reward models. We release all data, collected human feedback, and codes at https://FineGrainedRLHF.github.io .

1 Introduction

State-of-the-art AI is built on pre-trained language models that are then trained through interaction with humans [29, 28, 9], with a combination of supervised learning and reinforcement learning. Incorporating human feedback into the process of language model (LM) training has been shown as effective to reduce false, toxic and other undesired model generation outputs [29, 3, 2, 33, 10]. Many of these studies adopt reinforcement learning from human feedback (RLHF) [46], a framework that converts human feedback into an effective LM training signal to reach these goals. Specifically, humans are presented with two or more outputs and asked to select one or rank them, and this signal is then used to train a reward model, which computes a single scalar reward for each LM-generated sequence. The LM is then trained with RL to optimize the reward it receives (from the reward model).

Such a reward provides a relatively sparse training signal, especially for tasks that require the generation of long-form text-making RLHF in such domains unreliable [33]. Furthermore, previous research [8, 18, 11, 45, 25, 43] into automated evaluation of generated text shows that it can be challenging for human annotators to reliably compare the overall quality of two or more model outputs when the outputs contain a mixture of diverse undesired behaviors. They demonstrate how categorizing and localizing model errors (i.e., fine-grained evaluation) provides explicit insights about which part of the model output has what type of problem. We thus ask the question: how can we improve rewards for LM training via RLHF by using more fine-grained human feedback ?

Relevant: + 0.3  Factual: - 0.5

Relevant: + 0.3 Factual: + 0.5 Info. complete: + 0.3

LM outputs:

The atmosphere of Earth is a

layer of gases retained by

Earth's gravity…

The atmosphere is commonly

known as air. The top gases

by volume that dry air …

Step 1:

Collect human feedback and train the reward models

Step 2:

Fine-tune the policy LM against the reward models using RL

LM output:

The atmosphere of Earth is a layer of gases retained by Earth's

gravity.  The  most  common  gas,  by  dry  air  volume,  is  nitrogen.

The second most is oxygen. The third most is carbon dioxide.

Irrelevant / Redundant

Unverifiable / Untruthful

Missing

The third most is Argon.

What are the 3 most common gasses in earth's atmosphere?

Prompt:

What are the 3 most common gasses in earth's atmosphere?

Prompt:

Sampled Prompt:

Does water boil quicker at high altitudes?

It  takes  longer  for  water  to  boil  at  high

altitudes. The reason is that water boils at

a lower temperature at higher altitudes.

PPO

Update policy with rewards

Preference Reward: - 0.35

Update policy with rewards

PPO

(a) Preference-based RLHF

(b) Ours:

Fine-Grained RLHF

Human Feedback

Factuality RM

Information

Completeness RM

Preference RM

Fine-Grained Human Feedback

It  takes  longer  for  water  to  boil  at  high

altitudes. The reason is that water boils at

a lower temperature at higher altitudes.

Sampled Prompt:

Does water boil quicker at high altitudes?

The air that surrounds the

planet Earth contains various

The atmosphere of Earth is

the layer of gases, generally

known as air…

>         =         >

In this paper, we propose that humans give fine-grained feedback to LM output, associating categories of undesired behavior (e.g., false or irrelevant generations) and a text span at a density (e.g., sentence or sub-sentence-level). To enable LMs to learn from such fine-grained feedback, we introduce the FINE-GRAINED RLHF framework. As shown in Figure 1, we first use collected human feedback to train fine-grained reward models such that each of them focuses on one category and provides rewards at the density associated with that category. We then integrate these reward models into Proximal Policy Optimization (PPO) [37], a commonly used RL algorithm for training LMs with preference-based human feedback (§2).

We conduct experiments on two language generation tasks-detoxification [12] (§3) and long-form question answering (QA) [39] (§4). For detoxification, toxicity is the only error category and we explore learning with a dense reward. We adopt PERSPECTIVE [1], a widely used language toxicity detection model trained on millions of human annotations, as our reward model. We use it to calculate a fine-grained reward after the generation of every sentence. Our experimental results show the efficacy and data efficiency of training models with dense reward compared to a holistic sequence-level reward, supported by automatic evaluation results.

With experiments on long-form QA, we aim to examine training models with fine-grained rewards at the two granularity dimensions (density and error category), for which we construct a long-form QA dataset, QA-FEEDBACK, along with our collected human feedback. We carefully develop a pipeline to collect fine-grained human feedback on three error categories at different density levels: i) irrelevance, repetition, or incoherence (sub-sentence), ii) incorrect or unverifiable facts (sentence), and iii) incomplete information (whole sequence; see Figure 1). Our experimental results show improved results in each error category by learning with such fine-grained feedback, supported by both automatic and human evaluation results. In a scenario with multiple reward models representing different error types, we also show FINE-GRAINED RLHF allows us to combine reward models with different weights and thus control the model training process towards a customized combination of desired behaviors.

2 FINE-GRAINED RLHF

We introduce FINE-GRAINED RLHF, a framework that enables us to train fine-grained reward functions for generation outputs across different feedback types. We first define the RL environment and learning algorithm. Then we define the fine-grained reward models and describe how to incorporate the fine-grained reward model(s) into an RL algorithm, in contrast to previous RLHF studies that only consider a single reward.

Environment: language generation as a MDP. We focus on language generation tasks. For each task, we are given a set of task input prompts D = { x n } N n =1 . We follow [33] to define language generation as a Markov Decision Process (MDP) ⟨S , A , R , P, γ, T max ⟩ with a finite vocabulary V . Each MDP episode starts with a sampled prompt x = ( x 1 , x 2 , . . . , x l ) with x i ∈ V , and ends when the current time step exceeds T max or an end of sequence token is generated. S is the state space and s 0 = ( x 1 , x 2 , . . . , x l ) ∈ S is the initial state. An action in the environment a t ∈ A is a generated token (by the policy LM model P θ ) at time t from V ( a 0 is the begin sequence token). The transition function P : S × A → ∆ S appends a t at the end of the state s t = ( x 1 , x 2 , . . . , x l , a 0 , a 1 , . . . , a t -1 ) . This process continues until the end time step T ≤ T max is reached, which gives a generated sequence y = ( a 1 , . . . , a T ) . A reward function R : S × A → R , which comes from the reward model(s) in FINE-GRAINED RLHF, provides dense rewards before and when T is reached. P θ can be initialized with a pre-trained language model, and sometimes also with supervised fine-tuning on task-specific demonstrations. The reward function is defined later.

Learning algorithm: proximal policy optimization (PPO). PPO [37] is an actor-critic RL algorithm that is widely used in previous RLHF work to optimize the policy model against a reward model of human feedback. It uses a value model V ψ ( s t ) to estimate the value of state s t , and optimizes the policy model with a PPO clipped surrogate training objective. The advantage A t at timestep t is estimated by a generalized advantage estimation function [36]: A t = ∑ T t ′ = t ( γλ ) t ′ -t ( r t ′ + γV ψ ( s t ′ +1 ) -V ψ ( s t ′ )) , with γ as a hyperparameter and λ as the discounting factor for rewards. r t is the reward assigned to a t , which in our case is acquired using one or multiple learned reward models. The value model V ψ ( s t ) is optimized with an expected squared-error loss with the value target as V targ ( s t ) = ∑ T -1 t ′ = t γ t ′ -t r t ′ + γ T -t V ψ old ( s T ) , where V ψ old is the lagging value model. Finally, PPO is trained to optimize both policy ( P θ ) and value ( V ψ ) models with their respective objectives. No reward model is being optimized during PPO training. See Appendix B for more details.

Fine-grained reward models. Previous RLHF work adopts a holistic reward model R ϕ that maps input prompt x and generated output y to a single scalar reward representing its overall quality (Figure 1(a)). This single scalar reward is only assigned to the final token in the generated sequence, a T . Formally, r t = R ϕ ( x, y ) if t = T and 0 otherwise.

In contrast, we consider a reward function that is derived from one or multiple fine-grained reward models that (1) provide rewards densely (i.e., for subsequences of the generated output), and (2) compute rewards on distinct categories of undesired behaviors (e.g., false or repetitive generation), where each category is associated with an individual reward model.

For a fine-grained reward model R ϕ k that gives feedback on error category C k , we first segment y into L k segments ( y k 1 , y k 2 , . . . , y k L k ) corresponding to the density (e.g., sentence-level) of R ϕ k , where each segment y k j ends at timestep T k j . R ϕ k outputs a reward R ϕ k ( x, y, j ) for each segment y k j given x and y as the input, which is assigned to the final token in y k j . Additionally, to ensure the fluency of generated outputs, we follow [41] to add an approximate KL divergence penalty to each token a t with a weight β , that is not backpropagated through during training. Formally, assuming that we have K fine-grained reward models that represent different error categories, we will have a combined reward function for each token a t as:

where w k ∈ R is a weight assigned to reward model R ϕ k . Then we follow the same PPO training algorithm to optimize the policy model. We discuss how we define and train fine-grained reward models for the detoxification and long-form QA task in our experiments in § 3 and § 4 respectively.

3 Task 1: Detoxification

The task of detoxification aims to reduce the toxicity in the model generation y when given a prompt x . Toxicity is the only undesired behavior in this task, and we aim to explore learning with a dense reward in comparison to a single holistic reward. We conduct our experiments on REALTOXICITYPROMPTS, a dataset of 100K sentence-level prompts derived from the web that are known to easily elicit problematic generations in GPT-2 [31]. Using a dense sentence-level finegrained reward, we demonstrate that our fine-grained reward exhibits greater sample efficiency compared to a holistic reward , achieving lower toxicity with fewer training steps while maintaining better fluency (§3.1).

Holistic reward for (non-)Toxicity. We use the PERSPECTIVE API [1] as our reward model, which is widely used for language toxicity detection and is trained with millions of examples gathered from several online platforms and annotated by human annotators for toxicity. That means we use an off-policy reward model that is not trained on outputs from P θ init . The API outputs a score between 0 (non-toxic) and 1 (toxic). Given the entire model output y , the holistic reward for RL is 1 -PERSPECTIVE( y ).

Sentence-level (fine-grained) reward for (non-)Toxicity. To calculate the fine-grained reward , we query the API after the model generates each sentence instead of generating the full sequence. For each generated sentence y j , we assign PERSPECTIVE( [ y 1 , . . . , y j -1 ] ) - PERSPECTIVE( [ y 1 , . . . , y j ] ) as the sentence reward (i.e., how much toxicity is changed from generating y j ). Since there is only one error category, we omit the category superscript, using y j to denote the j th segment (e.g., sentence) in y .

3.1 Experiments

Implementation details. We follow previous work [17, 21] and use GPT-2 large model as the initial policy model P θ init . During both the exploration stage in RL training and inference, we use nucleus sampling decoding with p = 0.9 and temperature = 1.0. The generation length limit is set to 48. The value model used during RL training is initialized with GPT-2-base due to GPU memory constraint. We report RL training parameters in Appendix B. All scores are averaged over 3 independent runs.

Compared systems and evaluation. We report the performance of FINE-GRAINED RLHF , RLHF with holistic reward ( Hol. RLHF ), and the state-of-the-art controlled generation approaches GeDi [17] and DEXPERTS [21]. We follow previous work [17, 21] to report the toxicity score calculated on each full generation sequence from the PERPLEXITY API, as well as other commonly used metrics for REALTOXICITYPROMPTS, including n-gram diversity and GPT-2 XL perplexity (PPL) as a proxy for fluency. The lower the perplexity, the more fluent the generated text. The toxicity score is reported as the maximum score among 4 sampled model outputs, averaged over all test input prompts. Other metrics are reported as the average score of the same 4 samples.

Main results. Table 1 shows the experimental results on the REALTOXICITYPROMPTS test set. FINEGRAINED RLHF with sentence-level fine-grained reward attains the lowest toxicity and perplexity among all methods, while maintaining a similar level of diversity.

Sample efficiency analysis. Figure 2 shows the max toxicity and average perplexity on the development set during training. FINE-GRAINED RLHF has the toxicity drop much faster while keeping a low-level perplexity. This shows that learning from denser fine-grained reward is more sample efficient than holistic reward. One explanation is that fine-grained reward locates where the toxic

content is, which is a stronger training signal compared with a scalar reward for the whole text. The cost is that we have to query the reward model more times per example.

4 Task 2: Long-Form Question Answering (QA)

Long-form QA requires an LM to generate a textual response to a question with a comprehensive answer and explanation. To examine learning with fine-grained rewards at the two granularity dimensions (error category and density), we collect QA-FEEDBACK (§4.1), a long-form QA dataset annotated with human feedback on LM-generated responses. We define three error categories at different density levels and train a reward model for each (§4.2). We describe the experimental setup in §4.3. Both human and automatic evaluation show that FINE-GRAINED RLHF outperforms preference-based RLHF and supervised fine-tuning models on all error categories (§4.4). We then show that adjusting the weights of fine-grained reward models during RL training leads to distinct behaviors in LM generation, allowing us to customize the LM for users with different needs (§4.5). Finally, we conduct an in-depth analysis of the fine-grained reward models, revealing that they compete against each other, and provide an analysis of their impact on the resulting policy model.

4.1 QA-FEEDBACK: Long Form QA with Human Feedback

QA-FEEDBACK is based on ASQA [39], a dataset that focuses on answering ambiguous factoid questions [26] in an open-domain setting. We use their provided oracle knowledge contexts to reformulate the task into a reading comprehension setting: given the input x that contains a question q and a set of knowledge passages P = { p 1 , . . . , p | P | } , generate a long-form response y . On average, there are roughly 65 words in each gold response. Since ASQA does not release the test set, we create our own train/development/test data split from the original train and development sets. We name our newly constructed data, along with collected human feedback (discussed next), QA-FEEDBACK. Overall, we have 3,853 training, 500 development, and 948 test examples (details in Appendix C).

Initial policy and fine-grained human feedback. Before collecting human feedback, we follow [33] to initialize the policy model with supervised fine-tuning on a small set of examples. Specifically, we use 1K training examples to supervise fine-tuning of T5-large (the original baseline for ASQA) [32] to get P θ init . We name this initial policy model SFT . We then sample outputs from SFT for the remaining training and development examples and collect fine-grained human feedback in three error categoriesC 1 : irrelevance, repetition, or incoherence ; C 2 : incorrect or unverifiable facts based on knowledge passages; and C 3 : incomplete information . The collected feedback instances are then used as the training and development examples for training reward models. For each task prompt x , we only collect fine-grained feedback for one model output. Our data collection has IRB approval and is deemed exempt.

We instruct workers to identify any error in each model output y = ( a 1 , . . . , a T ) , marking the span of text associated with each identified error type. Formally, we define the set of user-annotated feedback for a task prompt x and model output y as F = { f i } where each f i = ⟨ c i , b i , e i ⟩ represents the user-identified span ( a b i , . . . , a e i ) of the error category C c i , where c i ∈ { 1 , 2 , 3 } . Importantly, we impose three restrictions in the annotation: (1) error spans of category C 1 or C 2 should not overlap with each other; (2) only spans that do not have error C 1 need to be assessed as containing error C 2 or not; (3) C 3 can only apply to whole output sequences. Additionally, we ask workers to mark passage sentences that contain missing information if a C 3 error is annotated. We also ask workers to rewrite y into a corrected version y ′ that addresses all annotated feedback F . Details about the feedback collection interface, instructions, and quality control are in Appendix C.

To analyze human-human agreement, a subset of 300 examples receive annotations from two distinct workers. We observe that while exact agreement in error span boundaries is low, workers achieve reasonably high agreement on whether a sub-sentence contains C 1 and whether a sentence contains C 2 . 2 Therefore, we decide to have the density for error type C 1 , C 2 , and C 3 as sub-sentence, sentence and full sequence. We provide more data analysis including human agreement in Appendix C.

Preference-based human feedback. For comparison purposes, we follow [29] to separately collect pairwise human preferences from the same group of workers. We sample 4 model outputs for each prompt x , which gives 6 pairs of model outputs. We ask the workers to indicate pairwise preferences

(ties are allowed) based on all errors they can find in each model output. They are not asked to explicitly annotate these errors.

Annotation details. On average, both annotation tasks of fine-grained and preference feedback for one question take a worker about 6 minutes to finish. In contrast, [39] report that they spend about 15 minutes to label a human-written response for each question, which is much more time-consuming than our feedback annotation. On average, we pay $1.65 per example for both tasks, leading to $16.50 hourly pay for our workers. We include details of the pay structure in Appendix C. We observe that human annotators can reach a higher agreement in each aspect of fine-grained feedback compared to pairwise comparisons because the feedback definitions are more concrete.

4.2 Fine-Grained Reward Models

We train three separate reward models R ϕ 1 , R ϕ 2 , and R ϕ 3 for C 1 , C 2 , and C 3 error categories respectively with a density of sub-sentence, sentence, and full sequence, respectively. Since reward models provide scalar reward scores and do not perform generation, we use the encoder-only Longformer-base [4] as our backbone model to handle long input sequences (more details of each reward model are in Appendix D).

C 1 : Irrelevance, repetition, or incoherence. R ϕ 1 targets to predict whether each sub-sentence in y contains a C 1 type error. We denote y = ( y 1 1 , . . . , y 1 L 1 ) , where y 1 j is the j th segment at R ϕ 1 's density (i.e., sub-sentence), with L 1 segments in total. We add a 2-class token-level classification layer (a single feed-forward layer) on the top of the Longformer encoder. The model input has the format of ' question: q answer: [sep] y 1 1 [sep] y 1 2 . . . ', and we take the classification output at each [sep] token to indicate whether the following y 1 j contains a C 1 error. We do not add passages in the model input because, intuitively, the detection of C 1 errors does not depend on them. To train R ϕ 1 , we apply a token-level classification loss to each [sep] token before y 1 j , where its gold label g j is ' has error ' if there is a f i ∈ F that has ( a b i , . . . , a e i ) overlapped with y 1 j and c i = 1 , and ' no error ' otherwise. When R ϕ 1 provides a reward during RL training as in Eq. 1, we read a reward R ϕ 1 ( x, y, j ) for every y 1 j given x and y . We define R ϕ 1 ( x, y, j ) = +1 if R ϕ 1 predicts ' no error ' for y 1 j and -1 otherwise.

C 2 : Incorrect or unverifiable facts. R ϕ 2 is developed for detecting a C 2 error at the sentence level in a similar way. The model input has the format of ' question: q context: p 1 p 2 . . . answer: [sep] y 2 1 [sep] y 2 2 . . . ', where p 's denotes the grounding passages and y 2 j represents the j th sentence. We train R ϕ 2 similarly to R ϕ 1 , with one exception: as we instruct the workers not to annotate a C 2 error for a span that is already labeled as containing a C 1 error, we do not calculate loss on sentences that are labeled as containing C 1 but not C 2 during R ϕ 2 training.

C 3 : Incomplete information. R ϕ 3 is trained to measure the information completeness of y , at the full sequence level. Motivated by [19], R ϕ 3 predicts a single scalar reward and is trained with a pairwise comparison loss [29]:

where R ϕ 3 ( x, y ) is the scalar output of the reward model for input x and output y ; ¯ y p and ¯ y l are sampled from the same input x , and ¯ y p has less missed information compared with ¯ y l ; D p contains the pairwise comparisons bootstraped from human feedback on C 3 errors (see details in Appendix D).

Preference-based reward model. The preference-based reward model is trained in a similar way to R ϕ 3 , with ¯ y p representing the human preferred response against ¯ y l in the loss function Eq. 2. It outputs a scalar score for the given x and y that represents the overall response quality.

4.3 Experimental Setup

Compared systems. We compare our proposed method, FINE-GRAINED RLHF with the initial T5 policy model trained with 1K examples ( SFT ) and RLHF with holistic preference-based rewards ( Preference RLHF ). The reward models used in RLHF experiments are trained on 2.8K examples with annotated feedback (but no gold human response). For analysis, we also use the human gold responses of all training examples to finetune a fully supervised T5 model ( SFT-Full ). Notice that SFT-Full requires much higher annotation cost because it takes longer (15 minutes per example [39]) for annotators to draft long-form responses.

Implementation details. Our policy model is based on T5-large [32] and is supervised finetuned on 1K training examples, as explained in §4. During RL exploration, we use top-k ( k = 20 ) sampling decoding with temperature = 0.7, which is set based on previous RLHF work [33]. The value model used during RL training is initialized with T5-base due to GPU memory constraint. The reward model weights we used in FINE-GRAINED RLHF are w 1 = 0 . 3 , w 2 = 0 . 5 , w 3 = 0 . 3 , unless otherwise specified. Although we use three reward models during RL training, we only observe very small relative additional cost (roughly 1% training time) compared to preference RLHF. During inference, we use greedy decoding to generate responses. We report more details including RL training parameters in Appendix B. All scores reported are averaged over 3 independent runs.

Evaluation. We conduct both human and automatic evaluation. Human evaluation is run on 200 randomly sampled test set examples of QA-FEEDBACK to compare Fine-Grained RLHF with all baselines. Each model output is sampled from inference results of 3 training runs. We use the same protocol of feedback collection to have the same set of workers annotate spans in each model output that contain (1) irrelevance, repetition, or incoherence error (rel.) and (2) incorrect or unverifiable facts (fact.) . They are also asked to compare the information completeness (comp.) for each output pair. To report evaluation scores for rel. and fact. error spans, we first map them to their corresponding error type density (sub-sentence and sentence). Then we report the error rate for each error type, measured as the percentage of sub-sentences that contains this type of error. Since spans with rel. error are not checked for fact. error (discussed in §4.1), we exclude sub-sentences with only rel. error when report the error rate of fact. error. For automatic evaluation, we report RougeLSum [20] as used for the original ASQA data, as well as the score from each fine-grained reward model ( R ϕ 1 , R ϕ 2 , and R ϕ 3 ). Specifically, we report the percentage of all sub-sentences (or sentences) in the test set predicted as ' no error ' by R ϕ 1 (or R ϕ 2 ). For R ϕ 3 , we report the averaged output score for all test examples.

4.4 Main Results

Figure 3 shows the human evaluation results for rel. and fact. error types. Table 2 shows the human pairwise comparison results for information completeness ( comp. ).

Error Rate

Ours vs.

Win

Tie

Lose

SFT

23.0%

65.5%

11.5%

SFT-Full

22.0%

61.0%

17.0%

Pref. RLHF

19.5%

71.0%

9.5%

FINE-GRAINED RLHF outperforms SFT and Preference RLHF on all error types. Figure 3 and Table 2 show that our FINE-GRAINED RLHF leads to generation that is much more factually correct and contains more complete information, compared to all other systems. It generates fewer irrelevance, repetition, andincoherence errors, compared with SFT and Preference RLHF. In the meantime, Preference RLHF, despite greatly reducing factual errors compared to the initial policy model SFT, generates even more irrelevance, repetition, and incoherence errors than SFT. FINEGRAINED RLHF outperforms Preference RLHF potentially due to more specific and localized training signals. In addition, we ask annotators to compare the overall generation quality of FINE-GRAINED RLHF and preference RLHF. Although Preference RLHF is trained directly with such preference feedback, FINE-GRAINED RLHF was rated better than Preference RLHF in 30.5% of all examples and worse in 24.5% of examples. The annotators indicate a tie in the remaining 45% of cases. Surprisingly, FINE-GRAINED RLHF outperforms SFT-Full with more factual and complete generation, despite a much lower annotation cost.

RLHF is particularly effective in reducing factual errors. Figure 3 shows that both FINEGRAINED RLHF and Preference RLHF are effective in reducing factual errors in model generation. Meanwhile, we see little or no improvement in reducing irrelevance, repetition, or incoherence errors. We provide more in-depth analysis for this observation in §4.5.

Table 3 shows automatic scores on the QA-FEEDBACK test set, which show similar trends as human evaluation in terms of system comparisons, while all four systems achieve similar Rouge scores.

4.5 LMCustomization with FINE-GRAINED RLHF

Since we use multiple reward models in FINE-GRAINED RLHF, adjusting their weights (see Eq. 1) during RL may lead to different LM behaviors. For example, adding more weight to a reward model associated with one specific desired behavior type (e.g., information completeness) may lead the generation more towards that behavior type compared to others (e.g., information relevance). This flexibility can potentially fit users with diverse needs. Therefore, in this section, we explore FINE-GRAINED RLHF's ability to customize the LM behavior.

LMcustomization. As in Table 4, we explore three configurations of reward model weights ( w 1 , w 2 , and w 3 for R ϕ 1 , R ϕ 2 , and R ϕ 3 ) and name them 'short', 'medium', and 'long' according to the LM's average generation length. For simplicity, we fix w 2 = 0 . 5 and w 3 = 0 . 3 , and use 0.4, 0.3, and 0.2 for w 1 , which leads to 'short', 'medium', and 'long' generation outputs respectively. We manually inspect 30 random examples and observe that (1) 'short' generates more relevant content, but is less factual and complete; (2) 'long', in contrast, gives the most factual and complete generation. This reflects that the LM is referencing a large amount of content from passages; (3) The 'medium' configuration balances the three rewards and has the highest Rouge score. 24/30 examples follow the above rule. Qualitative analysis and examples of LM customization are in Appendix A.

Trade-off between error types. We observe that a higher w 1 leads to a bigger rel. reward, smaller fact. and comp. rewards, and shorter generated outputs. One interpretation is that R ϕ 1 penalizes text spans that are irrelevant to the questions. As such, it encourages answering the question directly and penalizes referencing passages and generating auxiliary information. This reduces the model generation length and information completeness, and induces more factual errors.

4.6 Analysis

Reward models are competing against each other. In the prior section, we find that there is a trade-off between error types. To further look into this phenomenon, we explore the dynamics of each reward model during training. Figure 4 shows each reward model's rewards on the development set during training. All rewards are z-normalized for visualization. We see that the fact. reward is consistently increasing. The rel. reward increases rapidly in the first 250 steps and then starts decreasing, while the comp. reward exhibits an opposite trend, decreasing at first and then starting to increase. As discussed earlier, one interpretation is that relevance (precision) and information completeness (recall) can be adversarial objectives, so the rewards are competing. The three rewards reach an equilibrium point in later steps.

Ablation: Does the LM learn from all reward models? What if we remove one reward model? Table 5 explores the policy LM behavior when one of the three reward models is removed during training. Qualitative examples are in Appendix A. First, we observe that the corresponding reward decreases dramatically when the model is removed. When the rel. reward model ( R ϕ 1 ) is removed,

the outputs become extremely long and the comp. reward is extremely high. We observe the outputs and find the model is copying a lot of content from the passages. When the fact. reward model ( R ϕ 2 ) is removed, the rel. reward becomes the highest. We observe that the LM tends to answer the question directly and not reference the passages, which causes a lot of hallucinations. When the comp. reward model ( R ϕ 3 ) is removed, the outputs are concise and factual but not providing all relevant information to the question. Thus, it has lower information completeness and Rouge score compared with the LM trained with all reward models.

Reward model performance. We report and analyze the performance of each reward model in predicting its corresponding error category. The rel. reward model R ϕ 1 has a binary classification accuracy of 69.6, and an F1 score (for the ' has error ' class) of 68.5 on model-generated subsentences from the development set. We sample 20 sub-sentences where R ϕ 1 predicts the opposite of the human label, and observe that all of them either 1) contain relevant auxiliary information and are marked as ' no error ' by humans, or 2) are marked as irrelevant by humans but provide closely related background information to the question. In other words, R ϕ 1 is mostly struggling with predicting the relevance of auxiliary information, and it rarely fails to predict a direct answer as ' no error '.

The fact. reward model R ϕ 2 has an accuracy of 77.8 and an F1 score of 67.5. We sample 20 sentences where R ϕ 2 makes a prediction mistake and we observe that the mistakes often happen when the generated sentence is highly abstractive instead of directly copying information from the passage. We also observe that more than 80% of human labeled factual errors occur when the model generates a direct answer (not auxiliary information) that contains hallucinated information or a random entity from a passage. We notice that R ϕ 2 correctly captures more than 80% of such errors.

The comp. reward model R ϕ 3 has an accuracy of 70.9 in pairwise comparison. In contrast, the preference-based reward model only reaches an accuracy of 57.2. This helps confirm our intuition that assessing long-form generation outputs holistically can be more ambiguous and subjective than evaluating the outputs with a focus on a specific undesired behavior type.

Comparison with ChatGPT responses. We experiment with answering the questions with ChatGPT. To familiarize ChatGPT with the style of our LFQA task, we prompt it with the task instruction and a single random QA example (due to length limitation). ChatGPT achieves a RougeLSum score of 40.92 on the test set, which is much lower than our models. We do not use our trained reward models to evaluate ChatGPT outputs because reward models trained on T5-large may not generalize well to ChatGPT. We instead manually inspect the ChatGPT responses, and observe that they are mostly concise and factual, yet lack the auxiliary information necessary to clarify ambiguous questions. Qualitative examples are in Appendix A. This shows the difficulty for ChatGPT in learning user-desired behaviors through simple prompting.

5 Related Work

Reinforcement learning from human feedback (RLHF). RLHF [46, 42, 29] aims to optimize the policy language model to generate content that is desired by human. This framework has been explored to improve the model performance on a variety of natural language processing tasks such as text summarization [40], instruction following [29], question answering [24, 27] and reducing harmfulness [3, 2, 22, 10]. Most of these studies collect human preferences over pairs of model outputs on one or a set of desired attributes, in order to train a reward model to assign a holistic score for a generation output during RL training. [13] trains separate reward models that assign scores for different desired attributes, but still uses a single reward that combines scores from all reward models. In contrast, we explore RLHF with fine-grained reward models trained on human feedback where each reward model provides dense reward after every small text segment for a specific type of desired behavior. [30] explores using intermediate rewards to improves LM performance on reasoning tasks.

Learning from human feedback in NLP. There also exists work that explores non-RL methods to learn from human feedback. [44] trains a reward model that predicts a single score for each model output and selects samples with the highest reward scores for supervised fine-tuning. [38, 14, 42] train a conversational model to predict both the response and a binary user satisfaction score in order to improve the response generation. Besides such numerical human feedback, natural language (NL) human feedback has also been explored. [23, 6] collect and store NL human feedback in a feedback memory for the model to retrieve and then perform the end task conditioning on the retrieved feedback. [5, 35, 34] use a refinement model to refine model outputs conditioning on NL human feedback and

then use a reward model to select the best refined outputs for supervised fine-tuning. Methods for using a reward model to guide LM generation towards desired behaviors at inference time [21, 7] can complement our work that aims to improve the LM during training. [16] also explores incorporating human feedback into LM pre-training.

6 Discussion

Annotation Costs. It is important to note that the fine-grained human feedback used for training our fine-grained reward models does not incur a greater cost than holistic human preference. As outlined in § 4.2, our observations reveal that annotators require a substantial amount of time to compare two lengthy text outputs. For the long-form QA task, both fine-grained feedback and preference-based feedback takes approximately 6 minutes per sample for an annotator.

6.1 Broader Impacts

We propose the FINE-GRAINED RLHF framework that can incorporate multiple reward models to provide dense rewards for RL training, which leads to LM outputs that are optimized towards such rewards. Our framework can be applied to any text generation task, thereby enhancing LM performance by offering more nuanced guidance than holistic feedback. The key advantages of the FINE-GRAINED RLHF framework are two-fold:

Flexibility. Our framework significantly expands the versatility of reward models for RLHF. For example, future work involving fact-checking, sentiment classification, toxicity detection, among others, can all be incorporated within this framework. LMs can be trained against all these reward models via FINE-GRAINED RLHF.

Controllablility. Having multiple reward models that stand for different feedback types allows the end user to exert greater control over RL training (e.g., through different combinations of reward model weights; see details in § 4.5). This leads to customized model behaviors, a benefit particularly valuable for applications like educational tools where model personalization is crucial.

6.2 Limitations and Future Work

One major limitation of our framework comes from the additional compute cost of getting finegrained rewards, compared to RLHF with a holistic reward. For instance, in the detoxification task, we need to make multiple PERSPECTIVE API calls for each model output depending on how many sentences are generated, while RLHF with a holistic reward only requires one. In the long-form QA task, we need to calculate a dense reward from multiple reward models, which takes more compute time and GPU memory than a single reward model.

Another limitation is that different tasks may have different definitions of fine-grained feedback in terms of the feedback types and the density level of each type. Therefore, defining feedback that is well-suited for a task and training reward models accordingly requires non-trivial manual effort.

Finally, in this work, we carefully control the quality of annotated feedback, which is then used to train reward models for RL. In practice, when a deployed model is released to the public, end users don't always give clean feedback. Therefore, how to obtain effective learning signals from noisy human feedback in the wild still needs further investigation.

Some other interesting questions to explore in the future include: 1) Can we obtain fine-grained feedback from LMs like GPT-4 instead of humans to improve model performance and reduce annotation costs? 2) How can other non-RL approaches of using human feedback such as controlled generation during inference time complement FINE-GRAINED RLHF? 3) How would fine-grained reward and value model sizes affect policy model performance during RL training?

7 Conclusion

In this work, we introduce FINE-GRAINED RLHF, a framework that enables LMs to learn from multiple fine-grained reward models trained from human feedback, where each reward model detects a specific error category and provides dense rewards. We conduct experimental analysis on two text generation tasks to illustrate the performance gain of FINE-GRAINED RLHF than RLHF over holistic rewards, supported by both automatic and human evaluation. Furthermore, we show that an LM can be customized for specific needs using different combinations of fine-grained reward models.

Acknowledgments

We thank Jiacheng Liu for sharing the standard PPO training code, and Yizhong Wang for providing insights during early discussions of the project. We also thank UW TIAL members for participating in our pilot feedback annotation. We extend our thanks to UW NLP members who provided insights or feedback to our project. Lastly, we especially thank all our AMT workers for helping us annotate the high quality feedback data. This research was developed with funding from the Defense Advanced Research Projects Agency (DARPA) under Contract No. FA8650-23-C-7316. This work was also funded in part by the DARPA MCS program through NIWC Pacific (N66001-19-2-4031), NSF IIS-2044660, and ONR N00014-18-1-2826. The views, opinions and/or findings expressed are those of the author and should not be interpreted as representing the official views or policies of the Department of Defense or the U.S. Government.

References

Perspective API, https://github.com/conversationai/perspectiveapi .

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 , 2022.

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073 , 2022.

Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150 , 2020.

Angelica Chen, Jérémy Scheurer, Tomasz Korbak, Jon Ander Campos, Jun Shern Chan, Samuel R Bowman, Kyunghyun Cho, and Ethan Perez. Improving code generation by training with natural language feedback. arXiv preprint arXiv:2303.16749 , 2023.

Bhavana Dalvi Mishra, Oyvind Tafjord, and Peter Clark. Towards teachable reasoning systems: Using a dynamic memory of user feedback for continual system improvement. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages 9465-9480, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.

Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text generation. In International Conference on Learning Representations , 2020.

Yao Dou, Maxwell Forbes, Rik Koncel-Kedziorski, Noah A. Smith, and Yejin Choi. Is GPT-3 text indistinguishable from human text? scarecrow: A framework for scrutinizing machine text. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 7250-7274, Dublin, Ireland, May 2022. Association for Computational Linguistics.

Yao Fu, Litu Ou, Mingyu Chen, Yuhao Wan, Hao Peng, and Tushar Khot. Chain-of-thought hub: A continuous effort to measure large language models' reasoning performance. arXiv preprint arXiv:2305.17306 , 2023.

Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas Liao, Kamil˙ e Lukoši¯ ut˙ e, Anna Chen, Anna Goldie, Azalia Mirhoseini, Catherine Olsson, Danny Hernandez, et al. The capacity for moral self-correction in large language models. arXiv preprint arXiv:2302.07459 , 2023.

Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. Enabling large language models to generate text with citations. arXiv preprint arXiv:2305.14627 , 2023.

Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. RealToxicityPrompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020 , pages 3356-3369, Online, November 2020. Association for Computational Linguistics.

Amelia Glaese, Nat McAleese, Maja Tr˛ ebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209.14375 , 2022.

Braden Hancock, Antoine Bordes, Pierre-Emmanuel Mazare, and Jason Weston. Learning from dialogue after deployment: Feed yourself, chatbot! In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 3667-3684, Florence, Italy, July 2019. Association for Computational Linguistics.

Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. spaCy: Industrialstrength Natural Language Processing in Python. 2020.

Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Bhalerao, Christopher L Buckley, Jason Phang, Samuel R Bowman, and Ethan Perez. Pretraining language models with human preferences. arXiv preprint arXiv:2302.08582 , 2023.

Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. GeDi: Generative discriminator guided sequence generation. In Findings of the Association for Computational Linguistics: EMNLP 2021 , pages 4929-4952, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.

Kalpesh Krishna, Erin Bransom, Bailey Kuehl, Mohit Iyyer, Pradeep Dasigi, Arman Cohan, and Kyle Lo. LongEval: Guidelines for human evaluation of faithfulness in long-form summarization. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics , pages 1650-1669, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics.

Margaret Li, Jason Weston, and Stephen Roller. Acute-eval: Improved dialogue evaluation with optimized questions and multi-turn comparisons. arXiv preprint arXiv:1909.03087 , 2019.

Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out , pages 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics.

Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A. Smith, and Yejin Choi. DExperts: Decoding-time controlled text generation with experts and anti-experts. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 6691-6706, Online, August 2021. Association for Computational Linguistics.

Ximing Lu, Sean Welleck, Liwei Jiang, Jack Hessel, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi. Quark: Controllable text generation with reinforced unlearning. In Thirty-sixth Conference on Neural Information Processing Systems (NeurIPS) , 2022.

Aman Madaan, Niket Tandon, Peter Clark, and Yiming Yang. Memory-assisted prompt editing to improve GPT-3 after deployment. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages 2833-2861, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.

Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell-Gillingham, Geoffrey Irving, et al. Teaching language models to support answers with verified quotes. arXiv preprint arXiv:2203.11147 , 2022.

Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. arXiv preprint arXiv:2305.14251 , 2023.

Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. AmbigQA: Answering ambiguous open-domain questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 5783-5797, Online, November 2020. Association for Computational Linguistics.

Reiichiro Nakano, Jacob Hilton, S. Arun Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browser-assisted question-answering with human feedback. ArXiv , abs/2112.09332, 2021.

OpenAI. Gpt-4 technical report, 2023.

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022.

Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beatriz Borges, Antoine Bosselut, Robert West, and Boi Faltings. Refiner: Reasoning feedback on intermediate representations, 2023.

Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research , 21(1):5485-5551, 2020.

Rajkumar Ramamurthy*, Prithviraj Ammanabrolu*, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization. In International Conference on Learning Representations (ICLR) , 2023.

Jérémy Scheurer, Jon Ander Campos, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. Training language models with natural language feedback. arXiv preprint arXiv:2204.14146 , 2022.

Jérémy Scheurer, Jon Ander Campos, Tomasz Korbak, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. Training language models with language feedback at scale, 2023.

John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. Highdimensional continuous control using generalized advantage estimation. In Proceedings of the International Conference on Learning Representations (ICLR) , 2016.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Weiyan Shi, Emily Dinan, Kurt Shuster, Jason Weston, and Jing Xu. When life gives you lemons, make cherryade: Converting feedback from bad responses into good labels. arXiv preprint arXiv:2210.15893 , 2022.

Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming-Wei Chang. ASQA: Factoid questions meet long-form answers. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages 8273-8288, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.

Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan J. Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. ArXiv , abs/2009.01325, 2020.

Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. Recursively summarizing books with human feedback. arXiv preprint arXiv:2109.10862 , 2021.

Jing Xu, Megan Ung, Mojtaba Komeili, Kushal Arora, Y-Lan Boureau, and Jason Weston. Learning new skills after deployment: Improving open-domain internet-driven dialogue with human feedback. arXiv preprint arXiv:2208.03270 , 2022.

Wenda Xu, Danqing Wang, Liangming Pan, Zhenqiao Song, Markus Freitag, William Yang Wang, and Lei Li. Instructscore: Towards explainable text generation evaluation with automatic feedback. arXiv preprint arXiv:2305.14282 , 2023.

Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback without tears, 2023.

Xiang Yue, Boshi Wang, Kai Zhang, Ziru Chen, Yu Su, and Huan Sun. Automatic evaluation of attribution by large language models. arXiv preprint arXiv:2305.06311 , 2023.

Daniel M. Ziegler, Nisan Stiennon, Jeff Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. ArXiv , abs/1909.08593, 2019.

Appendices

A Qualitative Examples for Long-Form QA

A.1 Examples on LM Customization

As discussed in § 4.5, we can modify the weight of each fine-grained reward model during RL training to get LM with different behaviors. Here, we explore three configurations of reward model weights and name them 'short', 'medium', and 'long' based on the LM's average generation length. The 'short' configuration generates concise and short responses, while the 'long' configuration generates detailed and long responses. Table 6 demonstrates the different behaviors of our customized LMs. Given the same question, each LM generates different amount of auxiliary information in the response.

A.2 Examples on LM Errors

Table 7 and Table 8 show examples of LM outputs from all the compared systems (SFT, Pref. RLHF, and FINE-GRAINED RLHF). We mark the fine-grained errors on the model outputs. Overall, our FINE-GRAINED RLHF outperforms SFT and Pref. RLHF in all three error types.

Table 7: Examples of errors in LM outputs of SFT, Pref. RLHF, and our FINE-GRAINED RLHF. We mark the C 1 (irrelevance, repetition, or incoherence) and C 2 (incorrect or unverifiable facts) types of errors in the model outputs.

Table 8: Qualitative examples on another question. Here we highlight the information in blue that is covered by the LM trained with FINE-GRAINED RLHF but missed ( C 3 : incomplete information error) by SFT and Pref.RLHF.

A.3 Examples on Reward Model Ablation

As discussed in § 4.6, reward models are competing against each other, and we experiment with removing one of the three reward models during RL training. Table 9 shows an example of how LMs behave in such scenarios. See § 4.6 for our observations.

Table 9: Qualitative examples on the reward model ablation experiment. We show the LM outputs when we use all the reward models and when one of the reward models is removed during training. We also mark the errors in each output.

A.4 Comparison with ChatGPT responses

We compare the responses generated by ChatGPT (one-shot) 3 and our system in Table 10. As discussed in § 4.6, We find that ChatGPT responses are relevant and factual, yet lack the auxiliary information to answer the ambiguous questions. This shows that it is challenging for ChatGPT to learn user-desired behaviors through prompting and in-context learning.

B Algorithm and Training Details of FINE-GRAINED RLHF

B.1 PPO with Fine-Grained Rewards

The algorithm below shows in detail how PPO updates the policy LM P θ and the value model V ψ with K fine-grained reward models R ϕ k .

Algorithm 1 Fine-Grained Reinforcement Learning from Human Feedback (FINE-GRAINED RLHF) Input initial policy model P θ init ; initial value model V ψ init ; K reward models R ϕ k trained from human feedback; task prompts D ; hyperparameters γ , λ , ϵ , β ▷ § 2

1: policy model P θ ← P θ init , value model V ψ ← V ψ init

2: for step = 1, . . . , M do

3: Sample a batch D b from D

n

4: Sample output sequence y n ∼ P θ ( · | x n ) for each prompt x n ∈ D b

5: Compute rewards { r n t } | y | t =1 for each sampled output y n by running R ϕ k ▷ Eq. 1

6: Compute advantages { A t } | y n | t =1 and value targets { V targ ( s t ) } | y n | t =1 for each y n with V ψ

7: for PPO iteration = 1, . . . , µ do

8: Update the policy model by maximizing the PPO clipped surrogate objective:

9: Update the value model by minimizing a square-error objective:

Output P θ

B.2 Implementation Details

Model architectures. For the detoxification experiments, the policy model is initialized with GPT2large [31], and the value model is initialized with GPT2-base. For the long-form QA experiments, the policy model is initialized with a supervised fine-tuned T5-large [32], and the value model is initialized with T5-base. This design follows InstructGPT [29], which uses a larger (175B) policy model, and smaller value and reward (6B) models.

Training details on detoxification. For both the holistic reward baseline and the sentence-level (fine-grained) reward, we do a hyper-parameter search with the same set of hyper-parameters. For training, we run 200K episodes. The batch size (number of episodes per card during training) is 64. We use Adam optimizer with a linear learning rate scheduler and 10 warmup steps. We perform a hyper-parameter grid-search for peak learning rate ∈ { 5 e -6 , 1 e -5 , 2 e -5 } , KL coefficient β ∈ { 0 . 1 , 0 . 2 , 0 . 3 } , discounting factor λ ∈ { 0 . 95 , 0 . 97 , 0 . 99 } , and the frequency of exploration (number of sampled outputs) ∈ { 2 , 4 , 8 } . We find that the higher the KL coefficient, the lower the perplexity, and the higher toxicity. This is consistent with findings from previous RLHF studies ([29], [33]). For a fair comparison, we eventually choose a set of parameters that achieve a similar level of perplexity for both reward models. The optimal set of hyper-parameters for holistic reward is β = 0 . 3 , λ = 0 . 99 . For sentence-level reward β = 0 . 1 , λ = 0 . 95 . The learning rate is 1 e -5 , and the exploration frequency is 4 for both experiments. We choose the checkpoint with the lowest validation set toxicity for evaluation. Regarding computation time, we use 2 × 80G NVIDIA A100 GPU for training, and the run time is about 22 hours.

Training details on long-form QA. We conduct a similar hyper-parameter grid search as our detoxification experiments. For long-Form QA, the input length limit is 1024, and the output length limit is 200. Notice that this is much longer than detoxification, so we use a smaller batch size and fewer training episodes. We experiment with multiple combinations of reward model weights. Fixing w 2 = 0 . 5 (factuality reward weight), we perform a grid search on w 1 , w 3 ∈ [0 . 0 , 0 . 5] .

We eventually choose w 1 = 0 . 3 , w 2 = 0 . 5 , w 3 = 0 . 3 , which reaches a balance between three reward models and allows all three rewards to increase during training. For training, the batch size (number of episodes per card during training) is 32. We use Adam optimizer with a linear learning rate scheduler and 100 warmup steps. We perform a hyper-parameter grid-search for peak learning rate ∈ { 5 e -6 , 1 e -5 , 2 e -5 } , KL coefficient β ∈ { 0 . 1 , 0 . 2 , 0 . 3 } , discounting factor λ ∈ { 0 . 95 , 0 . 97 , 0 . 99 } , and the frequency of exploration ∈ { 2 , 4 , 8 } . The optimal set of hyperparameters for Pref. RLHF is β = 0 . 2 , λ = 0 . 99 . For FINE-GRAINED RLHF, β = 0 . 3 , λ = 0 . 95 . The learning rate is 1 e -5 , and the exploration frequency is 4 for both experiments. we run 80K episodes, which is approximately 5 epochs. We choose the checkpoint with the highest validation reward for evaluation. Regarding computation time, we use 2 × 80G NVIDIA A100 GPU for training, and the run time is about 15 hours.

A note on the error bars. All results we report in the paper are from 3 independent runs. The scores reported are all averaged across all runs. The error bars are represented as the shades behind each training curve in our figures. It shows the standard error across three runs.

C Long-Form QA Data and Human Feedback Annotation

C.1 Data Construction

ASQA [39] is a long-form QA dataset that focuses on answering ambiguous factoid questions in an open-domain setting that requires passage retrieval from a given Wikipedia passage corpus. We reformulate it into a reading comprehension setting: given the input x that contains a question q and a set of knowledge passages P = { p 1 , ..., p | P | } , generate a long-form response y . To construct P for each input x , we use the oracle knowledge contexts provided by ASQA for each x , that are text snippets from the passage corpus. We use BM25 4 to map each knowledge context (text snippet) to the closest passage from the passage corpus and use the resulting passages as P . Our train and dev examples come from the original ASQA train set and our test examples are the original ASQA dev examples.

C.2 Human Feedback Annotation

Fine-grained feedback. As discussed in § 4.1, we first use 1K randomly sampled training examples to train a T5-large based supervised model SFT as the initial policy model P θ init . Then we collect feedback on sampled outputs from SFT for the remaining 2,853 training examples and the 500 development examples, using the Amazon Machanical Turk platform. 5

Figure 5 shows the fine-grained human feedback annotation interface with an example from QAFEEDBACK. In addition to the task input-question q and oracle passages P , we also provide a human-written response from ASQA to the worker as reference. However, it is important to note that, in practice, the annotation of our fine-grained feedback should not require the human-written response. The only purpose for us to provide the gold response is to have our workers follow the same question interpretation and expected response of the workers who annotate for ASQA, such that our experimental comparison with supervised models ( SFT and SFT-Full ; details in § 4.3) is fair. However, we still instruct our workers to strictly use the given passages for checking factual errors. For each span error, we ask the worker to select one out of 5 categories shown in Figure 6 (left). 6 However, we collapse these 5 categories into two categories ( C 1 and C 2 mentioned in § 4.1) based on whether the error detection depends on the passages or not. When workers mark passage sentences as containing missing information, we instruct them to categorize each sentence as missing 'answer', 'major auxiliary information' or 'minor auxiliary information,' as shown in Figure 6 (right). Our instruction to the worker is provided in Figure 8.

Quality control. Before feedback collection, we design a qualification task to select qualified workers for this feedback annotation task. The qualification task consists of 5 questions with their corresponding passages and model outputs for the workers to annotate. We manually review about

70 submissions of the qualification task and select 15 workers whose annotation is marked by us as of high quality. Throughout the actual feedback annotation process, we constantly monitor the annotated data and send feedback to workers.

Preference-based feedback. For comparison purposes, we follow [29] to collect pairwise human preferences from the same group of workers we select from the qualification task. We sample four model outputs for each prompt x , which gives 6 pairs of model outputs. Similarly, we provide the worker with the human-written response and ask the workers to indicate pairwise preferences (ties are allowed) based on all errors they can find each model output. Figure 7 shows the preference-based human feedback annotation interface with an example from QA-FEEDBACK.

Pay structure. We pay a base rate of $1.5 per example for annotating fine-grained or preference feedback. If the example consists of ≥ 3 passages to read, we assign an additional $0.3 bonus to the example. On average, we pay roughly $1.65 per example for both tasks, which gives an $16.5 hourly pay for our workers.

C.3 Analysis of Collected Fine-Grained Feedback

Overall, among all error spans we collect, 76% of them are C 1 errors and the remaining 24% are C 2 errors. However, it is important to note that we instruct workers to label C 2 errors only at places that don't have a C 1 error. 75% examples are labeled as being incomplete; i.e., containing missing information that can be found in the given passages ( C 3 ). Among all marked passage sentences that contain missing information, 31%, 42% and 27% are missing answer, major auxiliary information and minor auxiliary information respectively.

To analyze human-human agreement, a subset of 300 examples receive annotations from two distinct workers. We observe that while the exact agreement in error span boundaries is low, workers achieve reasonably high agreement on whether a sub-sentence contains C 1 (reach an agreement for 83% of all sub-sentences) and whether a sentence contains C 2 (92%). 7 The agreement on whether a model output contains complete information or not ( C 3 ) is 85%. Therefore, we decide to have the density for error type C 1 , C 2 , and C 3 as sub-sentence, sentence and full sequence.

D Long-Form QA Reward Model Training Details

We train reward models with the 2,835 training examples with feedback collected and select the best model for each error category based on the their performance on the development set. The batch size and training epochs are 24 and 50 for R ϕ 1 and R ϕ 2 . Each training is run on a single 80G NVIDIA A100 GPU, taking 1 and 2 hours for training R ϕ 1 and R ϕ 2 respectively. 8 The batch size and training epochs are 12 (per GPU) and 30 for R ϕ 3 and the preference-based reward model. Each training is run on 2 × 80G NVIDIA A100 GPU and takes 2 hours. We use Adam optimizer with a linear learning rate scheduler for all reward model training. For each reward model, we search the learning rate over { 5 e -6 , 1 e -5 , 5 e -5 } , weight decay over { 0 . 001 , 0 . 01 } , and warm-up step ratio over { 0 . 1 , 0 . 2 } based on the dev set performance. Specifically, we use the model for R ϕ 1 and R ϕ 2 that achieve the best binary classification accuracy. For R ϕ 3 and the preference-based reward model, we select the model that achieves the best pairwise comparison accuracy. We also provide more training details for each reward model below.

R ϕ 1 for C 1 : Irrelevance, repetition, or incoherence. To train the reward model R ϕ 1 that detects error of irrelevance, repetition, or incoherence, we apply a token-level classification loss to each [sep] token before y 1 j , where its gold label g j is ' has error ' if there is a f i ∈ F that has a b i ,...,e i overlapped with y 1 j and c i = 1 , and ' no error ' otherwise. We observe that most of the spans marked as error type C 1 that are shorter than 5 words usually carry very little information or are annotated as a result of workers being very careful or strict. Therefore, we filter out such short spans before constructing training examples for R ϕ 1 . Overall, we get 7379 and 8059 sub-sentences with the ' has error ' and ' no error ' label respectively.

R ϕ 2 for C 2 : Incorrect or unverifiable facts. We train R ϕ 2 in a similar way as how we train R ϕ 1 . Instead of predicting the error for each sub-sentence, R ϕ 2 is trained to predict at the sentence level (i.e., y 2 j is the j th sentence in y ). Since workers do not annotate C 2 error for spans that are already labeled as having C 1 error, in order to avoid false negatives in training R ϕ 2 , we do not provide gold label nor calculate loss for sentences that only contain C 1 error from training. In other words, all sentences that contain a C 2 error has the gold label ' has error ' and sentences that contain no error has the gold label ' no error '. Overall, we get 1600 and 3411 sentences with the ' has error ' and ' no error ' label respectively.

R ϕ 3 for C 3 : Incomplete information. Instead of casting this as a classification task, R ϕ 3 predicts a single scalar reward given x and y and is trained with a pairwise comparison loss [29]. This is motivated by early work [19] that shows the better reliability of pairwise comparison than error classification when assessing a full generation sequence. To construct training data for R ϕ 3 , we bootstrap pairwise comparisons from the corrected model output y ′ as follows. We first map each sub-sentence in y ′ to a passage sentence in P that has a sub-string with the highest token-level F1 score with the sub-sentence, 9 and denote all mapped sentences as S . We then sample four responses from SFT, for each we do the same sentence mapping to get a set of passages sentences S ′ . We calculate score ( y ) = |S ′ ∩ S| / |S| as the information completeness score for each model response y . We follow [29] to pair up sampled responses for q and denote each sampled response pair as ( ¯ y p , ¯ y l ), where score (¯ y p ) > score (¯ y l ) . We drop the pairs where score (¯ y p ) = score (¯ y l ) . Then we follow [29] to train R ϕ 3 with the loss function in Eq. 2. We have a total number of 6821 pair examples in training.

Preference-based reward model. The preference-based reward model is trained in a similar way as R ϕ 3 , with ¯ y p representing the human preferred response against ¯ y l in the loss function Eq. 2. We drop the pairs where a tie is indicated. We have a total number of 14981 pair examples in training.

Input Question:

Whenwastheparliamentformedintheuk?

Passage1:Title(So)-Parliament of theUnitedKingdomClick tohide

S1. The Royal and Parliamentary Titles Act 1927 formally amended the name to the "Parliament of the United Kingdom of Great Britain and Northern Ireland",five years after the secession of the

IrishFree State in 1922.

S2. With the global expansion of the British Empire, the UK Parliament has shaped the political systems of many countries as ex-colonies and so it has been called the “Mother of Parliaments".

S4. In theory, the UK's supreme legislative power is officially vested in the Crown-in-Parliament.

S5.However,theCrownnormallyactsontheadviceof thePrimeMinisterandthe powers of theHouseof Lords are limited toonlydelayinglegislation;thuspoweris"defacto"vestedinthe

House of Commons.

S7. The principle of ministerial responsibility to the lower House did not develop until the 1gth centurythe House of Lords was superior to the House of Commons both in theory and in practice.

S8. Members of the House of Commons (MPs) were elected in an antiquated electoral system,under which constituencies of vastly different sizes existed.

The parliament formed in the UK was initially called Parliament of the United Kingdom of Great Britain and Ireland but the Royal and Parliamentary Titles Act 1927 formally amended the name to

the“Parliament of the United Kingdom of Great Britain and Northern Ireland",five years after the secession of the Irish Free State.Parliament of the United Kingdom of Great Britain and Ireland

was created on 1 January 18o1, while the parliament formed in the United Kingdom of Great Britain and Northern Ireland was created on 11 November 1920.

STEP1:Errorlabelingforthemodel-predictedresponse:Clicktohide*Hint:The mistake labeling prompt window is draggable**

After CAREFUL checking,Ifind no error in this predicted response!

The United Kingdom of Great Britain and Ireland was created on1 January 18o1,by the merger of the Kingdoms of Great Britain and Ireland under theActs of Union 18oo.On 1January18o1 the

parliament was formed in London.

xUnverifiable

xIrrelevant

*Hint:Iftheerrortypeselection doesn't work,trydoubleclicking.*

xMissing Major Auxiliary Info:[explanation] "passage_id":1,"sentence_id":[1]}

STEP2:Yourcorrectionofthemodel-predictedresponse:Clickto hide

TheRoyalandParliamentaryTitleAct9formallyamendedthenametotheParliamentftheUnitedKingdomofGreatBritainandNorthernrelandivyearsafterthsecessionoftheIrish

Free State.Parliament of the United Kingdom of Great Britain and Ireland was created on 1 January 1801.

Reset

Selectthemissinginfoerror.

Selected span:

by themergerof theKingdoms of GreatBritainandIrelandundertheActs of

Union1800.

MissingAnswer

Missing Major

Auxiliary Info

Missing Minor

Auxiliary Info

Selectthefactual/languageerror.

Youfindapieceofmissinginformationin:

Irrelevant

Repetitive

Inconsistent

Fact

Unverifiable

Fact

Incoherent

passage #

sentence #

(in rare cases, list multiple sentences separated with comma if necessary)

CONFIRM

CONFIRM

Your Task:Make pairwise comparisons on the following6model prediction pairs based on their overall quality.

***Hint:The6pairs are diffeent combinationsof4model predictions.We index eachmodel prediction and highlight them with different colorsfor yourconvenience.**

Click tohide

Compare Model Prediction #1 VS #2-O #1is better Oequal○ #2 is better

Pred #1: The United Kingdom of Great Britain and Ireland was created on 1 January 18o1,by the merger of the Kingdoms of Great Britain and Ireland under the Acts of Union 18oo. On 1 January 1801

the parliament was formed in London.

Pred #2:The United Kingdom Parliament was formed on 1 January 18o1,by the merger of the Kingdoms of Great Britain and Ireland under theActs of Union 18oo.In 1927,the Royal and

ParliamentaryTitlesAct changed the name of theParliament to theParliament of the UnitedKingdom of GreatBritain andNorthernIreland.

Clicktoshow

Compare ModelPrediction #1VS#3-O#1is better

requalO#3isbetter

Clicktoshow

CompareModelPrediction #1VS #4-O#1is betterequalO#4is better

Click to show

CompareModel Prediction #2VS #3-O #2is better

requalO#3is better

Clickto show

Compare Model Prediction #2VS #4-O #2 is better  equalO #4is better

In eachtask,youwillbegivenaquestion,asetofWikipediapassages(withtheirarticletitleprovided),areferenceresponse,andamodelpredicted response.Each passage is presented as a sequence of sentences (titleis indexed as sentence #o).Your goal is to 1) label mistakes made in the model prediction and 2)make corrective edits to the model prediction based onyourlabeled mistakes.

ImportantDefinitions:Anidealresponseto a question should providebothanswer(s)that directly respondstothe questionandcrucial auxiliaryinformationforbettercomprehension of the answer(s).Weconsider auxiliaryinformationascrucialif itisusedinthereference response.Additionally,all informationin an idealresponse should befactually consistent with(i.e.,groundedin) the passages.Note that thereferenceresponse is writtenbya human with potentially different grounding passages,and thus,you might find answers that canbe found in the passagesbut are not included in thereference,which are STILL expected tobe in the model prediction.On the otherhand,answers in the reference that cannot be found in orverifiable bythe passages areNOTexpected tobe inthemodel prediction.To conclude,all answers are expected in the model prediction IF AND ONLY IFit can be found inthe passages.Crucial auxiliary information is expected inthemodel predictionIFANDONLYIFit canbefound inboth thereferenceresponse and the passages.

Here arethedetailedannotationsteps:

STEP1-Read the question andlabel mistakesmadeina model-predicted response.As explained above,leverage thereference,BUTrely on the passages. Decide the mistake type and follow detailed instructions as follows. We encourage you to use CTRL+F/CMD+F for navigating the referenceandpassagesusingkeywords.

·ErroneousSpan(i.e.,substring):Highlight each span in the model prediction that contains one of thefollowingerrors.Label each span asshort aspossible andmakesure eachlabeledspanonlycontain onesingleinformationpiece.Youwill beprompted toselecttheerror based on the orderbelow(e.g.,if a span contains"irrelevant"information,ignore any"inconsistent fact"it contains).

i.[Irrelevant]:The span contains"irrelevant"information(e.g.,neither an answer nor crucial auxiliary information, defined in the first 2sentences in"Important Definitions").To detect"irrelevant"errors,you donot need toconsiderwhether theinformationis factually consistentwiththepassagesornot.

Note that a"repetitive"span should still bemarkedeven if itsprevious text beingrepeated contains anfactual orcoherenceissue (defined below).However,if the previous text is"irrelevant",it should be marked as"irrelevant"too.

ii.[InconsistentFact]:The span isfactually inconsistent with the passages.Enter the passageid and sentenceid(s)as evidence.Note that if you find multiple evidences in the passages,mark only one of them.The need for multiple passage ids usuallyindicates that you should separate the errorintomultiple ones(due to multiple information pieces).

iv.[Unverifiable Fact]: The span is factually unverifiable (i.e., not mentioned in any passage), after carefully checking all passages. Common sense(e.g.,"abicyle has twowheels") doesn't need tobe verified.However,do not count knowledge only commonlyknown in a specific region/community as commonsense.This canbe subjective,and simply follow your best judgment.

notcoherentwithitscontext.

·MissingInformation:Identifyinformation thatis expectedbut missinginthemodel prediction.Check"ImportantDefinitions"above to seehowtoidentifysuchinformation.Classifyeachpieceofmissinginformationas[MissingAnswer]or[MissingMajor/Minor AuxiliaryInformation], and enter the passage id and sentence id(s) as evidence.Mark the missing auxiliary information as majorif you think the information is indeed helpful for understanding the answer. Otherwise (e.g.,a bit off-topic),mark it as minor.Simply follow your bestjudgment.Followthesame"Notethat"ruleunder"[InconsistentFact]"above.

STEP 2 -Correct the model prediction to address your labeled mistakes in STEP 1. Instead of copying and pasting the reference into the box, make minimal local edits onthe original prediction.Make sure the corrected version is fluent and free of typos.In rare cases when you find no useful information in passages for answering the question, you can leave the box as blank if necessary.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes *FINE-GRAINED RLHF*: an extension to (the now popular) *reinforcement learning from human feedback* (RLHF) that allows incorporating more granular human feedback into the language model (LM) training process. Typically, RLHF leverages reward/preference models that are trained to output a *scalar* reward for the whole generated sequence, and this *reward sparsity* can hinder improvements for long-sequence generation. Furthermore, scalar rewards do not offer insights into which parts of the sequence were problematic or displayed undesired behaviors.\n\nTo address this, the using *fine-grained reward models*: for each \"category\" of feedback of interest, a separate reward model is trained. Furthermore, these models generate feedback at lower granularity (for example, at sentence-level). The policy LM can then be optimized with respect *weighted* sum of the rewards at each timestep, using standard RL algorithms like PPO.\n\nThe authors explore the impact on two language generation tasks: (1) detoxification and (2) long-form question answering.\nFor detoxification, the authors find that (with a reward based on an external API) training with rewards provided for every sentence yields better performance when compared to a holistic sequence-level reward. However this evaluation is fully automatic.\nFor long-form question answering (QA), the authors introduce a novel dataset, QA-Feedback, accompanied with fine-grained human feedback on initial policy LM. The feedback comes in three error categories at different levels of granularity: (a) irrelevance, repetition or incoherence (sub-sentence level), (b) incorrect or unverifiable facts (sentence level), and (c) incomplete information (whole sequence level). They collect this feedback through crowd-workers. They also collect *preference-based* feedback, for comparison with (hollistic) RLHF.\nBy training separate reward models for each category, the authors show that fine-grained RLHF outperforms the traditional RLHF and supervised baselines, both in automatic and human evaluation. Interestingly, by carefully adjusting the weights of different reward models, the authors show LM behaviors can be fine-tuned to optimize different combinations of desired behavior, and outperform. The authors also explore the reward model accuracy and even compare against a ChatGPT baseline.\n\nSTRENGTHS:\nI really enjoyed reading this paper. It explores a very prominent topic recently (learning from human feedback), it is well written and it does an very extensive analysis (at least for one of the tasks it tackles). “Reward sparsity” is a well known problem in RL, and recent works in RLHF for LLMs just kinda ignores it. Their proposed fine-grained RLHF is seems like a good approach to tackle it, makes intuitive sense and seems easy to implement. Their findings surrounding controllability could also have significant implications for (RLHF’d) LLMs. They also introduce a novel dataset for QA with fine-grained feedback.\n\nWEAKNESSES:\nThe main weakness of this paper was that only a single LLM was tried: while I believe that their findings will generalize to other LLMs, an experimental validation of this would make this paper even stronger. Even trying re-using the reward models trained for the initial GPT-2 policy on some other LLaMA would already be quite a bonus. \n\nAlso a couple of other minor ones:\n\n- I think task 1 (detoxification) in general is not a good setup: in uses closed source APIs for model-based (from a relatively weak one) PPL as evaluation. the simulated “sentence-level” reward is also not very elegant. I would ideally suggest trying to replace it with another simple, controllable generation task and preferably using human-eval. But ye most of these are covered by task 2\n- The multi-reward aspect of the algorithm makes RLHF now a sort of *multi-task* optimization problem, and bring with the problem of how to weight the different categories. The discussion on the trade-off in (4.5) is interesting and helpful, but I think more could be said on how to pick a weighting for a general case where we just want a single “quality”. Also a very simple pareto analysis (probably trying more weightings of the reward) and some plots showing the final trade-off (ideally in terms of human reward?) would add more technical insight\n\nQUESTIONS:\n- The levels of granularity in the paper make sense for the three categories considered, but have you thought about the more general problem of allowing arbitrary granularities? It seems it is mostly a UX/feedback collection process, and that nothing in the algorithm would prevent it.\n- Is there plans to open-source the (fine-grained) RLHF code? Integrating this into popular packages for RLHF (like TRLX) could heavily boost the impact of this work\n\nLIMITATIONS:\nSee above", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies how fine-grained RLHF improve over holistic preference RLHF.\nThe study is conducted on two tasks and fine-trained reward comes from sub-sentence, sentence, and full sequence level annotations on different feedback types (toxicity, relevancy, factuality and completeness).\nStudy show that fine-grained reward improved over holistic preference and demonstrated LM behaviors can be customized using combinations of feedback types.\n\nSTRENGTHS:\n- The paper is clearly written with good visual aid, has sufficient details on datasets, reward model training, descriptions on experiment hyperparameters and frameworks.\n- The paper demonstrates benefits of fine-grained reward model on two tasks, with various reward density and reward types. It is convincing to see  it works across very different problem settings.\n- There is clear and concise analysis on reward model and human rating agreement to understand the quality of reward model.\n- Experiment sections contains relevant ablations that supports main results and analysis of how different types of reward model works together.\n\nWEAKNESSES:\n- (minor) in the detoxification analysis, I'm not fully convinced that the gain comes from reward, it is possible that the API has better accuracy annotating shorter sequence. Some additional analysis similar to your reward model human agreement analysis on task2 could help explain it better. \n- I didn't find the mention of reward model size which I assume would be the same as the value model of t5-base which is smaller than the policy model. In previous works there is advantages of using larger than policy network value/reward function. It might be important to show what is the effect of reward/value model size on fine-grained reward vs holistic reward.\n- (major) on task2, there lacks analysis comparing F.G.RLHF vs Pref.RLHF on holistic preference eval. It would be a lot more convincing if there is gain in that eval. Otherwise it could be argued that these different approaches are optimizing toward very different signals.\n- the comparison ChatGPT using ROUGE score: it is a reference-based score that might not correlated well with human preference.  Using ROUGE score itself to compare these systems are not sufficient or even detrimental. Your eval with rel/fact/comp and the holistic preference are much better metrics.\n\nQUESTIONS:\nSee weakness section.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work improves the reinforcement learning from human feedback (RLHF) with more fine-grained reward signals.\nFor the feedback density, the authors suggest to reward the policy model at sub-sequence level.\nThe experiments on the detoxification task shows that the fine-grained rewards can outperform the holistic sequence level ones.\nFor the feedback diversity, they set three reward categories: 1) **rel.**, irrelevance, repetition, and incoherence, 2) **fact.**, incorrect or unverifiable facts, and 3) **comp.** information completeness.\nThe first two signals, i.e., rel. and fact., are with the sub-sentence and sentence level density, and the comp. is for the full sequence.\nA new QA-FEEDBACK dataset with fine-grained error annotations and pair-wise preference ranks (for the baselien preference-based RLHF) is constucted for the training and evaluation.\nCompared with the preference-based RLHF, the suggested fine-grained one behaves better on all three error types in both human and automatic evaluation.\nThis framework also demonstrates the promising ability of customizing LM behaviors by adjusting the weights of different reward models.\n\nSTRENGTHS:\n1. This work suggests a new perspective to RLHF for LM, where fine-grined and denser reward signals can benefit LMs. \n2. The proposed fine-grained RLHF framework is simple and effective, validated via thorough experiments and analysis.\n3. A new dataset is contributed for the research community, which is carefully created.\n4. The paper is well-written and easy to follow.\n\nWEAKNESSES:\n1. The models used in this paper are relatively limited in size, only including large and base models. I'm curious how your method would scale to larger models, such as the more commonly used 13B and 30B models, and what kind of performance it would achieve. \n2. LM customization is interesting, but lacks human evaluation. I suppose that incorporating human evaluation could help to further understand this.\n\nQUESTIONS:\n1. More different weights settings for the LM customization.\n2. Efficiency and speed evaluations of the suggest method and baselines.\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces FINE-GRAINED RLHF, a framework that enables training and learning from reward functions that are fine-grained in different error categories. Through experiments on detoxification and long-form question answering, the authors demonstrate how learning with this FINE-GRAINED reward function leads to improved performance, supported by both automatic and human evaluation. The paper's contributions include a new framework for fine-grained reward learning, a new dataset for long-form question answering, and experiments demonstrating the effectiveness of the proposed approach. Though, the proposed method still has some limitations in terms of computational cost, algorithm complexity, and task-binding. In all, it has some contributions to the RLHF field.\n\nSTRENGTHS:\n- The paper introduces a new framework for fine-grained reward learning, which is a novel approach to language model training. The authors also provide a new dataset for long-form question answering, which can be used for future research in this area.\n- The paper is well-written and clearly presents the proposed framework and experimental results. The authors provide detailed descriptions of their methodology and evaluation metrics, which enhances the quality of the paper.\n- The proposed framework has the potential to improve the performance of language models by providing more fine-grained feedback during training.\n- The idea is simple but reveals the potential of RLHF at finer grains.\n\nWEAKNESSES:\n- I know that data labeling is cumbersome and expensive, but the 3,853 training data collected in this work is still a bit small, not sure if this is enough to draw the conclusions mentioned in this work.\n- The proposed method is a bit complicated, it will introduce more computational cost in RLHF, and it will introduce more hyper-parameters for fine-tuning.\n- The design of T_i^k is not fully explored, which may have a big impact on the performance of the proposed method.\n\nQUESTIONS:\n- The authors say that \"the higher the KL coefﬁcient, the lower the perplexity, and the higher toxicity\". But only three KL coefficient is tested (i.e., 0.1, 0.2, 0.3) in the experiments, it is not enough to get such a conclusion. Also, what are the results on the generation fluency when the KL coefficient is 0, which means we do not employ a KL penalty? It would be better to explain the KL penalty in more detail.\n- Why do you choose different models for different tasks? I.e., for the detoxiﬁcation experiments, the policy model is initialized with GPT2Large, and the value model is initialized with GPT2-base; for the long-form QA experiments, the policy model is initialized with a supervised ﬁne-tuned T5-large, and the value model is initialized with T5-base.\n- Is the proposed method a good solution in terms of LLM's hallucination?\n\nLIMITATIONS:\n- The training dataset is not large enough (especially for the Long-Form QA task), which limits the verification of a conjecture: As the size of training data increases, the effect of the FG-RLHF proposed in this paper will weaken.\n- The proposed method is expensive, not only in the additional computational cost of the reward model, but also in the different deﬁnitions of ﬁne-grained feedback in terms of the feedback types and the density level of each feedback type.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}