Chain of Hindsight aligns Language Models with Feedback
Authors: Hao Liu
Carmelo Sferrazza
Pieter Abbeel
Venue: ICLR 2024 poster (2024)
Keywords: Reinforcement Learning
Reinforcement Learning from Human Feedback
RLHF
Primary area: reinforcement learning
TL;DR: We present a method for learning from human feedback that's simpler than RLHF

Abstract: Learning from human preferences is important for language models to match human needs and to align with human and social values. 
Prior works have achieved remarkable successes by learning from human feedback to understand and follow instructions. Nonetheless, these methods are either founded on hand-picked model generations that are favored by human annotators, rendering them inefficient in terms of data utilization and challenging to apply in general, or they depend on reinforcement learning, which often suffers from imperfect reward functions and relies on extremely challenging optimizations. In this work, we propose a novel technique, Chain of Hindsight, that is easy to optimize and can learn from any form of feedback, regardless of its polarity. Our idea is inspired by how humans learn from extensive feedback presented in the form of languages. We convert all types of feedback into sequences of sentences, which are then used to fine-tune the model, allowing us to take advantage of the language comprehension capabilities of language models.
We condition the model on a sequence of model generations paired with feedback. By doing so, the model is trained to generate outputs based on feedback, while learning to identify and correct negative attributes or errors.  Applying our method to large language models, we observed that Chain of Hindsight significantly surpasses previous methods in aligning language models with human preferences. We report significant improvements on summarization and dialogue benchmarks, with our approach markedly preferred in human evaluations.

Published as a conference paper at ICLR 2024
CHAIN OF HINDSIGHT ALIGNS LANGUAGE MODELS WITH FEEDBACK
Hao Liu
Carmelo Sferrazza UC Berkeley
Pieter Abbeel UC Berkeley
UC Berkeley
hao. wiwwberkeeeee....... edu
caferrazza&berkeleyy. edu
cetpeel@@ss.berkeeee edu
ABSTRACT
Learning from human preferences is important for language models to match human needs and to align with human and social values. Prior works have achieved remarkable successes by learning from human feedback to understand and follow instructions. Nonetheless, these methods are either founded on hand-picked model generations that are favored by human annotators, rendering them inefficient in terms of data utilization and challenging to apply in gencral, or they depend on reinforcement learning, which often suffers from imperfect reward functions and relies on extremely challenging oplimizations. In this work, we propose a novel lechnique, Chain of Hindsight, that is easy to oplimize and can learn from any form of feedback, regardless of its polarity. Our idea is inspired by how humans learn from extensive feedback presented in the form of languages. We convert all types of feedback into sequences of sentences, which are then used to fine-tune the model, allowing us to take advantage of the language sequence comprehension capabilities of language models. We condition the model of model genorations paired with feodback. By doing so, the model is trained to generate outputs based on feedback, while learning identify and correct negative attributes or errors. Applying our method to large language models, we observed that Chain of Hindsight significantly surpasses previous methods in aligning language models with human preferences. We report significant improvements on summarization and dialogue benchmarks, with our approach markedly preferred in human evaluations.
INTRODUCTION
Large language models have achieved amazing results in natural language understand- ing (Radford et al. 2018 2019 Brown et al.) 20200). However, in order to ensure that these technologies have a positive impact on society, it is paramount importance for them to be aligned with human values. One of the most critical elements in achieving this is the use of human feedback. Human feedback allows us to evaluate the performance of such models in a way that is both objective and subjective. It help identify issues with accuracy, fairness, and bias, and can provide insights into how the model can be improved, in order to ensure that the model outputs align with societal norms and expectations. Driven by the importance of incorporating human feedback into language models, researchers have been developing and testing various methods for human-in-the-loee thotthh-oopp systems. These methods aim to make the process of incorporating human feedback more efficient, resulting in models that are able to achieve improved performance and accuracy, while also providing higher fairness
and more el hical oulputs (Hancock el. al. 2019: Perez et al. 2019 Yi et al. 2019 Ouyang
el al. 2022 Schulman et al.) 2022 inter alia).
The successes in language modeling have been largely attributed to the utilization of supervised finetuning (SFT) and Reinforcement Learning with Human Feedback (RLHF) techniques. While these approaches have demonstrated promising results in enhancing the performance of language models on specific tasks, they also suffer from notable limitations. SFT relics human-annotated data and positive-rated model generation to fine-tune pretrained language model. However, this approach is heavily reliant on the availability of labeled data, which may entail significant expenses and time investments. Moreover, relying
htttps://gttie
http:///gtttub.ccch///////aaaaat http//hhooS//chatn--od hinddggggh

---

Published as a conference paper at ICLR 2024
Base
SFT
C-SFT
SFT-U
RUHF
Tie
CoH
Summarization Task
Dialogue Task
Comparison
25
50
75
100 0
25
50
75
100
Rate
Win Rate
Figure 1: Human evaluation pairwise comparison between CoHl and various approaches the sunmarization and dialogue tasks. Base denotes the pretrained model, SFT-U denotes SFT with unlikelihood loss, C-SFT denotes conditional SFT. CoH substantially outperform reinforcement learning from human feedback (RLHF) and supervised finetuning baselines.
solcly on positive-rated data may constrain the model's ability to identify and correct negative attributes errors, thus reducing its gencralizability to new and unscen data. Alternatively, RLHF enables learning from all data, regardless of feedback rating. Nonetheless, this method requires learning reward function, which may be subject to misalignment and imperfections (Gao et al., 2023). In addition, the optimization of reinforcement learning algorithms can be challenging, presenting significant difficulties in its application.
In this work, we aim to overcome the limitations of SFT and RLHF by combining their strengths to leverage all feedback, without resorting to reinforcement learning. Our key idea is that humans are capable learning from rich and detailed feodback in the form of comparisons. Our hypothesis is that by conditioning language models sequence of generations paired with feedback and training them accordingly, they can learn to identify and correct errors and negative altributes.
Moreover, prior research has underscored the efficacy of pretrained language models for both in context learning and instruction tuning (Radford et al. 2019 Brown al. 2020 Wei al. 2021, inter alia). Building upon these insights, we introduce & novel approach: converting all human foodback into a sequence and subsequently finctuning models to comprehend and effectively utilize such feedback. Specifically, we propose finctuning the model to predict outputs while conditioning on one more model outputs and their corresponding feedback in the form of comparisons to the other outputs.
In essence, our approach finetunes the model by conditioning it to generate outputs while taking into account one or more model-generated outputs and their associated feedback, presented in the form of comparisons to other outputs. During the training phase, the model is given feedback expressions like 'A unhelpful answer' and 'A helpful answer'. It is then tasked with predicting outputs that align more closcly with the foedback, such as in the following example: 'How can you explain neural networks to 6-yearoold? A unhelpful answer: (a subpar answer) A helpful answer: {an excellent. answer).. Furthermore, our framework allows for the integration of natural language feedback, such '(a subpar answer) is less preferred answer compared with {an excellent answer)', which not only informs the model the preference but also provides additional task-specific guidance. At inference time, when presented with positive feedback indicated by 'A helpful answer', the model is guided to generate the desired outputs, thereby ensuring a preferable behavior.
Our proposed approach enables models to learn from both positive and negative feedback, allowing the identification and correction of negative attributes or errors. We name our method Chain of Hindsight. (CoH) as it conditions conducted comprehensive evaluations of our approach the domains of summarization and dialogue tasks, revealing substantial performance enhancements compared to SFT and its various iterations, as well as RLHF, across both automated assessments and human evaluations.
a sequence of hindsight feedback. We
Our main contributions are twofold: (a) We introduce a novel learning framework, referred to as CoH, which effectively harnesses all available feedback data to enhance model performance without necessitating reliance on RLHF. Notably, our approach CoH maintains the same training objective pretraining, rendering it straight to train and readily scalable; (b)
2

---

Published as a conference paper at ICLR 2024
Modal Complction
Necral senwarks are sief io...
How to explain noural networks child?
less prefered
How exploin neceal networlls child??
compared with
GPF
How to explain noural networks to child?
good answer
necrrl sutwaric lite rubor froie.
bed answer
How to explain noural networks to a child? Bad:
Glood
Hindsightt Feedback
Figure 2: Chain of Hindsight (CoH) turns human preferences into rich and detailed feedback in the form of comparisons. In the diagram, we explain this by showing that a question is being prompled to GPT model. The model then generates a multitude of responses, which are subsequently ranked according to human preferences(e.1) e.g., A is less preferred compared with B). Subsequently, we construct CoH sequences by converting human preference into natural language feedback and combine them with the model's outputs. These constructed sequences are then employed the finctuning phase, aligning with the same objectives as in the pretraining phase.
We conduct extensive experiments to showcase the effectiveness of our method in comparison to existing baselines, including state-of-the-art RL.FF methods.
2 CHAIN OF HINDSIGHT
Our goal is to improve the performance of a Transformer- based language model by leveraging human-rated data and feedback, and to achieve this, propose novel approach that goes beyond conventional SFT methods and RLHF methods.
Turning all feedback into a sequence. Our approach aims to take into account all feedback and instructions provided by humans. To achieve this, we present the model with a sequence of model generations, along with corresponding feedback and explanations provided
by humans. Our approach uses a conventional Transformer model architecture that is causal and decoder-only, as proposed in the work of (Brown et al. 2020 Vaswani al. 2017 on attention mechanisms. This means that at each timestep, the model can only attend to the past timesteps and itself. Given a text represented by tokens x [x],-... ,x), the standard causal language modeling objective is defined to maximize the log likelihood of
autoregrossively: log p(x) log II p(z,/x<:). In CoH, we construct by combining multiple
i=1
model outputs with feedback which are then used for instruction finetuning. For instance, when a model prompted to explain neural networks to a child, generates multiple responses to the prompt. These responses are then combined together into sequence and paired with feedback instructions generated based on human ratings. An example is illustrated in Figure 2 During the training phase, the model is presented with both positive and negative feedback denoted as 'Bad' and 'Good', and the model is conditioned to predict outputs that better match the latter feedback such as 'How to explain neural networks to year old? Bad: {a bad answer) Good: fa good answer).'. Furthermore, our framework allows for the integration of natural language leedback, such 'How can you explain neural networks to a G-year-old? Bad: {a subpar answer) Good: {an excellent answer)', which provides additional task-specific guidance and context. By incorporating wider range of diverse positive and negative feedback, it further enhances the model's performance. In this study, we opted for templated feedback generated from ratings rather than open-ended feedback from humans in the loop. The feedback type varies depending on the task, we list the contextual natural language foodback in Appendix B
Natural language feedback examples
A good summary: (positive), worse summary: (negative)
You are a helpful assistant: (positive): you are an unhelpful assistant: (negative) A bad answer is (negative), a good answer is (positive)
3

---

Published as a conference paper at ICLR 2024
In theory, one could employ open-ended feedback from humans in the loop. However, for this study, we chose to generate feedback using pre-determined templates based on ratings. During the inference phase, we prompt the model with positive feedback in the form of 'Good' to guide the model in generating favorable outputs.
To enable models to learn from feedback, we require the model to prodict each token a; that are generated by the model. Loss not applicd on other tokens because it hinders model generation inference time. This is achieved through masking, which can be expressed as:
logp(x) = log II lo(x) (x4) p(=,)(=)1)=0,, where l0(s)) (I;) denotes whether loken 2y not 2=1
part of the hindsight feedback. In other words, it is if r, is not part of the feedback and O if it is part of the feedback. The model is trained to predict each non-feedback token I; given the previous tokens [zy]]=0
Al Algorithm 1 Aligning language models from leedback with Chain of Hindsight.
Required: Pretrained Language Model M, Human Foedback Dataset D
Required: Maximum training iterations Initialize
for iter =1 to n do
Randomly sample a minibatch of model outputs and their associated ratings from dataset D.
Construct training sequences by combining sampled model outputs with feodback based ratings.
Instruct finetune model M on the training sequences. end for
Training. We work with a dataset of model outputs and their corresponding human preference, such as positive and negative ratings, from which we sample minibatches of model outputs. To generate hindsight feedback in natural language, we randomly sample a feedback format and incorporate the human ratings. We combine the hindsight feedback and model outputs into a chain of hindsight, which serves as the input for our autoregressive model. The objective is to predict the input sequence autoregressively, and we use cross-entropy loss to optimize the model. We average the loss over each timestep in the last model output sequence. In the regime of human preference learning, the positive and negative data often being similar to each other (e.g.; the Anthropic helpful and harmless dataset).. Since CoH condition the model on an example when predicting another one, the model can simply 'copy' the example without learning to understand the underlying task. To address this, randomly mask between 0% and 5% of past tokens during training, which help regularize the model and prevent it from overfitting to the spocific examples seen during training (Srivastava) et al. 2014 Liu et al. 2022). In order to retain model's performance on general language modchig tasks, we added regularization term which maximize the log likelihood of the pretraining dalaset. following prior works (Ouyang et al. 2022). We apply this technique to our method and all baselines in evaluation. Our approach is shown in Figure and the
algorithm is summarized in Algorithm
2.1 RELATION TO PRIOR PARADIGMS
We discuss the connections of CoHl to prior paradigms of learning from preference data.
Supervised finetuning (SFT). SFT is a commonly used method for preference learning, involving the use of pasitively labeled data for finetuning (Ouyang et al. 2022: Schulman et al. 2022). Our approach, however, diverges from SFT by incorporating both positive and non positive rated data, as well as utilizing feedback input. In comparison to SFT, CoH leverages a broader spectrum of information.
Conditional SFT. This mothod shares similaritics with the Decision Transformer model (Chen et al.) 2021), which involves conditional training of SFT with feedback serv- ing as prefix tokens. In essence, both CoH and Conditional SET utilize feedback tokens as conditional input. Nonetheless, the distinction lies in CoH' utilization of sequence
4

---

Published as a conference paper at ICLR 2024
of feedback-example pairs, enabling our approach to condition more comprehensive information when making predictions.
SFT with unlikelihood. SFT with unlikelihood introduces unlikelihood loss
negatively rated data (Welleck et al. 2019 Li et al. 2019) to the traditional SFT framework.
Reinforcement learning with human feedback (RLHF). RLHF (Schulman et al. 2022 Ouyang et al.) 2022 Stiennon et al., 2020) entails the acquisition of a reward function based on human preterences and the of reinforcement learning to maximize this reward. In contrast. to RLHF, CoH offers a substantially simpler training process, and as our experimental evaluations will demonstrate, it consistently outperfoms RLHF in terms of performance.
3 EVALUATION SETUP
Training Datasets. We use a combination of three datasets for learning from human feedback. The three datasets are:
WebGPT. The WebGPT dataset Nakano et al. 2021 includes total of 19,578 comparisons where each example comprises a question, a pair of model answers, and metadata. The answers are rated by humans with preference score, which helps to identify the better of the two answers.
HH. The Anthropic's Helpful and Harmless (HH) dataset (Ganguli al. 2022
Bai et al. 2022a contains human rated dialogues Each example in this dataset consists of pair of conversations between human and a languages model, and one of the two conversations is labeled as preferred by human labelers.
Summarization. The summarization dataset (Stiennon et al., 2020) consists of feedback from humans regarding the summarizations generated by a model Human evaluators were requested to choose the superior summary from two options presented to them.
Evaluation Benchmark and Metrics. We consider both automatic evaluation and human evaluation summarization and dialogue benchmarks.
Summarization Benchmark. Following prior RLHF works (Stiennon al. 2020 Nakano et al.) 2021: Bai et al.) 2022a), we consider automatic evaluation and human evaluation on the TL;DRs dataset (Völske et al. 2017). The original TL:DR dataset contains about 3 million posts from reddit. com across a variety of topics (subreddits), as well summaries of the posts written by the original poster (TL;DRs). We use the filtered version provided by Sttienon et al. (2020), which contains 123, 169 posts. We evaluate the performance on the validation set. For evaluation metrics, labelers rated summaries for coverage (how much important information from the original post is covered), accuracy (to what degree the statements in the summary are part of the post), coherence (how casy the summary is read on its own), and overall quality. More details about evaluation dimensions and instructions for human labelers are available in Appendix A
Dialogue Benchmark. We also evaluate on the validation split of the Anthropic's Helpful and Harmless (HH) dataset (Ganguli et al. 2022: Bai et al. 2022a). where where each example comprises a pair of conversations between a human and large language model, with one of the two conversations preferred by human. For evaluating the dialogue, we consider metrics such as helpfulness and harmlessness. helpful model should follow instructions and infer intention from a few-shot prompt another interpretable pattern. Since the intention a given prompt can be unclear or ambiguous, we rely on judgment from our labelers, and the main metric use is the labelers' preference ratings.
To collect data for our evaluation, it would be too costly and time-consuming to deploy our finetuned model to chat with humans. Instead, we construct "pseudo" dialogues using positive examples. We replace each model response from a previous dialogue with our model's output, generated by conditioning the model on the human response and past model outputs. We take this approach instead having humans directly chat with the
https: http:///awaaaatctttttatttttataaa http://cbbbpptccorraaoooooo
http:://wwggnnneee...ooaatttt//p wttp:iieeeeeooeeoteeeeee feadback

---

Published as a conference paper at ICLR 2024
Pretrained
SFT
Conditional SFT
SFT unikalhoodd
ALIF
CoH
35
30
20
15
10
0
Rouge 1
Rouge 2
Rouge
Avg
Metric
Figure 3: Evaluation summarization. Comparison between RLHE, SFT and CoH. The metrics are ROUGE scores on TL;DR summarization task.
finetuned model to reuse human- generated data, as collecting interactive data can be very costly and is prone to low data quality issues. More details about evaluation dimensions and instructions for human labelers are available in Appendix A
Baselines. Our primary baselines are SFT, SFT with unlikelihood (denoted SFT- U), conditional SFT (denoted as C-SFT), and RLHF, for connections between them and CoHl please refer to Section 2.1 We use GPT-J 6B (Wang and Komatsuzaki 2021] and OPT (Zhang et al. 2022 as the base pretrained models. while other language models can also be used. Following prior works (Ouyang ct al., 2022 Schulman et al.) 2022), we adopt the PPO algorithm (Schulman et al.) 2017) ) to implement RLHF bascline. We tune the
hyperparameters of PPO and reward learning to obtain the best possible results. To ensure fair comparison, carefully tune the training hyperparameters for all other baselines.
RESULTS
Table 1: Pairwisc human evaluation on sum- marization task.
Our main goal in conducting these evalua- tions is to assess the effectiveness of our pro- posed methodology, which focuses on sum- marization and dialogue benchmarks. We conduct both automatic and human evalu- ations, in order to benchmark our approach against established baselines, including SFT, conditional SFT, SFT with unlikelihood, and RLHF approach (Ouyang et al. 2022 Schulman et al. 2022)
Human evaluation win rate (%)
Base
Tie
Coll
A
Accuracy Coherence Coverage
24.5
26.8
48.7
24.2
15.6
18.5
65.9
50.3
19.6
22.4
58.0
38.4
Average
19.9
22.6
57.5
37.6
SFT
Tie
CoH
A
Accuracy
25.5
32.6
41.9
16.4
Coherence
30.5
25.6
43.9
13.4
Evaluation summarization. In Fig- ure 3 we present the ROUGE scores our models test set of summarization dataset. Our proposed approach, CoH, substantially outperform baselines, including based pre- trained model, SFT, conditional SFT, SFT with unlikelihood, and RLHFF. Despite the simplicity of our approach, CoH outperforms RLHF across all the metrics. We notice that RLHF performs the second best, with con- ditional SFT closely follows behind.
Coverage
28.5
25.4
46.1
17.6
Average
28.2
27.9
44.0
15.8
C-SFT
Tie
CoH
A
Accuracy
26.7
31.9
38.4
11.7
Coherence
32.5
22.9
41.6
12.1
Coverage Average
29.5
26.7
43.8
14.3
29.6
28.2
42.3
12.7
SFT-U
CoH
A
Accuracy
18.7
17.9
63.4
44.7
Coherence
21.8
15.8
62.4
40.6
Coverage
23.6
17.2
59.2
35.6
Average
21.4
17.0
61.7
40.3
To further evaluate the performance of our proposed approach, we conducted human evaluation as shown in Table I Base de- notes the pretrained model, SFT-U denotes
RLHI
Tie
CoHl
A
Accuracy
31.8
29.5
38.7
Coherence
31.6
20.5
47.9
16.4
Coverage
28.9
21.9
49.2
20.3
SFT with unlikelihood, C-SFT denotes
Average
30.8
24.0
45.3
14.5
ditional SFT. We conducted pairwise comparisens between Coll and the beselmes beentiee
found that doing is easier task for human labelers compared to evaluating multiple

---

Published as a conference paper at ICLR 2024
Pretrained
SFT
Conditional SFT
unlikelihood
RLHF
CoH
0
10
20
30
40
50
60
70
80
Score
Figure 4: Evaluation on dialogue. Comparing CoH with RLHF and SFT baselines. The metric the accuracy of classifying the preferred dialogue.
options the same. We hired 75 human labelers who were proficient in English from third-party platform to provide ratings. In the pairwise comparison, human labelers were presented with two summaries, CoH. They were instructed to select the best (or tie) among the two according to the three metrics mentioned above. The metrics
generated by the baseline and the other generated by
accuracy, coherency and coverage following prior works (Ouyang et al. 2022),, we used the same instructions therein, and additional inst instruct human labelers to select tie, the full details of human evaluation instructions provided in Appendix
A Table | presents the human evaluation results on summarization task. CoHl substantially outperform RLHF and conditional SFT, showcasing the effectiveness of CoHl in aligning language models with human preferences.
Evaluation on dialogue. We evaluate our method on the HH dataset, by testing its ability to classify which of dialogue pair is preferred. Figure 4 presents the comparison between baselines and our method. SFT shows substantially improvement over base pretrained model; adding unlikelihood de- grades performance which indicates unlike- lihood hurts model generation ability; con- ditional SFT shows improvement over SFT, showcasing the benefit of learning from neg- ative examples; RLHF performs second best and is substantially outperformed by our CoH. The results demonstrate the effective- noss of CoH in lcarning from preferences. We further evaluate on the dialogue task based on HH dataset. We use the same setting of 75 human labelers and pairwise comparison as in the summarization human evaluation. For this task, we provide human labelers with instructions to evaluate whether the answer helpful and harmless (Bai et al. 2022ah. The results are presented in Table Coll substantially outperform RLHF and conditional SFT, showcasing the effective- ness of CoH in aligning language models with human preferences.
Table 2: Pairwise human evaluation on dia- logue task.
Human evaluation win rate (%)
Base
Tie
CoH
A
Helpful Harmless
15.8
34.8
49.4
33.6
14.5
35.9
49.6
35.1
Average
15.2
35.3
49.5
34.4
SFT
Tie
CoH
A
Helpful Harmless Average
19.6
45.7
34.7
15.1
18.6
37.4
44.0
25.4
19.1
41.5
39.4
20.3
C-SFT
Tie
CoH
Helpful Harmless Average
21.8
46.9
31.3
9.5
22.4
35.2
42.4
20.0
22.1
41.0
36.8
14.7
SFT-U
Tie
CoH
A
Helpful Harmless
13.4
31.3
55.3
41.9
14.5
28.7
56.8
42.3
Average
13.9
30.0
56.0
42.1
RLHF
Tie
CoH
Helpful Harmless
25.8
40.8
33.4
7.6
20.9
38.8
40.3
19.4
Average
23.4
39.8
36.9
13.5
Language feedback. We enhance the effectiveness of our approach by evaluating its performance in the context of binary feedback alone, as opposed to the combination of binary feodback and fine-grained language feedback, which is the default setting of our method. We denote this baseline without natural language feedback as CoHl w/o LF. assess the performance of these variations, conducted human evaluation task focused on the summarization domain, employing the input of 75 human evaluators. The outcomes, presented in Table show that both default approach and our 'w/o LF variant substantially outperform RLHF. In addition, our findings indicate that the inclusion of natural language feedback enhances the results. Human preference ratings show 14.1%

---

Published as a conference paper at ICLR 2024
Pretrained
SFT
Conditional SFT
SFT unlikellhood
RLHF
CoH
25
Score
15
10
GPT-2 0.58
GPT 1.5B
OPT 2.7B
GPT-J 6B
Figure 5: Model scaling trend. Comparing CoH with RLHF and SFT basclines on summarization benchmark with different model sizes. CoH outperforms RLHF, showing strong scaling capabilities.
preference for models with language feedback, whereas models without language feedback received an 11.6% preference. The results demonstrate the effectiveness of our CoHl framework. Since the framework CoH offers flexibility to incorporate natural language feedback into training, designing more effective natural language feedback is one of our future directions.
Evaluation model scaling trend. To assess the efficacy of CoH across various model sizes, we conducted comprchensive evaluation. The findings in Figure 5 demonstrate the impact of varying model sizes on the performance of the CoH method relative to SFT baselines and RLHF. Notably, for smaller model sizes, CoH exhibits a marginal decrement in performance compared to SFT baselines. However, the model size increases, CoH consistently surpasses all SFT and RLHF basclines and displays posi- tive scaling trend, indicating its efficacy in enhancing model performance as model complexity increases.
Table 3: Ablation study of natural language feedback on summarization task based on human evaluatoon.
Average win rate (%)
RLHF
Tie
Coll
30.8
24.0
15.3
RLHF 32.1
Tie 26.5
CoH w/o L.F
42.4
CoH w/o LF
Tie 74.3
CoH 15.1
10.6
5 RELATED WORK
Learning from hindsight. In this paper explore learning from chains of hindsight with human feedback, an approach that enables model to learn from errors and revise generations. The key idea of learning from hindsight experience was explored in goal conditioned RL (Kaelbling, 1993 Andrychowicz et al. 2017 Schaul et al. 2015 Andrychowicz et al. (2017) proposes hindsight experience replay (HER) to relabel rewards and transitions retroactively to learn from sparse feedback. While HER relies on reinforcement learning and distance function to learn from hindsight experience, we propose a new method called CoHl that constructs a chain of hindsight experience using human feedback and finetunes the model directly. Our approach offers several advantages over other methods, such as HIR (Zhang et al. 2023). which also makes use of incorrect model outputs. HIR can be seen a special case CoH with length of one chain-of- of-hindsight. Unlike HIR, which employs complex training process involving likelihood loss, cont.asti ive loss, and entropy loss, our approach is straightforward and easy to implement. Concurrently, Korbak et al. (2023) studies conditioning on human preference during pretraining and shows improved performance in aligning language models with human preference. Their method is similar to CoH with length of one Maii--f-hhnndiiee Our work focuses finctuning pretrained language models while Korbak et al. (023) focuses on improving pretraining.
Learning from human feedback. Prior work have explored using human feedback to improve various tasks, such as summarization [Bhm et al. 2019 Ziegler et al. 2019
Stiennon et al. 2020), dialogue (Yi al. 2019 Hancock et. al. 2019 Bai et. al. 2022a
8

---

Published as a conference paper at ICLR 2024
2018 Bahdanau et al. Askell et al. 2021 Scheurer et al. 2022), translation Krentzer et al.
2016), semantic parsing (Lawrence and Riezler 2018). story generation (Zhou and Xu 2020).
review generation (Cho et al. 2018), evidence extraction (Perez al., 2019). and instruction following (Ouyang et al.) 2022: Bai et al. 2022a). The main techniques behind them can be categorized as supervised finetuning (SFT) or training on filtered human annotations and learning a reward function from human feedback for reinforcement learning, which is often dubbed as RLHF (Christiano et al. 2017; MacGlashan et al. 2017 Lee et al., 2021 Warnell ct al. 2017] and has been used to train RL agents without the need for hand-designed rewards. Ouyang et al.] (2022) demonstrates improved language model alignment performance by training models with S and RLHF using human leedback. Our work belongs to the category of SFT, and differs from SFT in that our method conditions on leedback and can learn from examples without positive ratings. Our method is complementary to RLHF and can be directly combined together for further improvement. Using instructions to provide models with human preference and desired behaviors is demonstrated in Bai et al. (022b), where models are prompted with a set of statements/ statements/principle and are trained with RLHF In our work, we provide models with a sequence of model outputs and their feedback and train models to generate desired outputs conditioned on feedback /control tokens.
Instruction finetuning and conditional training. Finet.uming on chain of hindsight using human leedback is akin to instruction finetuning. Driven by the impressive in-context learning ability of large language models, finetuning pretrained models on instructions has been shown to improve language models in many benchmarks (see e.g. Wang et al.
2022
Mishra et al. |021 Ye et al.] 202T Chung et al., 2022 Wei et al. 2021 Sanh et al.
2021
Zelikman et al. 2022 Huang et al. 2022 inter alia). Mostly the instructions are
reformatted examples from NLP benchmarks (c.g. Wei et al. 2021 Chung et al., 2022).
CoT prompis (Wei et al. 2022) are widely considered as instructions in prior works (Chung
eel. al. 2022; Wei et al.; 20210, specifically in the form of step by step explanations writlen by humans. In relation to these, our chain of hindsight consists of human written hindsight
feedback and ranked model outputs. Conditional training (Keskar et al. 2019 Ficler and
Goldberg 2017 Laskin et al. 2022: Chen et al., 2021: Fan et al. 2018 explores conditioning the model on some control tokens for controllable generations. In relation to it, CoH generalizes to condition control token. By doing so, CoH enables the model to understand the differences wetween control tokens and their correspouding outputs. Our work suggests a promising direction of using hindsight feedback to construct instructions from model outputs, and can be combined with prior instruction finetuning and conditional training works for further improvements.
Lu et al. 2022
sequence of control tokens instead of one
CONCLUSION
In conclusion, we introduce Chain of Hindsight (CoH), which is inspired by how humans learn from rich feedback in the form of comparison. We condition language models on sequence of hindsight feedback, allowing them to effectively leverage all examples regardless of their preference score. Extensive experiments on summarization and dialogue datasets show that CoH substantially outperform RLHF and other baselines.
Limitations and Future Work. Although our method substantially outperform baselines, it does have some limitations that need to be addressed:
Constructing CoH may result in long sequences, particularly with multiple feedback instances, leading to increased training computational expensess
Our work heavily relies hired human labelers for evaluation due to their higher reliability compared to automated metrics. However, this approach incurs substantial costs, alt hough this issue is not unique to our method.
In terms of future prospects, our CoH--based training from human feedback opens the door to exciting possibilities, such as integrating external environment feedback like unit tests and extending its applicability to various domains. Furthermore, current focus on learning from hindsight using preexisting preferences paves the way for exploration in online preference learning, enabling iterative model improvements.
9

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This project is supported in part by Office of Naval Research grant N00014-21-1-2769 and SNSF Postdoc Mobility Fellowship 211086 and ONR MURI N00014-22-1-27753 We express our gratitude to the BAIR communitics for their insightful discussions and feedback. We thank Google TPU Research Cloud for granting us access to TPUs.
REFERENCES
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Picter Abbecl, and Wojeiech Zaremba. Hindsight experience replay. Advances in newral information processing systems, 30, 2017.
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Doep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, al. A general language assistant as a laboratory for aligument. Xiv preprint arxiv:2112. 00861, 2021.
Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron Courville, and Yoshua Bengio. An actor-critic algorithm for sequence prediction. or Xie preprind arxivv1600 160700706, 2016.
Yunlao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, el. al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint ar orx:::2004, 058662, 2022a.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, al. Constitu- tional ai: Harmlessness from ai feedback. arXiv preprint arXiv:221 08073, 2022b.
Florian Böhm, Yang Gao, Christian M Meyer, Ori Shapira, Ido Dagan, and Iryna Gurevych. Better rewards yield better summaries: Learning to summarise without references. arXiv preprint arXiv: 1909.01214, 2019.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot learners. Advances in newral information processing systems, 33:1877 1901, 2020.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lec, Aditya Grover, Misha Laskin, Picter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34: 15084-15097, 2021.
Woon Sang Cho, Pengehuan Zhang, Yizhe Zhang, Xiujun Li, Michel Galley, Chris Brockett, Mengdi Wang, and Jianfeng Gao. Towards coherent and cohesive long-for formf text generation. arXw preprint arXiv:1811.0051 2018.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodci. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, pages 4299-4301 4307, 2017.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling inst muctiionn-ffelleeee language models. aixiv preprint arXim:2210. 0.1116 2022.
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. arXw preprint arXi: Arziv-1805.00
2018.
Jessica Ficler and Yoav Goldberg. Controlling linguistic style aspects in neural language generation. Xio preprint arxiv: Arxiv- 1 1707.02633, 2017.
10

---

Published as a conference paper at ICLR 2024
Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint ar Xrx: 07858, 2022.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Lealy. The Pile: An 800GB dataset of diverse text for language modeling. Computing Research Repository, arXiv:2101.00027, 2020, version
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages 10835-10866. PMLR, 2023.
Xinyang Geng, Arnav Gudibande, Hao Liu, Eric Wallace, Picter Abbecl, Sergey Levine, and Dawn Song. Koala: A dialogue model for academic research. Blog post, April, 1, 2023.
Braden Hancock, Antoine Bordes, FimraneF Mazare, and Jason Weston. Learning from dialogue after deployment: Feed yourself, chatbot! arXiv preprint 0rXi::1001.0544 2019.
Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can smlfovem arxiv preprint arXiv:2210.11610 2022.
Leslie Pack Kaelbling, Learning to achieve goals. In IJCAI, volume 2, pages 1094 8. Citeseer, 1993.
Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation. PREPRINT; 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rv: 6980, 2014.
Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Bhalerao, Christopher L. Buckley, Jason Phang, Samuel R Bowman, and Ethan Perez. Pretraining language models with human preferences. arXiv preprint arXi::2302.0855
2023.
Julia Kreutzer, Shahram Khadivi, Evgeny Matusov, and Stefan Riezler. Can neural machine translation be improved with user feedback? arXiv preprint XXi::180400595 2018.
Michael Laskin, Luyu Wang, Junhyuk Oh, Emilio Parisotto, Stephen Spencer, Richie Steigerwald, DJ Strouse, Steven Hansen, Angelos Filos, Ethan Brooks, Maxime Gazcan, Himanshu Sahni, Satinder Singh, and Volodymyr Mnih. In-conlext reinforcement learning with algorithm distillation. ar preprint aiv: Arriv-2210. 14215, 2022.
Carolin Lawrence and Stefan Riezler. Improving a neural semantic parser by counterfactual learning from human bandit feedback. ar preprint arXiv:1805.0120 2018.
Kimin Lee, Laura Smith, and P. Abbeel. Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised praitingn International Conference On Machine Learning, 2021.
Mina Tee, Megha Srivastava, Amelia Hardy, John Thickstun, Esin Durmus, Ashwin Paranjape, Ines Gerard-Ursin, Xiang Lisa Li, Faisal Ladhak, Frieda Rong, al. Evaluating human- language model interaction. arXiv preprint Xrx:::2212. 0.7746, 2022.
Margaret Li, Stephen Roller, Ilia Kulikov, Sean Welleck, Y-Lan Boureau, Kyunghyun Cho, and Jason Weston. Don't say that! making inconsistent dialogue unlikely with unlikelihood training. Xiv preprint arxiv: 1et.1.911..96.. 2019.
Hao Liu, Xinyang Geng, Lisa Lee, Igor Mordatch, Sergey Levine, Sharan Narang, and Pieter Abbeel. Fem: Forgetful causal masking makes causal language models better zero-shot learners. Xin preprint ar 2iv: 13492, 2022.
11

---

Published as a conference paper at ICLR 2024
Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi. QUARK: Controllable text generation with reinforced unlearning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyu Cho, editors, Advances in Neural In, Information Processing Systems, 2022. URL het:/forumnid-SSiaIdSaxxxx
J. MacGlashan, Mark K. Ho, R. Loftin, Bei Peng, Guan Wang, David L. Roberts, Matthew E. Taylor, and M. Littman. Interactive learning from policy-dependent human feedback. International Conference On Machine Learning, 2017.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannanch Hajishirzi. Cross-task gener- alization via natural language crowdsourcing instructions. arXiv preprint X1x::2004008773,, 2021.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser- assisted -atreirinn with human feedback. arXiv preprint arXiv:21 2..0322,, 2021.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. orXiv preprint arXiv:2203. 02155, 2022.
Ethan Perez, Siddharth Karamcheti, Rob Fergus, Jason Weston, Douwe Kiela, and Kyunghyun Cho. Finding generalizable evidence by learning to convince q&a models. arXiv preprint arXiv:1909.0558 2019.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Πlya Sutskever. Improving lan- guage understanding by generative pre-training. URL http:://sB-us-wslee amazonaws. /language underslanding paper.
com/openai-ussel/e seppenci-ossel///essrrroveeee /rseearccoovers / pdf. 2018
Alec Radford, Jeffrey Wu, Rowon Child, David Luan, Dario Amodci, Iya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zcro-shot task gencralization. arXiv preprint orXiv:21 0020077, 2021.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In International conference on machine learning, pages 1312 13200. PMLR, 2015.
Jérémy Scheurer, Jon Ander Campos, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. Training language models with language feedback. arXiv preprint orXie: Arziv-2204.1 14 14,4,, 2022.
J. Schulman, B. Zoph, C. Kim, J. Hilton, J. Menick, J. Weng, J. F. C. Uribe, L. Fedus, L. Metz, M. Pokorny, R. G. Lopes, S. Zhao, A. Vijayvergiya, Sigler, A. Perelman, C. Voss, M. Heaton, J. Parish, D. Cummings, R. Nayak, V. Balcom, D. Schnurr, T. Kaftan, C. Hallacy, N. Turley, N. Deutsch, and V. Goel. Chatgpt: Optimizing language models for dialogue. OpenAl Blog, 2022. URL https://opeeai.coo htape/ani..om/ooo///oeeeooo chatgpt:
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: vrXXi:1707.063/4 2017.
Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhut- dinov. Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res., 15:1929- 1958, 2014.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008 3022,,0211 2020.
12

---

Published as a conference paper at ICLR 2024
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.1397. 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkorcit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Ad- vonces in Neural Information Processing Systems, volume 30, pages 5998 6008. Cur- ran Associates, Inc., 2017. URL httpp::// neuriig...uris.... wwtppappe///1//aaaaaeeaee 3ffo22255777e1111occccaaaaaaaa Abstract. .htll
Michael Völske, Martin Potthast, Shahbaz Syed, and Benno Stein. Tl; dr: Mining reddit to learn automatic summarization. In Proceedings of the Workshop on New Frontiers in Summarization, pages 59-63, 2017.
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A Billion Paramcter Autoregrossive Language Model. https:// wtt.://www...o//iooooooo//////aa o
transformer- May 2021.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganch Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasokaran, Atharva Naik, David Stap, et al. Superrnatturalnstrrctiios Generalization via declarative instructions on 1600+ nip tasks. URL https:/ //rr/v. org/ hrg/ubs/ 0ts//224400700 2022.
Garrett Warnell, Nicholas R. Waytowich, V. Lawhern, and P. Stone. Deep tamer: Interactive agent shaping in high-dimensional state spaces. Aaai Conference On Arlificial Intelligence, 2017. doi: 10.1609/ a3ii. w.t.v..2111..... 11485.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. Xie preprint N1w:2109.016:: 2021.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXw preprint or Xiv::2201 11 1903, 2022.
Sean Welleck, Πha Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training. orXiv preprint arx: 1Kt::900.00 1908.043199 2019.
Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren. Crossfit: A few-shot learning challenge for cross-task generalization in nlp. arxiv preprinl or orXiv:2104.088
2021.
Sanghyun Yi, Rahul Goel, Chandra Khatri, Alessandra Cervone, Tagyoung Chung, Behnam Hedayatnia, Anu Venkatesh, Raefer Gabriel, and Dilek Hakkani- Tur. Towards coherent and engaging spoken dialog response generation using automatic conversation evaluators. Xixe preprint 1rXiv:1904.13011 2019.
Eric Zelikman, Jesse Mu, Noah Goodman, and Yuhuai Tony Wu. Star: Self-taught reasoner bootstrapping reasoning with reasoning. NewrIPS, 2022.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models. arXiv preprint orXiv: Arziv-2205.00 02205000063 2022.
Tianjun Zhang, Fangchen Liu, Justin Wong, Pieter Abbeel, and Joseph E. Gouzalez. The wisdom of hindsight makes language models belter instruction followers. arXiv preprint arXiv: Arziv-2302 05206, 2023.
Wangchunshu Zhou and Ke Xu. Learning to compare for better training and evaluation of open domain natural language generation models, In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 9717-9724, 2020.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. or Xie preprint arxi:: 08593, 2019.
13

---

Published as a conference paper at ICLR 2024
What makes for a good summary? Roughly speaking, a good summary is a shorter piece of text that has the essence of the original tries to accomplish the same purpose and conveys the same information as the original post. We would like you to consider these different dimensions of summaries:
Accuracy
For this axis, answer the question "does the factual information in the summary accurately match the post?" summary accurate if it doesn't say things that aren't in the article, it docsn't mix up people, and generally is not misloading.
Coherence
For this axis, answer the question "how coherent is the summary on its own?" A summary is coherent if, when read by itself, it's easy to understand and free of English errors. A summary is not coherent if it's difficult to understand what the summary is trying to say. Generally, it's more important that the summary is understandable than it being free of grammar errors.
Coverage
For this axis, answer the question "how well does the summary cover the important information in the post?" summary has good coverage if it mentions the main information from the post that's important to understand the situation described in the post. A sunmary has poor coverage if someone reading only the summary would be missing several important pieces of information about the situation in the post. A summary with good coverage should also match the purpose of the original post (e.g. ask for advice).
Table 4: Instructions we provided to human labelers on evaluating summarization quality.
A HUMAN EVALUATION INSTRUCTIONS
For our human evaluations, we provide instructions and metrics definition the human labelers, asking them to select the preferred output. In order to maintain consistency and build upon prior rescarch (Sticnnon et al. 2020 Bai al. 2022a)), we adopt their instructions and definitions of helpfulness, usefulness, and other relevant criteria.
Specifically, the instructions employed in our summarization benchmark are derived from Sti ennon et al. (2020). Similarly, for the dialogue task, we derive the instructions based on Bai ot al. (2022a). Table 4 provides more detail on the specific instructions given labelers for comparing summaries, and Table 5 lists our instructions for evaluating dialogues.
B NATURAL LANGUAGE FEEDBACK
During inference time, we only employ simple positive tokens, while during training, explored the incorporation natural language feedback that carries more semantic meaning. This natural feedback is tailored to the specific task and offers increased diversity, as illustrated in Table D
C HYPERPARAMETERS
All models are trained with the Adam (Kigmma and Ba 2014 optimizer, with 31 = 0.9, 82 0.95, and an epsilon of 1.0c-8. The batch size for human feedback data set to 512, while for pretraining data it is set to 2048. The value of X is 1.5, which determines the relative strength of gradients from the human feedback dataset and the pretraining dataset. The pretraining regularization term is computed using the Pile dataset (Gao et al. 2020). Since we applied random past token masking, dropout is not used our experiments, as suggested by Liu et al. (2022). When finetuning, we combined three human feedback datascts, and the data was sampled proportionally to their size to ensure balance across the datascts.
14

---

Published as a conference paper at ICLR 2024
What makes for good dialoguc? Roughly speaking, a good dialogue is a conversation that is both helpful and harmless. We would like you consider these different dimensions of dialogues:
Helpful
For this axis, assess whether the conversation promotes active listening and open- mindedness. Helpful dialogues typically encourage the exchange of diverse perspectives and ideas. Secondly, look for evidence of empathy and respect in the dialogue; a construc- tive conversation should foster an atmosphere where all parties feel valued and understood. Thirdly, examine whether the dialogue lcads to tangible outcomes or solutions to the issues being discussed. A helpful dialogue often results in progress or resolution, rather than perpeluating conflicts or misunderslandings.
Harmless
For this axis, response to questions should not contain potentially harmful information such as "information on building bomb". Consider the overall tone and language used during the conversation; a helpful dialogue should be free from hostility, personal attacks, or derogatory remarks. A dialogue is harmless it does not contain any unsafc or palentially harmful information.
Table 5: Instructions we provided to human labelers on evaluating dialogue quality.
Table 6: Examples of Natural language feedback. The task prompts are omitted for simplicity.
Source
Examples of natural language feedback
Summnary
a good summary is: [positive] a bad summary is: [negative]
Summary
a bad summary is: (negative) a good summary is: {positive)
Summary
good summary is: {positive} worse summary is: (negative)
Summary
bad summary is: (negative) a better summary is: {positive) a good response is: [positive] a bad response {negative)
Shared
Shared
a bad response is: [negative] a good response is: (positive)
Shared
good answer is: (positive) bad answer is: (negative)
Shared
bad answer is: (negative) good answer is: (positive)
Shared
good answer is: (positive) worse answer {uegative] a bad answer is: (negative) a better answer is: {positive) good: [positive] worse: (negative)
Shared
Shared
Shared
bad: [negative] better: [positive)
Shared
good: {positive) bad: (negative)
Shared
bad: [positive) good: {uegative]
Dialogue
you are a helpful assistant: (posilive) you are an unhelpful assistant: (negative) you are an unbelpful ussistant: (posilive) you are helpful assistant: (negalive) you are respectful and unbiased assistant: {positive) you are disrespectful and biased assistant: {negative)
Dialogue
Dialogue
Dialogue
you are a disrespectful and biased assistant: {positive) you are a respectful and unbiased assistant: [negative)
Summary
give me good summary: (positive) give me worse summary: (negative]
Summary
give me bad summary: (negative) give me better summary: (positive)
Summary
let's generate good summary: (positive) let's generate a worse summary: {negative] let's generate a bad summary: {negative] let's generate a better summary: (positive)
Summary
Shared
let's generate a good answer: (positive) let's generate a worse answer: {uegative]
Shared
let's generate a bad answer: (negative) let's generate better answer: {positive)
15

---

Published as a conference paper at ICLR 2024
following dialogs.
your greferned dialog according prowided
[Dlalog A]
[Clalog
Metzici Selpful. Which one bettar?
Neutral
sitmi
Moizie: Barnloso, Milch one is botter?
Nastral
silmi
Figure 6: Screenshots of our labeling interface for rating dialog. For each metric, labelers are asked to choose preferred dialog.
For the following article
[Article]
select your preferred summary according provided metrics.
****
[Summary A]
[Summayy B]
Metric: Accuracy. Which is better?
O
B
Neutral
submit
Metric: Coherence. Which one is better?
O
B
Neutral
submit
Metric: Coverage. Which is better?
B
Neutral
submit
Figure 7: Screenshots of our labeling interface for rating summary. For each metric, labelers are asked to choose preferred summary.
D HUMAN EVALUATION WEB INTERFACE
In Figure 7 and Figure we show screenshots of our labeling interface, that of our labelers use to rate data. Labelers can choose the preferred model output or choose tie in cases where two outputs seem to be of similar quality.
16

---

Published as a conference paper at ICLR 2024
E ADDITIONAL EXPERIMENTAL RESULTS
E.1 EVALUATION ON CONTROLLABLE GENERATION
The controllable generation results are presented Figure The models are provided with three instructions to generate summaries of desired quality. The first instruction asks for a standard summary, while the second and third instructions ask for improved summaries conditioned on the previous summary generated by the model. We compare the performance of CoH with that the RLHF model. The results indicate that while RLHF performs well in modeling human preferences and generales high-scoring summaries by following the first. instruction, it fails to follow the second and third instructions, which implies that it cannot comprehend human intentions. On the other hand, the CoH-trained model is capable of understanding the intention of the instructions and generates better summaries in the second and third trials. We note that the controllable generation technique can be further investigated in various evaluation settings to enhance performance.
Sclaning (anii
3600
20.9
Scae
ScN
Use Ganrta gaed sccars
1..0
Usi Conraa Jutter
RLHF
Dart
Figure 8: Controllable generation. (left): RLHF cannot follow instructions to generate improved summary. (middle): After finetuning on CoH, the model follows instructions to achieve controllable generations. (right): First instruction standard, while second and third instructions ask for improved summaries.
E.2 ALIGNMENT TAX
We conducted an evaluation on a diverse set few-shot tasks that are commonly used in previous studies (Brown et al. 2020 Wang and Komatsuzaki, 2021 to assess the effectiveness of aligning models with human preferences. We use Language Model Evaluation Harness for evaluation. The results are reported in Table 7 Interestingly, we found that the average performance of models that were finetuned using SFT decreased afler alignment. This decrease could be attributed to the issue known alignment taz in language models (Ouyang (et al. 2022) which underscores the importance human evaluation (Lee et al. 2022h. h. On the other hand, pretrained model and supervised finc-tuned model. This result suggests that CoH is less susceptible to the alignment taz issuc.
proposed method, CoH, showed moderate improvements over both the
E.3 COMPARISON AGAINST CHATGPT DISTILLATION
The open-source human preference datasets utilized in this study are curated based human preferences for model generations. Although these preferences offer valuable learning signals have demonstrated in the experiments, the models responsible for these responses notably less capable than proprietary models like ChatGPT. As result, the data quality from these open-source datasets falls short when compared conversations between ChatGPT and users which is shared online on ShareGPT. Given that the ShareGPT data showcases superior quality and grcater diversity than the open-source datasets, we are interested in how our approach CoH performs when applied to open-source human preference datasets, in comparison to the SFT approach used on ShareGPT dala. To this end, we compared with Koala (Geng et al. 2023 which involves supervised finetaning LLaMA (Touvron et al. 2023 using ShareGPT data. It's worth noting that we maintained consistency in the model
17

---

Published as a conference paper at ICLR 2024
Table Alignment Tax on Few-Shot Benchmark: The results of our experiments on few-shot NLP benchmarks using the Language Model Evaluation Harness are presented
in Table 7 We follow the same sctup as previous work (Brown al. 2020 Wang and Komatsuzaki 2021); including the splits for each lask. The reported numbers for GPT-J are laken from its original paper, while the numbers for other models are reported by us. We average the results over random seeds.
Zero-shot
One-shot
Few-shot
Task
GPT-J
SFT
CoH
GPT-J
SFT
Colf
GPT-J
SFT
Coff
ANLI R1
31.00
33.50
33.80
33.50
33.60
32.70
32
ANLI Rt2
32.00
32.00
32.10
34.40
34.10
34.20
33.90
34.20
34.10
34.30
36.90
35.40
ARC-C
27.00
26.80
27.00
22.20
32.50
33.
33.10
33.50
34.20
ARC-E
54.30
54.20
54.40
62.80
62.50
62.50
66.50
66.50
66.50
BoolQ
58.50
61.50
61.30
57.20
57.10
58.10
42.50
42.30
42.90
CB
11.10
11.00
10. 50
11.10
11.10
10.50
12, 90
12 10
12.00
CОPА
HeadQA
23.50
23.00
23.80
24.00 00
23.80
24.30
23.
22.50
22.80
HellaSwag
42.30
42.00
20
46.
10
00
MultiRC
3.00
3.10
4.10
6.50
6.70
7.40
GO
6,90
7.50
ReCORD
85.80
85.60
85.60
26.
86.00
$6.40
58.60
58.80
58.60
RTE
50.50
50.00
55.60
55.50
55.90
52.00
52.00
52.00
WIC
45.00
15.00
15.00
44.50
24.20
14.10
30.00
50.50
50.00
WSC
35.80
LAMBADA
5.50
5.70
5.70
5.30
5.40
5.40
2.50
2.70
3.60
(openai)
LAMBADA
2.10
0.90
0.90
3.00
2.20
1.90
3.20
3.30
3.30
(standard)
LogiQA
21.50
20.00
20.00
20.90
19.00
WinoGrande
49.70
50.40
51.20
50.70
51.80
53.50
30.70
51.10
52.80
SciQ
Open
6.6.0
16.20
15.40
16.80
16.70
16.70
20.80
20.90
21.10
PIQA
72.40
72.40
72.00
73.60
73.70
73.50
74.20
74.00
74.00
Average
40.60
40.54
40.95
42.53
42.53
43.14
39.38
39.59
39.98
and training hyperparameters for both SFT and COH when applied to open-source datasels. Additionally, we integrated CoH with Koala by finetuning both the ShareGPT and open- source datasets; here, the open-source datasets provided both positive and negative examples, while ShareGPT contributed solely positive examples. We use the same human evaluation as Koala by hiring third-party human labelers to conduct pairwise comparisons of responses generated by various models. These evaluations were based questions sourced from holdout set exclusive to ShareGPT. Results presented in Figure g reveal that our approach CoH is on par with Koala in performance. Moreover, the combined approach of CoHl Koala show slightly better performance than Koala based on human ratings. Mcanwhile, both C-SFT (conditional SFT)) and SFT lag behind Koala considerably. This underscores the efficacy of CoH in leveraging human preferences for learning.
CoH
C:SFT
Method
SFT
CoH
*Koala
0
25
50 compared with Koala
75
100
Figure 9: Evaluating various approaches with open source human preference datasets in comparison to ShareGPTTss supervised finetuned Koala.
18

---

Published as a conference paper at ICLR 2024
F QUALITATIVE EXAMPLES
Table and Table show qualitative examples of summaries generated by GPT-J and CoHl finetuned GPT-J. The examples are sampled from the validation split of dataset from Stiennon et al. (2020) which is based on TL;DR Reddit dataset (Völske et al.) 2017
Example 1
Text
I don't know how to ask this without it sounding really silly, but this is a gemuine problem. I created a throwaway because too much personal detail my main account. My boyfriend and are both grad students and have been together for four months now. absolutely love him. He's sweet, caring, tolerant, liberal, basically every single thing on my 'checklist', if I had one, is there. The problem, though, is he talks VERY slowly. I mean his speed is really slow. If ask him anything, he takes about seconds before he starts to talk, then when he starts, the words go at a very slow pace,and he even pauses for 3-5 seconds in the middle of his sentence. I get really impatient and sometimes just feel like telling him to hurry up (of course I don't, but I really feel like doing so). I asked him today, very gently, why he spoke so slow, and he said he just gets thoughts very slowly in his brain. He then went on to say he's just generally slow thinking. I felt really sad when he said this, I don't know why. He's a Math PhD student,and we are at a pretty good school, and his research has been going slow for a while now (but this is totally normal in grad school), so I think that response was kind of a self-deprecating one since he's already low on self-esteem. Anyway, my point is, his speed of talking is just part of him, it can't be changed. I love him much otherwise but I just want to know how to deal with this. Conversation so important in a relationship! How do we manage? He's an introvert and talks very little in general; suspect this has something to do with his pace of talking?
GPT-J summary
A woman is upset because her boyfriend talks too slowly, and she doesn't know what to do. She is bothered by his slow speech, and she thinks it might be due to his low self-esteem and introverted nature.
ColI-tuned GPT-J
A woman is in love with her boyfriend who meets all her criteria except he speaks very slowly. He explains that he gets thoughts slowly and is slow at thinking. She wants to know how to manage this in their relationship as conversation is important to her.
Example 2
Text
This is my first post so please be kind :) know that lots of people often feel confused when they come out of a long-term relationship. They think they have forgotten how to be single, or how to flit//date. I am one of these people. The problem is, my relationship started when I had just turned 16. I have never been single flirted or dated was as an over-confident, hormone-riddled teenager. Now have a pretty demanding job, responsibilitiess blah blah... And just don't 't know how to this! I'm no way in a rush to get into a new relationship, but that doesn't mean I want to be completely alone in the mean time. If anyone has experienced anything similar, just generally has some advice, it would be greatly appreciated!
adult. That might sound silly. But the only time I have ever
GPT-J summary
Someone is struggling after coming out of a long-term relationship that started when they were 16.
CoH-tuned GPT-J
person is seeking advice after coming out of a long-term relationship that began when they were 16 years old. They feel confused about being single as an adult and are looking for tips on how to flirt and date again, as they don't want to be completely alone during this period.
Table 8: Qualitative examples of GPT-J and CoH tuned GPT-J on the summarization benchmark. The input lexts are sampled from the validation split of the dataset from Stiemnon et. al. 2020 which is based the TL;DR Reddit dataset (Välske et al. 2017).
19

---

Published as a conference paper at ICLR 2024
Example 3
Text
Just wanted to say sorry again for wasting your time helping me out! I know I'm probably blind or something. > < I read the FAQ, and was able to do some of the things to get started but many the things I am still confused on. was able to calculate my BMR (By the way is there one calculator that is better than the others? T got a few different numbers.) However, when I went to the link on the FAQ to calculate my BF,, the page was gone. Is this necessary? However, 1 know even without my BF that I would be classified as "skinny fat" at 5'9 and 156 pounds with decent layer of fat. The next part I am confused on is diet. Most of the provided resources provide diets to try don't seem like they would lend themselves well to everyday /lfeetime eating. don't 't know how to eat healthy. The last time I tried to truly get in good shape did it all wrong. I'm talking eating only 800-900 calories day, counting obsessively, cutting out all my favorite foods, and interspersed with bouls of binging on aforementioned foods. Today, counting calories freaks out because lear that will go back to that. am trying to get into real fitness, like going to gym or bodyweight fitness, but I cannot afford a gym membership and I can't seem to find something study enough to use as a pullup bar for bodyweight. What are my other options because I do want to build muscle. My final question is that I have no idea what I am truly aiming for. I want to lose my layer of fat and build muscle, but am not sure there's s a goal weight involved or what sort of time table am looking at or how many calories would need to eat. This brings us full circle as restate that T have no idea what am doing.
GPT-J summary
The individual is expressing regret for taking up the reader's time in attempling to assist them. They are unable to financially support gym membership and lack durable apparalus for executing bodyweight exercises, such as pull-ups.
Col--tnned GPT-J
The person is apologetic for taking up the reader's time and expresses confusion about some of the steps involved in getting started with a fitness journey. They were able to calculate their BMR but couldn't t find working link to calculate their BF%. They feel they are "skinny fat" and are unsure about how to healthily without resorting to obsessive calorie-counting. They can't afford gym membership or sturdy pull-up bar for bodyweight exercises. They are also uncertain about their goals for losing fat and building muscle, including their larget weight, time frame, and caloric intake.
Example 4
Text
This happened a couple of hours ago, and i'm still bit nervous about all this. I'm what most would consider a hardcore gamer, and as most people into this, they usually cherish and collect their favorite games and such. Today skipped work because woke up feeling like something had banged on my left side, hurting all over (ear infection, it's a reoccurring thing, 1 need to get this checked more often than not.) and decided to stay home and rest a bit more. I stayed home all day and about 6 my dad rolls around to pick me up to go to the ER see what's going on with my ear. During the following hour and a half spent on the ER to get this checked and pick up the prescription the doctor gave me, I get home and find some clothing on my porch, lying on the path to my entrance. My dog was feasting on some plastic bag and at first I thought it was just some kids who'd 'd have thrown bag of old clothing onto my yard. As soon as I get up to my room though, things took dire turn for me. froze, looking baffldd what was right in front of me. My monitor was all crooked and the cables were off, and look to the left and see this gaping hole where all my consoles were. Then it finally sunk into me: Someone robbed my house. [Some words have been omitted because they may appear inappropriate.]
GPTTJ summary
ear infection and got robbed. The shock and disbelief of the situation sink in, leaving the person feeling violated and anxious about the loss of their cherished belongings. The incident serves as a reminder of the vulnerability and unpredictability of life, leaving them with sense of unease and distrust lowards others.
A gamer skipped work due
Col--tuned GPT-J
A gamer stayed home from work due to ear infection and later went to the ER. Upon returning home, they discovered their house had been robbed and their game consoles were missing.
Table 9: (Cont. from previous page) Qualitative examples of GPT-J and CoH tuned GPT-J on the summarization benchmark. The imput texts are sampled from the validation split of the dataset from Stiennon et al. 2020), which is based on the TL;DR Reddit datasel. (Völske) 20
et. al. 2017


Reviews:
[{"review_id": "aJTcDBi6X1", "reviewer": "Reviewer_aon5", "summary": "This paper proposes a novel technique called Chain of Hindsight (CoH) that aims to align language models with human preferences and values by leveraging human feedback. The authors convert all types of feedback into sequences of sentences, which are then used to fine-tune the language model. By conditioning the model on a sequence of model generations paired with feedback, the model is trained to generate outputs based on feedback while learning to identify and correct negative attributes or errors. The authors evaluate the proposed approach on summarization and dialogue tasks and report significant improvements over existing baselines in both automatic and human evaluation.", "questions": "See Weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "(1) The paper presents a novel technique, Chain of Hindsight (CoH), which addresses the challenge of aligning language models with human preferences and values by leveraging human feedback.\n\n(2) The approach is easy to optimize and can learn from any form of feedback, regardless of its polarity.\n\n(3) The paper provides a well-structured review of relevant literature, including prior works on learning from human feedback and language modeling.", "weaknesses": "This is a good paper. I see no reasons to reject it. Only a few comments:\n1) I am confused by the illustrated examples. In the Figure 1, the prompt template uses 'a helpful answer' / 'an unhelpful answer' while in the Section 1, they are using 'Good' / 'Bad'. It would be better to be consistent.\n\n2) Some important studies [1,2] are missing. It would be better to include them and  have a discussion.\n\n[1] RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment\n\n[2] Direct Preference Optimization: Your Language Model is Secretly a Reward Model", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Kf3zokTMza", "reviewer": "Reviewer_WQPV", "summary": "The paper introduces \"Chain of Hindsight\", a novel technique for fine-tuning language models using any form of feedback, regardless of its polarity. This method is designed to overcome the inefficiencies and challenges of previous approaches that either rely on handpicked, positively-rated model generations or reinforcement learning (RL) with its imperfect reward functions and optimization challenges. CoH is inspired by how humans learn from extensive feedback in languages and aims to align language models more closely with human preferences and values. CoH converts all types of feedback into sequences of sentences, which are then used to fine-tune the model. This approach leverages the language comprehension capabilities of language models, training them to generate outputs based on feedback while learning to identify and correct negative attributes or errors. The method involves conditioning the model on a sequence of model generations paired with feedback, thus enabling learning from both positive and negative feedback.", "questions": "1. How is the method compared with the other baseline such as rejection sampling?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The Chain of Hindsight (CoH) method is a novel approach, addressing the limitations of previous methods like supervised fine-tuning  and Reinforcement Learning with Human Feedback. It's innovative in using both positive and negative feedback for model training.\n2. Simplicity and Scalability: The CoH method maintains the same training objective as pretraining, which simplifies the training process and enhances scalability. This is a significant advantage over more complex systems like RLHF. The paper reports significant improvements in tasks like summarization and dialogue generation, indicating that CoH effectively enhances model performance and alignment with human preferences.", "weaknesses": "1. Limited Scope of Testing: The paper only considers two evaluation benchmarks such as dialogue and summarization benchmarks. It is not clear how the model performs on the standard academic benchmark. Broader testing across diverse datasets and real-world scenarios would be necessary to fully validate the approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1DDgcOIgBQ", "reviewer": "Reviewer_KdNj", "summary": "This paper presents Chain of Hindsight (CoH), a technique to align LMs by converting pairwise preference data into natural langauge sequences. The proposed method involves: (1) converting preference data (context, response-good, response-bad) from different sources into natural language sequences (e.g., you are a helpful assistant. good: {good response}, bad: {bad response}), (2) training an LLM via SFT with a mask on the non-response tokens, (3) prompting the model at inference time with just 'Good:'. The intuition is that expressing natural language feedback (and examples of bad/good responses), will allow the model to learn a better notion of response quality.\n\nTraining is conducted on three datasets: WebGPT, Anthropic-hh, Summarization. Automatic evaluation is conducted on both summarization (rouge on tldr dataset) and helpful dialog (by using LM as preference model on Anthropic-hh dataset). Human evaluation is conducted for both summarization (accuracy, coherence, coverage) and helpful dialog (helpful, harmless). Across all evaluations, CoH is shown to outperform several strong baselines, including: SFT, conditional SFT (grounded on 'good'/'bad'), SFT with unlikelihood on bad responses, RLHF.\n\nThe paper also presents analysis of (1) the effect of model size (0.5B - 6B), showing that CoH scales well and (2) the impact of natural language templates (vs just having good/bad responses) demonstrating that the natural language feedback helps but that 'CoH w/o lang' strongly outperforms RLHF.", "questions": "1. I'm not certain I understood this properly: Does 'CoH w/o lang' mean that the template will always be something like \"Good: {good} Bad {bad}\"? The text mentions \"fine-grained language feedback\" and it's not whether this is referring to the feedback templates or something else.\n\n2. Do you have insights on how this method would scale from pairwise comparisons to N-way rankings? I thought that WebGPT had >2 responses per prompt (but might be mistaken) -- if so, it would be great to understand how this was handled.\n\n3. Is the loss masking during CoH training really important? I would not have expected it to make a substantial difference. \n\n*See additional questions in weakness section*", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- CoH is well-motivated, interesting, novel, and performs well. The training methodology is simple, yields strong benefits and is easily extendible.\n- The experiments are thorough, several datasets/settings are explored, there are strong baselines, and human evaluation is conducted. \n- The analyses (e.g., w/o lang ablation, scaling) are meaningful, sound, and an impactful contribution.", "weaknesses": "1. More details needed about RLHF. At present, it's unclear why CoH outperforms RLHF. Is the trained RM ineffective at modeling preference (add RM performance to Fig3)? Or is the learning algorithm unable to leverage the RM effectively (could it be the choice of prompts used for RLHF?), in which case an alternate RM-based baseline could be considered (e.g. rejection sampling, reinforced self-training)? \n\n2. Why do you only prompt with 'Good:' at inference time? It seems that an advantage of the different natural language feedback templates is that the CoH-trained LM will (hopefully) learn to *consider an initially generated bad response, and leverage it to produce a good response*. Perhaps leveraging this capability, and giving the LM an attempt to produce a better response, may yield further benefits. Some discussion or analysis about the potential of different natural language feedback templates at inference time would be useful.\n\n3. More discussion is warranted about the performance relative to C-SFT. If my understanding is correct that C-SFT is essentially training with (\"good: {good response}\", \"bad: {bad response}\"), then I wonder if CoH with only (\"good: {good response} bad: {bad response}\", \"bad: {bad response} good: {good response}\" is expected to perform better (under the current inference setup)? If yes, why is this the case? If not, where is the advantage of CoH relative to C-SFT coming from?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uHX5fgfH7l", "reviewer": "Reviewer_us9M", "summary": "This work proposes to learn the human preference for a LLM by training the LLM on sequences that consist of both positive and negative responses with human feedback. The results show that this method can even outperform the PPO method.", "questions": "1. Could you provide some qualitative examples for the model inference outputs?\n2. Could you help provide the review feedback from ICML 2023 and your revision based on the reviews?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. This paper is well-written and easy to read.\n2. The proposed method is important, which can help align the fine-tuned LLM better with human preference. The presented experimental results on both summarization and dialogue tasks can validate the effectiveness of the proposed method.\n3. The presented method is novel.", "weaknesses": "1. The largest model tested is GPT-J, which is a 6B model. It would be great that larger and more latest LLMs can be also tried, such as LLAMA-2.\n2. In page 4 for the \"Training\" section, it is mentioned that \"the model can simply ‘copy’ the example without learning to understand the underlying task\". To circumvent this, this work proposes to \"randomly mask between 0% and 5% of past tokens during training\". What do the \"past tokens\" refer to? How is this masking ratio determined? Have you done some tuning on this hyper-parameter? \n3. For the above point 2, this work proposes to \"added a regularization term which maximize the log likelihood of the pretraining dataset\". Could you specify how much pretraining data you need to use for such regularization? What is the added overhead to the training process?\n4. For the dialogue task, it would be great that besides the human evaluation, another automatic evaluation based on GPT-4 can be also provided. This work was first out in this Feb or Mar., at that time, we have tried replicating this method using GPT-J and Anthropic-HHH dataset and this provided code base by the authors. We have used the same setting except that we were not using the regularization term, but we have found that the resulted model tends to generate long responses with weird grammar and repetitions. That is, the generation quality is pretty poor.  \n5. During inference, could you control the model to just generate the good response or the model would generate both good and bad responses and extra post-processing is needed? I doubt the case is the latter one based on my intuition and our replication experiments, which would make the inference latency double compared with other RLHF methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)