Learning from Natural Language Feedback
Authors: Angelica Chen
Jérémy Scheurer
Jon Ander Campos
Tomasz Korbak
Jun Shern Chan
Samuel R. Bowman
Kyunghyun Cho
Ethan Perez
Venue: Accepted by TMLR ()

Abstract: The potential for pre-trained large language models (LLMs) to use natural language feedback at inference time has been an exciting recent development. We build upon this observation by formalizing an algorithm for learning from natural language feedback at training time instead, which we call Imitation learning from Language Feedback (ILF). ILF requires only a small amount of human-written feedback during training and does not require the same feedback at test time, making it both user-friendly and sample-efficient. We further show that ILF can be seen as a form of minimizing the KL divergence to the target distribution and demonstrate proof-of-concepts on text summarization and program synthesis tasks. For code generation, ILF improves a Codegen-Mono 6.1B model's pass@1 rate by 38% relative (and 10% absolute) on the Mostly Basic Python Problems (MBPP) benchmark, outperforming both fine-tuning on MBPP and fine-tuning on repaired programs written by humans. For summarization, we show that ILF can be combined with learning from human preferences to improve a GPT-3 model's summarization performance to be comparable to human quality, outperforming fine-tuning on human-written summaries.  Overall, our results suggest that learning from human-written natural language feedback is both more effective and sample-efficient than training exclusively on demonstrations for improving an LLM's performance on a variety of tasks.

Tomasz Korbak Published in Transactions on Machine Learning Research (02/2024)
Learning from Natural Language Feedback
Angelica Chen'
angelica chen angelica.chen
New York University
Jérémy Scheurer*
jerema@oupolloreecocch.att
Apollo Research"
Jon Ander Campos
jonander@cohere.com
New York University; HiTZ Center, University of the Basque Country UPV/EHU
tomekk@anthropic.co
New York University; FAR AT; University of Sussea
Jun Shern Chan
womshern@myy.com
New York University; FAR Al
Samuel R. Bowman
sb6065 0.60650mryyeddd edu
New York University; Anthropic PBC
Kyunghyun Cho
kyunghyun.cho cho Dryy www...ocooooocccccooooommm..
edu
New York University; Genentech; CIFAR LMB
Ethan Perez
perezOnyu.ede edu
New York University; FAR AI; Authropic PBC
Reviewed OpenReview: https: openreview. net/ per/formm? id= @d=mo3hI5Mim
Abstract
The potential for pre-trained large language models (LLMs) to use natural language feedback inference time has been exciting recent development. We build upon this observation by formalizing algorithm for learning from natural language feedback at training time instead, which we call Imitation learning from Language Feedback (ILF). ILF requires only a small amount of human-written feedback during training and does not require the same feedback at test time, making it both friendliy and sample-effiiint. We further show that ILF can be seen as a form of minimizing the KT. divergence to the target distribution and demonstrate proof-of-concepts on text summarization and program synthesis tasks. For code generation, II.F improves CODEGEN-MONO 6. 6.1B model's pass@1 rate from 22% to 36% on the MBPP benchmark, outperforming both fine-tuning written repaired programs. For summarization, we show that ILF can be combined with learning from human preferences to improve a GPT-3 model's 's sunmarization performance to be comparable to human quality, outperfrringg fine-tuning on human-written summaries. Overall, our results suggest that ILF is both more effective and sample-efficient than training exclusively on demonstrations for improving an LLM's performance on a variety of tasks.
MBPP and on human-
"qqual contribution.
Work done while NYU.

---

Published in Transactions on Machine Learning Research (02/2024)
//1
Change about 3/1
1) T generates multiple outputs for a given task.
2] Natural language feedback is given about model outputs.
Y/2
Change about 3/2
y/
Change about !/3
4) tigh-quulity refinements are used to fine-tune T
(1) (2)
3) Teefiee generates multiple refinements per output that incorporate the feedback into the original model outputs.
2 .3 ....
(1) (2)
y/2 1/2
(1) (2)
1/3 33 ....
A high-level overview of how Imitation learning from Language Feedback (ILF) improves a base LLM via learning from natural language feedback. After тe generates multiple outputs y1, 32, for a given task, human annotators provide natural language feodback about how the outputs can be improved. Another model, Refine intakes both the original outputs and the natural language feedback, and generates refinements of yı, y2. ys that incorporate the feedback. The higher quality refinements are then used fine-tune σ--
Figure 1:
1 Introduction
Despite achieving strong performance across variety of NLP tasks (e.g. summarization, question answering,
and dialog [Radford & Narasimhan 2018 Radford et al. 2019 Brown et al. 2020 Rac et al. 2021 inter
alia)), large language models (LLMs) often generale text that violales human prelerences.
Examples of
these violations include misinformation (Lin et al. 2021 offensive language /Gehman et al.
2020 factually
incorrect summaries (Stiennon et al. 2020), and buggy or insecure code (Chen et al. 2021 2022).
Kang et al.
We hypothesize that these failures be attributed in large part to modern LLM pre-training set-ups. For instance, modern pre-training datasets often consist mostly of minimally filtered text scraped from the Internet, which often contains insecure (Kang et al., 2022) or buggy (Chen et al.) 2021) code and toxic text (Gao et al. 2020). This training signal also consists exclusively of offline demonstrations, without any signal from trial-and-orror or interactive guidance that penalizes the model's buggy outputs. As such, we hypothesize that supervising LLMs with explicit human-written feedback on the model's own outputs can be more effective at training models to produce functionally correct code.
In particular, intuitive and rich form of feedback to provide to LLMs is natural language feedback. We argue that. LLMs are naturally able to incorporate written feedback, which has been shown to significantly improve a code generation model's pass rates when the feedback is provided at test time (Nijkamp et al, 2022 Austin et al.) 2021). In our work, we build upon this observation by exploring the use of natural language feedback during the training process itself, rather than just during inference. We conjecture that such feedback provides expressive and targeted information about code generation model's current failings in sample-efficient manner. More broadly, this approach also represents a weak version of scalable over- sight (Amodei al. 2016 Bowman et al.) 2022), in that model overseers improve model merely by evaluating its outputs, without manually generating new demonstrations, in way that takes advantage of the capabilitics that are being supervised. To train LLMs with language feedback, we propose an algorithm called Imitation learning from Language Feedback (ILF), which extends our previous non-archival work in Scheurer et al. (2022), who study the impact of learning from language feedback on text summarization models. Schourer et al. (2022) improves summarization model by training the base model on improved summaries generated from the model's original summaries and human-written feedback. Our work builds
upon Scheurer al. (2022) in number of ways: (1) by formalizing the algorithm and generalizing it into
2

---

Published in Transactions on Machine Learning Research (02/2024)
form that can be applied to any task (our ILF algorithm in Section 2 (2) by detailing how the reward
function can be adapted for both text summarization (Section 4.1 and code generation (Section 3.1 ; (3) by demonstrating proof-of-concept of ILF for code generation; and (4) by scaling up our work on ILF for summarization via crowdsourcod human annotations and conducting extensive experiments demonstrating the effectiveness of our method against other baselines.
ILF improves the quality of outputs generated by a baseline model Ty by training a separate model Refine to use language feedback to revise the low-quality outputs. (We refer to the revised outputs refinements.) We then improve πe by fine-tuning it on the TRefine -geeerttde refinements that of sufficiently high quality, yielding final improved model Tg. The high-level algorithm is shown in Figure 1 This procedure may be run iteratively to continue improving the model, which we show can be seen as minimizing the expected KL divergence from a target ground truth distribution (Section 2
We demonstrate the efficacy of ILF on two tasks proof-of-coneept for code generation, and a larger scaled- up study of ILF for text summarization. For code generation, we show that ILF improves a CODEGEN-MONO BB model's pass@1 1 rate the Mostly Basic Python Problems (MBPP) benchmark (Odena et al. 2021 from 22% (after being fine-tuned on the ground truth programs provided by the MBPP dataset) to 36%. It also significantly outperforms zero-shot performance (26% - 36%, see Section (.3). For text summarization, we show that ILF improves LM-generated summaries monotonically with the amount of feedback provided (with experiments of up to 5K samples of feedback). In all data regimes, ILF leads to comparable or better results than fine-tuning on humannurrilen summaries, suggesting that our approach is a strong alternative to supervised learning on human demonstrations, We also show that combining ILF with best-of-N sampling, where the samples are ranked by model trained from binary comparison feedback, results in even higher quality summaries that are comparable to the quality of human-written summaries. Taken together, these results establish that learning from language feedback is a promising avenue for training LLMs.
2
High-Level Method
Here, we formally describe the problem we aim to tackle before introducing our algorithm.
2.1 Preliminaries
Suppose start with vocabulary V and pre-trained autoregressive language model To parameterized by 0. To V" + [0, 1] is a probability distribution over sequences of tokens æ E V*, where V is the set of all finite concatenations of tokens in V. We also have distribution of tasks t ~ and reward function R(x,() () which outpuls some real-valued reward that encodes how high-quality a particular output æ is for task t. Lastly, we define a fine-tuning function FINETUNE(Tę, D) that applies a gradient-basse optimization algorithm to Ty using the associated loss objective calculated over dataset D.
2.2 Imitation Learning from Language Feedback
Our high-level goal is to sample diverse outputs 21 no(-lt) for any given task t sampled from the task distribution. We do so by fitting wy to approximate ground truth distribution (f)) that assigns a probability to Iι that is proportional to its quality, as measured by the reward function R. Fitting we to approximate गद can be seen as minimizing the expected KL divergence from कद to ny over the task dist ribbutoni pr:
min E [K[(T] [t)) t-p
(1)
where
7j (21) & exp (3R(z,,
(2)
Minimizing the objective in Equation 1 is equivalent to supervised Icarning, f.e. minimizing the cross-cntropy
loss:
[(0)
E
[Co(t)]
(3)
tnpT
3

---

Published in Transactions on Machine Learning Research (02/2024)
where
Co(1) = ( log (x)))))
(4)
Rather than computing this loss over the exponentially large space of all possible æ1 ss's, we instead use Monte- Carlo sampling over a small set of x 's drawn from के. However, this is still intractable because we cannot sample directly from mē. Instead, we approximate oy using importance sampling with a proposal distribution qe (21):
Co(l) (x( (((i log no (0(1)11)) (e(m))
which assigns higher weights to higher quality programs 21.
2.3 Proposal Distribution 9
Intuitively, we aim to design qe be close as possible to oद, which we accomplish by incorporating pieces of natural language feedback f that give information about how to transform low-roward program Ty into higher-reward program Zt. This can be achieved by: (i) identifying low-quality output Io ma(-14 that currently has low reward R(xo, t), (ii) asking for natural language feedback f about how to improve 2o, (iii) using f to transform the original output To into a refinement that incorporates the feedback and has reward R(x1,t) t) > R(zo, t), and (iv) assigning higher weight to ø1-
We can formalize this procedure as follows. Let (xaltt,, Jo, f) be a distribution over outputs xι that improve zo by incorporating the feedback f and p=(f (o) be the distribution of pieces of feedback for output xo and task t. We can then define our proposal distribution as
2e (x1) = Σ To(xo))) x pp p)((,, x (w to f), f);
(6)
which results from marginalizing q (x1) over all pairs of (f and factorizing the joint distribution of æ1 To and f given
In some tasks (i.e. summarization), we assume that every initial output xn ~ (-It) is low-quality and can be improved upon (Section 4. However, for other tasks (i.e. code generation), we may be able to identify initial outputs that are already high-quality and do not need feedback. For example, if the code is already functionally correct. In this case, we specify an additional quality function d(x, t) such that
(lx = {() 1, o, otherwise
if R(z,t)
(7)
for some reward threshold 7. We can then modify Equation where @0 t) = and t(x1,t) =
to place all support on pairs of 01)
qe
Filtered
(x))
Σ (o( (t) do (6(20. ()) (=(((t, (x11t, 2o: f) x 6, (0(21,))) (t,,1)
(8)
mo.5
where we write 8;()) for the Kronecker delta distribution, where
8,(5) = { 1, ifi=j
0, ifi
To sample from of Filtered and gt, we defer discussion to Sections and respectively.
Now that we have devised way to sample from qr, can rewrite Equation 5
Co(t)
E x m q
log no (zilt),
(9)
10c

---

Published in Transactions on Machine Learning Research (02/2024)
where w, is the importance weight of the i-th sample from qr. However, since only sample from q we cannot directly compute wi. Instead, we assume that ge(=f) is relatively constant for all samples of due to the high quality of human feedback. This allows us to simply compute the un-normalized value of x (x7) (from Equation 2 and use self-normalizatio instead. we sample using low temperatures (resulting 3 - 00); then this is approximately equivalent to computing the likelihood of only the best refinement øi for each task i, i.e.
[())
~
I flog No (zi10)
(10)
tepT
Notably, this objective is equivalent to supervised learning over the best refinements per task. Using natural language feedback simply allows us easier access to these refinements.
Our objective of approximating the ground truth distribution के. which is proportional to the reward R, has clear connections maximizing reward in reinforcement learning (RL). However, the goal in RL to find the best policy that maximizes the reward, whereas our algorithm results in a training dataset of high-quality outputs 21 for task t, which allows us train our model to generate diverse set of outputs that achieve high rewards on average. The broad diversity of high-quality outputs endows ream users and systems with more control over which aspects they prefer and want to avoid.
3 Code Generation
Program synthesis, the automatic generation of computer programs from an input specification (e.g- a natural language task description or a set of input-output, examples) (Manna & Waldinger, 1971] is an important task for the field of software engineering. Effective program synthesis can not only improve the efficiency of software developers (Ziegler et al. 2022). but also increase the accessibility of writing code
in general. Recently, pre-trained large language models (LLMss) have demonstrated impressive success
program synthesis (Chen et al. 2021 Li et al. 2022a Austin et al. 2021 Nijkamp et al. 2022 Xu et al,
2022a| inter alia) but still struggle to consistently generate correct code, even with large-scale pre-training (Chen et al.) 2021). We select code generation as testbed for our approach not only due to its importance but also due to the ready availability of a precise reward function (i.e. functional correctness of the code) and evidence that current state-of-the-art code generation models still perform poorly on difficult programming challenges (OpenAl 2023).
3.1 ILF for Code Generation
In code generation, each task consists of pairs (f, T of a natural language task description T (e.g. "Write function that computes the prime factorization of an input integer.) and suite u UNITTESTS()) U of unit tests associated with task t. Our reward function R is unit test verification function EVAL: V x T 1) that indicates whether program no(- It) passes all the unit tests in UNITTESTS(I):
{
1, passes test suite UNITTESTS((). 0, otherwise
EV)L(E,1)
(11)
We use EvAL(æ, t) as our reward function R(z, t) and the identity function as our quality function o(x, t). It follows that Equation S can be adapted for code generation as follows:
Filtered (x1) =
Σ (zol))))) 8, (EvaL(zo, t) z0. t)) x p=(flt,zo) (s(2i]t, morf x&i (E)
(12)
This proposal distribution is guaranteed to place higher probability mass on higher-quality programs (in terms of unit test pass rate) than my since the term ó1 (EVaL((zI, (1 equals 0 for incorrect programs ø1.
We approximate sampling from de
Filtered by considering each of the terms in Equation 12 in order:
5

---

Published in Transactions on Machine Learning Research (02/2024)
Incorrect program generated 7g
1. We first sample from (o( x 60 (EVAL(zo, t) (o. by rejection sam- pling from Te. In other words, we sample programs zp from xp for task and only keep those that fail the test suite (tee EVAL(2a, t) O; step 2 of Algorithm I
Failed unit hests
def sun every other (1at): "*"Rat
8
every other elemant of lst, starting with index 0, ".* return (um(lat)
Fine-tune mg on refrnements
Natural language feedback provided by human
We approximate sampling from pr(f]t, zo, EvAL(zo, t) 0) by having humans annotate programs 2o (paired with their corresponding task descriptions and test suites u) with natural language feedback (step 3 of Algorithm D.
Thee code is wrong because it every instead of every other one. Fix it by slicing Let using increment before taking the sum.'
attempts
Refinement generated by ReRiin using the feedback and original program
3. We approximate sampling from To ( (an by sampling from Teefine a model capable of generating refine- ments given the task description, original programs, and humann-writeen feedback.
Passed unt
tasts
aum every other (lat): """Return the sum of every other 1st, starting with index n.. raturn aum (2st[::21)
TeRfin
4. Finally, the term & (EVAL(411,) (t, cor- responds to another filter: we only keep re- fined programs I1 that pass the test suite.
Figure 2: guage feedback (ILF) for code generation. Given initial LLM тe, we sample programs from that do not pass unit tests (indicated by the red X). Human annotators write natural language feedback for the in- correct program and a model Reffine generates a re- finement i.e. an improved version of the original program that incorporates the feedback and passes the unit tests. Finally, we fine-tune on the refinements.
An overview of imitation learning from lan-
The complete algorithm is summarized in Figure and Algorithm Next, consider more concrete details of how this sampling is accomplished.
Using Refine Incorporate Feedback ILF assumes the availability of feedback but not neces- for variely
sarily of the repaired of reasons. We assume that program synthesis may be a task for which writing high-level natural lan- guage feedback is often less laborious than performing program repair. Although writing feedback involves identifying at a high level what is wrong with the program and how should be fixed, program repair may involve the additional steps of refactoring, looking through documentation, and testing. Moreover, past work (Austin al. 2021 Nijkamp al. 2022) has indicated that certain large LLMs can proficiently incorporate the feedback inference time, assuming access to accurate and high-quality feedback. As such, ILF as- sumes access to some model TRefine that is capable of producing refinement given the original program and feodback. In many cases, Tetfiie can perform this task via few-shot prompting, without additional training (Nijkamp et al. 2022). We explore such possibilities in Appendix A.4 As such, ILF for code generation does not necessarily require extra training data for TRefine.
Algorithm A single round of imitation learning from natural language feedback for code generation.
1: Input: Dataset D, initial LLM тe, unit test verification function EVAL, LLM TRelnee V - [0. 1] trained to incorporate feedback into code
2: C+ {(x0, t, u) 20 ~ To (-(t), EVAL(TO,)) 0, (t, E
> Humans write feedback f for xp E C. DRefined +- ((1,21) 21) Re (1 [) 1, (2o. f,t) Cannotated)
Cannotated <- {zo. (.to.f))) (Fo, t, E
TRefin generates
refinements 21 that incorporate feedback f into Zo-
+- FINETUNE(TT, DRefined)

---

Published in Transactions on Machine Learning Research (02/2024)
Refune take a variety of forms, but due to constraints available funding and compute resources, we fine-tune a pre-trained CODEGEN-MONO 6. 1B model as our Thefin For the fine-tuned Reefnne we create a training dataset by further annotating a subset of Cannotated with refinements that repair incorrect programs Zo by incorporating foodback f, such that EVAL( t) = for (Io, f.t) E Canuotated- Further details of our dataset and annotation procedure are in Section 3.2
3.2 Experiments and Results
Having described our high-lovel approach, we now explain the experimental sctup we use to test ILF for code generation. Our data and code are open-sourced al ILF Lu--for-code--eneratiioo
Dataset We train and evaluate our models on the Mostly Basic Python Problems (MBPP) dataset (Odena et al., 2021). MBPP contains 974 Python programming tasks designed to be solvable by entry-level coders. Each task contains a natural language lask description (e.g., "Write a function to return the prime factor- ization of the imput."), ground truth solution, and a suite of three unit tests. Since the task descriptions are sometimes ambiguous, we include one unit test in the task description. The addition of the unit test helps to specify the input and output format of each task. We hold out the remaining unit tests for the evaluation of our generated programs.
MBPP includes a designated prompt/trane into the following splits:
split of the dataset, but we re-split the dataset
MBPP Refine: These tasks with IDs in the range 111-310 for which CODEGEN-MONO 6.1B did not generate any correct completions. For the experiments where Refine is fine-tuned model, this split is used to train Refinee
MBPPTrain:
These are tasks with IDs in the range 311-974 for which CODEGEN- MoNo 6. 1. did not generate any correct completions. This split is first used to evaluate the correctness of refinements generated by TRefnee Then, the correct refinements in this split are used to train xe to obtain TT* (step 5 in Algorithm
MBPPTea:: These are tasks with IDs in the range 11-110 that we use to evaluate the final performance of TT- Unlike the previous two splits, use all tasks in this split, rather than only the tasks for which CODEGEN-MONT 6.1B did not originally generate correct programs for. This allows to better compare the bascline performance of To with that of xp-.
We use this modified split so that a larger portion of the dataset can be used to train the final model Tp", whereas smaller portions are allocated for training Teefnee and evaluating πæ-. We do not make use of the prompt split (IDs 1-10).
Models Throughout this paper, we use a pre-trained CODEGEN- 6.1B model Nijkkamp et al. 2022
as our Tg. It is pre-trained sequentially on THEPILE (Gao et al. 2020), BIGQUERY Nijkamp et al. 2022 and BIGPYTHON (Nijkamp et al., 2022). We selected this model because it open-source, can be fine-ture tune on a single x lt al.
100 A100 (80 GB) node, and demonstrated passülk scores comparable to CODEX-12B (Chen
2021 Nijkamp et al. 2022).
To implement our algorithm, independently fine-tune two separate instances of CODEGEN- 6. to create Titfiine and the final model πT-. We train Teefie using pairs of incorrect programs and human-written feedback as inputs, with human-written refinements as targets (using the format in Figure 7 In contrast, we train rT- using natural language task descriptions from MBPP as the inputs and Feefin--eneeatee refinements as the targets. Further training details are in Appendix A.2
We also investigate using few-shot gpt-3. and gpt-4 as our Toofnn in Appendix A.4 demonstrate that ILF does not necessarily require training custom RRafine model. Oftentimes, prompting off-the-shell LLM suffices.
7

---

Published in Transactions on Machine Learning Research (02/2024)
We evaluate all code generations in this paper using the pass metric introduced in Kulal et al. 2019). It estimates the rate for which 21 of k model samples passes all the unit tests. We use the empirical estimate of this quantity from Chen et al. (2021), an unbiased estimator given by:
Evaluation
pass@k = lEtakk
(13)
for n total programs (where n 2 k) and correct programs for the given task.
Human Annotation We hire annotators via Surge A to write both natural language feedback and refinements for incorrect programs generated by CODEGEN-MONO 6.1B. For each task that CODEGEN- MoNO 6.1B generated no correct programs for, we ask the workers to first select one of the incorrect programs to write feedback and refinement for. We specify that the workers should select a sample that seems relatively easy to correct (Le. could be minimally corrected to pass the unit tests). Then, they are asked to write feedback that describes what is wrong with the current code and how to fix it. For the refinement, they are asked to copy over the original code and make the minimam number of edits necessary to incorporate the feedback and pass all the unit tests. The full set of worker instructions can be found in Appendix A.3
We keep all annotations for which the refinement passes all tests in the task's test suite, the leedback correct (as manually verified by the authors), and the Levenshtein edit distance between the refinement and the original program is less than 50% of max(len(refinement) len(original program)). The final dataset consists of 195 triples of (incorrect program, human-wrilien feedback, hunan-wwillen refinement). On average, workers are paid $23 per annotated sample and take 27 minutes/sample, with a 10th percentile of 4 minutes and a 90th percentile of 43 minutes.
Although the ILF algorithm only requires the collection of human-writien feedback for the tasks MBPPrrain (assuming access some Teefme that is already fine-tuned or can generate refinements via few-shot prompting), we collect both human-written feedback and refinement for all splits of the data that can conduct further analyses of our method. For instance, this allows us to compare fine-tuning geeerae-geee refinements with fine-tuning on human-written refinements. When scaled to other pairs of model and task, ILF requires new feedback annotations, but it is possible that using ILF on one dataset will improve the model's abilities on another aset for a similar lask. We leave analyses of further scaling ILF across different tasks and models to future work.
Table 1: Initial zero-shot CODEGEN-MONO 6.1B performance on the entire MBPP dataset "1+ Correct" refers to the percentage of tasks for which CODEGEN-MONO 6.1B generated at least one program that passed all unit tests.
Zero-Shot CODEGEN-Men Moxo
Metric
6.1B
Pass@1
31%
Pass@10
63%
1+ Correct
67%
CodeGen-Mono 6.1B Incorporates Feedback We first verify that baseline model can use feedback to repair incorrect code, prerequisite for ILF to work. We evaluate CODEGEN-MONO 6.1B's ability to generate refinements given pairs of (incorrect code, natural language feedback), both in few-shot manner and after fine-tuning. Feedback is only required for tasks for which my is initially unable to produce a correct response, so we first evaluate CODEGEN-MONG B. zero-shot on all of MBPP, generating 30 programs per task with temperature 0.8. Table shows the resulting pass rates. There were 321 tasks for which zero-shot
uwu. www.ehg.ghh.....
8

---

Published in Transactions on Machine Learning Research (02/2024)
Table 2: Evaluations of 1-shot refinements generated by CODEGEN-MONO 6.1B (before ILF) given either related or unrelated text feedback in the prompt. Feedback is provided only for tasks on which CODEGEN- MoNo 6, 6.1B previously did not output any correct programs.
CODEGEN-MONO 6.1B
Prompt Type
Pass01 Pass@10 f
Code + feedback
2.0%
13.8%
Code + unrelated feedback
0.4%
4.0%
CoDeGRN-MONo 6.1B yielded no correct samples (from Table 1 (100% - 67%) x 974 tasks ss 321). We then annotate one incorrect program per task with both feodback and refinement, as described in Section
2
Few-Shot Feedback Incorporation We use the human feedback annotations to create few-shot feedback prompts, formatted as in Figure 7 We evaluate CODeGEN-MONO 6.1 ability to produce refinements that incorporate the feedback and pass the unit tests. However, producing a refinement that passes the unit tests does not guarantee that the feedback has been incorporated; there can be multiple solutions to a programming task, including ones that are functional but completely different and not using the feedback to improve upon the original code. Alternatively, the model may already be able to repair programs without feedback. Thus, we also evaluate the pass rate after shuflling the leedback samples in the dataset, to evaluate if the model's ability to repair code degrades when presented with unrelated feedback.
The results are shown in Table 2 CODEGEN-MONO 6.1B's ability to incorporate relevant foedback on this particular set of program is low, with pass@10 reaching only 13.8%. However, the gap in accuracy between CODeGEN-MONO 6.1.-e....ee..... -egeerrrte refinements on relevant versus irrelevant feedback is significant, with pass@10 decrcasing by 71% (relative; 13.8% - 4.0%), indicating that the model is indeed using the feedback.
Training Reefne Next, we examine whether we can improve our ability to repair programs given feodback by fine-tuning a separate model specifically to perform this task. As mentioned prior in Section 3.1 ILF does not strictly require training a separate model for TRefine A large pre-trained language model may be able to refine the model outputs in few-shot manner, without further training. However, due to the closed nature of some models and the amount of compute and API funds required for larger models, we also choose to fine-tune our own Reefne Our training examples consist of triples of incorrect program, human-written feedback, and human-written refinement. We train the model to maximize the likelihood of the refinement given the program and leedback. The incorrect programs are generated by CODEGEN-MOws 6. 1B zero-shot on MBPP tasks, and the feedback and refinements are written by human annotators, as discussed in Section 3.2 We only include tasks for which none of CODEGEN-MONO 6.1B's generated programs are correct, yielding 44 tasks in the training dataset (forming the split MBPP and 128 tasks in the evaluation dataset (forming the split MBPPTrain). We ask human annotators to write refinements of the original code that incorporated their own previously written feedback, passed the unit tests, and make only minimal edits to the code (see Section 3.2) The format of the training data also matches the few-shot prompt format (Figure 7 but without the in-context examples of refinements. We denote this model as Refine as described in Scction
Table 3 shows the pass rates for TRefine on MBPP which were produced by sampling 30 refinements per task with temperature 0.8. Fine-tuning significantly improves CODEGEN-MONO 6.1 ability to incorpo- rate foedback compared to 1-shot refinement, incrcasing pass rates more than threc-fold (2-19% pass@1, 13.8+447% pass@10, from Tables 2 and 3). Furthermore, 61% of tasks had at least one correct refinement. This is particularly significant when considering the fact that we selected only tasks for which a non-fin tind--nteedd MoNo 6. model did not originally output any correct programs for (the rightmost column Table 3). For the 61% of validation tasks that Retuee generated correct refinement for, we randomly selected one such correct program for each task to form the training dataset for our final model nor; yielding
9

---

Published in Transactions on Machine Learning Research (02/2024)
Table 3: Pass rates of - refinements versus zero-shot CODEGEN-MONE 6, 1B programs for tasks in MrPPTrain- MBPPTrain contains only tasks for which CODeGEN-MONO 1B did not generate any correct completions.
Zero-shol
Metric
TRafin
CODEGEN- MoNo 6.1B
Pass@1 1
19%
0%
Pass@10
47%
0%
1+ correct
61%
0%
final training dataset of 78 examples. However, show in Appendix A.6 that Reefine 's ability to generate correct refinements declines monotonically as the number of bugs addressed in the feedback increases.
Table 4: Final performance of тT. on MBPP compared to other ablations and baselines. All results are calculated using 30 output samples with temperature 0.8. All the methods are built on the CODEGEN-MONO 6.1B model.
Method
Feedback Source
Fine- Tine-Tuning Data
Pass Rates of :T
Pass@1
Pass@10
ILF
Humans
Teefie refinements
36%
68%
1-shot InstructGPT
1-shot InstructGPT refinements
19%
55%
Ablations
2-shol InstructGPT
2-shot InstruciGPT refinements
25%
59%
Gold Standard
Human-written refinements
33%
68%
Baseline
MBPP ground truth programs
22%
63%
Zero-Shot
26%
59%
3.3
ILF Yields Pass Rates Higher Than Fine-Tuning on Ground Truth Data or Human- Written Programs Alone
Given that our refinements improve over the initial programs, now fine-tune the refinements to improve our code generation model. As discussed earlier, use the correct refinements (as evaluated by the unit tests) that Reefun generated for its evaluation dataset the training dataset for wg- Since Tg- is meant to generate code from natural language task description (rather than to incorporate feedback into a refinement)), the inputs of our training dalaset are the MBPP prompls and the largets are the 78 TRefine" generated refinements described in the previous section. We also compare the performance of v against that of CODeGEN-MONO 6. ovaluated in a zero-shot manner, CODEGEN-MONO 6.1B fine-tuned on the ground truth programs from the MBPP dataset, and MoDN 6.1B fine-tuned on our human- written refinements. In realistic cases, human-written refinements are not readily available, so we consider fine-tuning this dataset to be gold standard. ILF does not require ground truth programs or human- written refinements we merely use them here for evaluation. For all fine-tuning experiments, we train on programs corresponding to the same set of task IDs as the ones used in T 's training dataset.
Additionally, we evaluate the impact of ablating the human annotations in our algorithm by using an LLM in place of humans generate the feedback and refinements (replacing steps 3 and 4 in Algorithm 1 For the LLM, we use GPT-3.5 fine-tuned with Feedback Made Easy (FeedME; text-davinci-on1 on the OpenAI A We refer to this model InstructGPT, which is the scries of OpenAl models that FeedME belongs to (OpenAl 2022b). We use InstructGPT to generate both the feedback and refinements the original programs. We use the same prompt format as for Reefue (Figure 7 with two slight modifications: (1) the feedback prompt ends at "Feodback:" whereas the refinement prompt uses the entire prompt: (with the previously InstrnctGPP--genrraae foedback inserted after "Feedback:"), and (2) we use in-context
3Details htt.pp/ppaaccccoo///eeee/nneeeee
10

---

Published in Transactions on Machine Learning Research (02/2024)
examples, with &o indicated in Table 4 We then fine-tune CODEGEN-MONG 6.1B the model-generated refinements.
The results of our ILF algorithm compared to the baselines and ablations are shown in Table ILF yields the highest pass@1 and pass@10 10 rates, despite how few samples of feedback and refinements we use. The passâ1 rate particular shows a significant. increase in improvement over the baseline of fine-tuning on MBPP ground truth programs, increasing from 22% 36%. Pass@1 improvements are especially helpful for assisting with software engineering, where it is more helpful to suggest single correct completion rather than 10 possible completions for the user to select from.
ILF also outperforms both zero-shot inference and fine-tuning on human-written refinements on the pass@] metric, yielding increases of 26% + 36% and 33% + 36% in pass&1 rales, respectively. However, training on human- human-written written refinements yielded comparable pass@ 10 rates as ILF, which unsurprising since Refuue was trained on human-writter refinements. When human-written foedback and TRteiie-gencral refinements are ablated (the "Ablations" section of Table ILF also outperforms training both 1-shot and 2-shot InstructGPT-geneeve refinements, with increases of 19% + 36% and 25% - 36%, respectively.
Training data source MBPP gold programs
50
Mettiee refinements
40
Human-written refinements
Count
30
20
10
0
10
10
102
103
10
Perplexity
Figure 3: Histogram of the perplexities of the various training data sources, as measured using a pre-trained CODeGEN-MONO 6.1B model.
Analysis of Training Data Sources However, we also note the surprising fact that merely training small sample of the MBPP gold programs did not make significant difference in accuracy over zero-shot inference. We speculate that the gold programs from the MBPP dataset may be somewhat out-of- distribution for CODeGEN-MONO 6. B. To test this hypothesis, gold programs, the Prefine-generatee refinements, and the human-written refinements using the pre-trained CODEGEN-MONO 6. model. The results are shown in Figure 3 While the distributions all three data sources look similar, the MBPP dataset contains more high-perplexity programs (i.e. programs with perplexity 102) than cither the -eneate-e refinements or the human-written refinements. As a result, it likely easier for CODEGIN--ONO 6. BB to learn from the latter two datasets, since they are closer to BODEGEN-MONG 6B's original distribution while still being functionally correct.
computed the perplexity of the MBPP
Furthermore, ILF is particularly useful for settings where large amounts of gold codc are not available. In this setting, ILF can be thought of as a method of not only generating more training data, but training data that is closer to the model's original outputs in data representation space and that specifically repairs the kinds of bugs that the original model generates. As result, fine-tuning the model on Refine TRefine-generee generated refinements does not require adjusting the weights as much as fino-tuning the model on the MBPP gold programs would, even though both training datasets contain the same number of functionally correct. programs.
11

---

Published in Transactions on Machine Learning Research (02/2024)
% with one passed completion
Poss@1
Pass@10 *
*
*
35%
65%
75%
CodcGen-Mono trained s Instru: refinements
32%
70%
CodeGen Mana 6.18 trained InstructGPT refinements
30%
65%
55%
CodeGen-Mono 6.18 trained Baer generaled refinementts
28%
*
60%
25%
50%
CodeGen-Mono
55%
22%
50
100
150
200
50
100
150
200
150 Number w/ reedback
100
200
Number Teedback
Number feedback
Figure 4: Training dataset size versus CODEGEN-MONT 6.1B pass rates MBPP tasks 11-111 after fine- luning on approach). X marks the performance of πœ., whereas the solid lines plot the performance of CODEGEN- MoNO 6.1B after fine-tuning on correct refinements generated by InstructGPT. using feedback also generated by InT. The dashed line indicates the zero-shot pass rate of a pre-trained CODEGEN-MONO 6.1B
refinements, versus the performance of Tо- (the model produced by our
model.
3.4 Scaling Up Model Feedback Does Not Offer the Same Benefits As Human Feedback
Since high quality human feedback can be expensive to collect, we also evaluated how much model feedback might yield the same benefit as our sample of human-written feedback. To do so, we randomly select & tasks from the set of MBPP tasks for which CODEGEN-MONO 6. 1B did not originally output a correct answer, and prompt InstructGPT to generate both the feedback and the refinemnt. We then evaluate the refinements for correctness and train CODEGEN-MONO 6. BB the correct refinements. We use & E {50, 100, 200) and generate 30 output samples at temperature 0.8 for all stages of the experiment. We are limited to these k values due to the small number of tasks have in MBPPTrain: but future work may investigate scaling up these experiments by using larger datasets or automatically generating new tasks and unit tests for the training dataset. Further training details are listed in Appendix A.2
The results are shown in Figure 4 Although increasing the quantity of InstruelGPT-geennnaee leedback offers modest improvements in pass rates, these improvements do not yield pass rates as high as those of πe- even though Tg- uses only total of 122 pieces of feedback throughout its training process (44 for training Reefne and 78 for generating refinements to train лa- on). However, as pre-trained large language models continue to improve dramatically in quality, we expect that this gap between human- and model-written feedback will increasingly narrow.
We also analyze some of the qualitative differences between human-written and model-generated feedback in Appendix A.5 and provide some specific examples in Appendix A.7 In summary, model-generated feedback is more likely to be less correct, less relevant, and less informative (i.e. addresses fewer bugs) than human- written feedback.
4 Summarization
We also adapt and evaluate ILF on the real-world task of text summarization. Applying ILF to summa- rization differs from applying it to code generation in a number of ways. Unlike code generation, text summarization has no automated method for evaluation that is completely precise. Although automated metrics such ROUGE, BLEU, and METEOR exist, past work has shown that these metrics often do not correlate highly with human-rated evaluations of summarization quality, especially on aspects related to coherence and relevance (Fabbri et al. 2021 As such, we utilize human evaluations to compare the per- formance of ILF versus baseline methods on text summarization. We also utilize an instruction-fine-tunee text-davinci-001 model to rank the generated refinements and select which ones to train on.
Furthermore, we assume that all summaries generated by тe can be improved upon. This climinates the need to filter for only low-quality initial outputs and simplifies the proposal distribution to Equation 6 Taken
12

---

Published in Transactions on Machine Learning Research (02/2024)
together, the high-level algorithm for ILF applied to summarization is shown in Figure 5 Algorithm 2
and detailed in
4.1 ILF for Text Summarization
Give an example of lowercase vowel.
In text summarization, each task consists of context (a source document) that we aim to generate improved outputs 21 (e.g,, high-quality summaries) for, according to human preferences. As before, ø1 are refinements of initial outputs Zo that are generated via incorporating language foedback f. To implement the reward function R (Trom Equation 2), we condition an instruction-lin- tuned LLM on binary question such as Does this new text [x1] incorporate the feedback [f] provided on the initial leel [ro] ? Answer Yes or No., where the label is either Ugood ("Yes") or Ybad ("No") Then if we let I be an indicator variable that indicates whether I1 is a high- quality summary of c, we can approximate the reward function R using the probability of the positive answer Ugood, ie:::
"A" vowel.
Make it lowercase.
A
Is feedback incorporated?
cosO voAil.
a
Figure 5: In ILF for text summarization, Reffin generates refinements of the original output based on the feedback. We use an LM to pick the best refinementt and funettune the original LM to maximize the likelihood of the chosen refinement.
C( = Pr(xi a high-quality summary of c)
(14)
c)
(By desinition)
(15)
p(I C,
(Conditional independence)
(16)
Plygood zo: f .))
(17)
(ygood Fo, f, I1, c) p(ybad, To. f.
where we assume that I, 31, and a are couditionally independent given wi and f.
4.2 Dataset
We evaluate the effectiveness of ILF on the task of text summarization using the TL;DR, dataset Völske et al. 2017). which consists of Reddit titles, posts, and their corresponding summaries. [Steennnn al. (2020) adapt this dataset and show that it is more realistic task for evaluating summarization models compared
to the commonly used CNN/DM dataset (Hermann et al. 2015). To ensure the quality of our dataset, we
follow the same preprocessing steps outlined in Stiennon et al. (020 and extract train dataset with 5000 samples, development datasct with 200 samples, a validation datasct with 500 samples, and test dataset with 698 sample
We then hire experienced annotators through Surge A to create our language feedback dataset, which we open source along with our code!" For each sample, we first generate three summarics for each Reddit post using the instruction- instruction-fin--tunel model text-davincii-011 (FeedME) (Ouyang al.) 2022 OpenAT| 2022b). Two of these summaries are used for a binary comparison, in which annotators indicate their preference. The third summary serves as the initial output for which we solicit language feedback. This feedback should address the single most important shorteoming of the summary and can be related to coverage (how well the summary covers the important information in the post), accuracy (the factual accuracy of the summary), coherence (the coherence the summary on its own), or other. We do not impose any restrictions on how the feedback should be written. In addition to providing feedback, annotators are also asked to write an ideal
4For the complete prompts, please see Appendix B.2 5The train and development datasets taken from Stiennon et al. (220 's train dataset, and the validation and test set.
taken from their test. dataset.
9https:/ //spg//g///rgraaaaa
Data: httpp:://hhtt htatpp://bugginggace..o http:s//aagiiiaaae...ooattttttts
Code:
learning from_ law.iin__rraaa_aaaaaeeeeeaa language_ faanuuggeeeeebbckkk feedback
13

---

Published in Transactions on Machine Learning Research (02/2024)
summary that is maximally 48 tokens long. The same crowd worker annotates all three tasks for given sample. Overall, the dataset collection and human evaluations cost 40KS. On selected samples of the binary comparison task, we achieve an of 70.0%. The human summarics we collect are of excellent quality, as demonstrated in a human evaluation, where we compare our human-writden summaries to the ones aulomatically extracted from Reddit (Völske et al.) 2017) (also used as basclines in Stiennon al. (2020): Scheurer et al. (2022))- We find that our hunan-written summaries are preferred 3.20 of the time, making them a much stronger baseline than the Reddit-soureed summaries.
agreement of 81.0% and annotator-annotato agreement
Algorithm 2 Imitation learning from natural language feedback for text summarization.
1: Input: Number of iterations K, sequence of sets of source documents C = [C1. Cx], language model To, refinement language model TRefine; reward model It
2: for & in I...K do Initialize fine-tuning dataset DE = 00
for document in Cg do
To (Faoec))
Human provides feedback f on (c, (0) [xi....,zI]] TRefine (21/e,
7:
8:
x1 = argmax,, R(x [ (cc)
D The highest-reward refinement is selected.
Add (c,z1) to Dr 10: end for
11: Tg- < FINETUNE(TT; D2)
12: end for
4.3 Refinement Ranking and Selection Methods
To select the best refinements to train To on, compare the following methods.
Generating Refinements We condition FeodME the initial summaries of our train dataset (generated with FeedME) and the hunan-written feedback and generate 5 refinements zi...., 2] using the instructions in App. B.1
Scoring Refinements with InstructRM We chose refinement with scoring function R that scores refinements for how effectively they incorporate feedback. For R we use the instruction-fine-tunee LM FeodME and ask it whether refinement is better than the initial summary (see $4.1 for more details). Wo then evaluate the probability that the refinement incorporates language feedback on the initial summary and accordingly a high-quality summary, ie., P(ygood prompt). LMs are sensitive to the exact prompt used (Perez et al.] 2021 Lu et al. 2021), so we write 5 different prompts (soc App. B.2 and select the refinement with the highest average p(Mgood (prompt and call this method InstructRM Ensemble.
Scoring Refinements with Embedding Similarity Previous work (Scheurer et al. 2022) use a con- trastive pre-trained text-embedding function (Neelakantan al. 2022] to embed the feedback f and refine- ments =1,-..,27 27 and select the refinement with the highest cosine similarity to the foedback. They this scoring function because foedback would often describe what the ideal text should look like. This method is less general because it assumes that good refinements are semantically similar to the feedback, which is not necessarily the case for all tasks or forms of feedback.
4.3.1 Refinement Ranking Results
We now evaluate the above ranking methods on the development dataset by calculating the fraction of times the refinement selected by a mothod better than a randomly-sclocted refinement ("win rate"), according ranking given by human evaluators (see App. B.5 for more details). The results, shown Table 3 show that the embedding similarity selection does not outperform random selection, while most (4/5) Instruct
14

---

Published in Transactions on Machine Learning Research (02/2024)
Table 5: We compare various ranking methods for selecting refinements using a human evaluation. Instruc- tRM Ensemble is used throughout our paper.
Scoring Function
Win Rate % vs. Random Selection
Speciifc
Max Length
65.0 + 2.7
Heuristic
Embedding Similarity
48.3 + 3.0
InstructRM Prompt
55.0 3.0
InstructRM Prompt
58.0 2.9
Zero-Shot
InstructRM Prompt
56.5 2.9
InstructRM Prompt
55.8 2.8
RM Prompt
50.0 3.0
InstructRM Ensemble
56.0 + 3.0
ILF OPT-RM (best-of-64)
OPT-RM best-af-66 64 FeedME
ILF: Fine-tuned Refinements
Fine-tuned Human Summaries
Fine-tuned Initial Summaries
FeedME (zero-shot)
50
Human Summary (%)
45
Win rate vs.
35
30
20
15
100
1000 Train Dataset Size
5000
Figure 6: The rate with which human evaluators prefer summaries generaled by all methods. The solid lines mark the fine-tuning methods and the dashed brown line marks zero shot generation. "ILF + OPT- RM (best-of-64)" combines learning from natural language feedback and binary preferences, and "OPT-RM best-of-64 FeedME" uses learning from binary preferences to rank and select the best summaries generated by FeedME. "ILF OPT-RM (best-of-64)" generates summaries of similar quality to human summaries.
prompts do. While the embedding similarity worked well in previous work (Scheurer al. 2022 it does perform well on our dataset. We believe this is because the feedback collect, written by many annotators, is much more diverse, while in Scheurer et al. 20022) the authors wrote the feedback themselves. InstructRM Ensemble has a win rate of 56.0 + 3.0% against random selection, demonstrating that an LM can evaluate its own output to some extent. Based on these results, we recommend using the InstructRM Ensemble approach, as performs well and is less sensitive to the particular prompt.
Throughout our paper, we use InstructRM Ensemble as our scoring function to select refinements and refer to our method of generating and selecting refinements Reforement with Feedback Best of N.
15

---

Published in Transactions on Machine Learning Research (02/2024)
4.4 Comparing Feedback Learning Algorithms
In this section, we compare ILF against other methods for learning from feedback, including learning from binary foodback and supervised fino-tuning. We present an overview of each method and then provide the results of our evaluations.
4.4.1 Feedback Learning Methods
Finc-tuning on Refinements (ILF) For this evaluation, we use a single itcration of ILF to learn from language leedback. We fine-tune GPT3-175B (davinci) (Brown al. 2020 to maximize the log-likelihood of the refinement given the imput prompt (consisting of the Reddit title, and post), ie., log p(z1p((appp |prompt), using the refinements generated with Refinement with Feedback + Best of N. For all our fine-tuning methods we add À log p(prompt) to the loss [Radford & Narasimhan 2018 OpenAl 2022a),, which maximizes the log-probability of the prompt. The prompt-loss weight À E [0. 1] is chosen on our development dataset (see paragraph DFine-tuning on Human Sunmaries"). The selected hyperparameters are detailed in App. C.4
and the fine-tuning prompts in App. B.3
Fine-tuning on Human Summaries Here we fine-tune GPT3-175B on the dataset of numan-written summarics Thuman: with the objective of maximizing the log- og-probability of human summarics given the input prompt (consisting of the Reddit title and post) with the additional loss term, i.c. log (t/Phnus/pprmppt + A log logp(prompt). To ensure the best performance of our fine-tuned models, we conduct thorough hyperpa- rameter tuning on the human-written written summary datasets of various sizes (100, 1K, 5K). The hyperparameters optimized include the number of training epochs, the prompt loss weight X, and the learning rate multiplier, as detailed in the OpenAl documentation (OpenAT 2022a). We use the perplexity of the predicted summaries on the development dataset to select the most effective hyperparameters. The selected hyperparameters are applied to all datasets, i.e,, fine-tuning on refinements, initial summaries, and human-written sunmaries, with the same sample size. More details on hyperparameter tuning can be found in Appendix C.4
Fine-tuning on Initial Summaries We fine-tune GPT3-175B on the dataset of initial summaries (gen- crated by FeedME). The objective is maximize the log probability of the initial summary given the prompt (consisting of the Reddit. title and post)) with the additional loss term i.e. log logp( (oompprropp)--.. + A loggp(poopp) Details on hyperparameter tuning can be found in the paragraph Fine-tuning Human Summaries and Appendix C.4
Learning from Binary Feedback: Best-of-N N We compare ILF against binary feedback a baseline, the standard approach for learning from feedback. One way of learning from binary feedback is to train a reward model and use it to do best-of-N sampling. We use best-of-N because it is often competitive with RL from human feedback (Nakano et al. 2021), a highly effective but more sophisticated approach (Stiemoon et al. [2020]; ; Ouyang et al. (2022). To train the RM, we fine-tune OPT-13B (OPT-RM) (Zhang et al. 2022 to classify whether a summary zo is high quality or not. To do so, we use the instruction Is the above an excellent summary of the given text? An ercellent summary is coherent, accurate, concise, and detailed. Answer with Yes or No., where the label y is either Ugoad (" Yes") or yoad (5 No'). Given human labels on which of two sunmaries is preferred, we label the preferred summary with Bgood and the other summary with Ybad- We then fine-tune the LM to maximize logp(v)==) + A Alog p(zo), where À E [0, 1], chosen using the development dataset, and E {ygood, Bbod]. Using the fine-tuner tuned LM, we evaluate a given summary by computing (()gooa/Zo) and select the summary with the higher probability. We find that this approach leads to more accurate RMs than other RM training hots such as the commonly used method from Stiennon et al. (2020); see Appendix C.3 for comparisons and Appendix B.4 for the used prompts. We perform Bayesian hyperparameter optimization for OPT-RM and sweep over the learning rate, batch size, and prompt-loss weight X, using classification accuracy on the development dataset as the selection criteria (see Appendix C.4 for more details).
MF cannot be fine-tuned via OpenAT's APT.
16

---

Published in Transactions on Machine Learning Research (02/2024)
ILF + Learning from Binary Feedback As a final step, combine ILF and learning from binary leedback, by first fine-tuning GPT3-175B on the refinements as described in the paragraph fine-tuning on refinements (ILF). We then train the reward model, OPT-RM, and use it to perform best-of-N sampling, as outlined in the paragraph on learning from binary foedback. At test time, we generate 64 summaries with our fine-Luned model and rank them based on their probability being a high-quality summary, Puorm (Hgoaa (o), using OPT-RM. The summary with the highest normalized probability is then selected.
4.4.2 Evaluation of Feedback Learning Methods
We evaluate the effectiveness of our learning algorithm, by comparing it to human written reference sum- maries, several fine-tuning baselines, and OPT-RM on the task of text summarization using 100, 1K, and 5K train samples. Using test. dataset of 698 samples, we generate a summary for each method and evaluate them with human evaluators who rank them based on quality, using standard ranking scheme that allows for lies between summaries (see App. C.4 for more details). Based on the rankings, calculate the fraction of times each method's sampled summary outperforms the human- written reference summary, referred to as the "win rate". We sample summarics up to 48 tokens in length (as in Stiennon et al. (2020) using muclous sampling (Holtzman et al. 2019 with p = 0.95 and temperature t = 1.0 (see App. C.4 for further details on hyperparameters and postprocessing). We use best-of-64 sampling with summaries sampled from FeedME for learning from binary feedback.
4.4.3 Results
ILF Versus Other Fino-Tuning Methods Our results, shown in Fig. G demonstrate that fine-tuning on refinements (ILF) ourperorss all other fine-tuning method including sampling from FeedME, with win rate against human summaries of 1.73 (for fine-tuning on 5K samples), while the other methods achieve win rates of 27.3 = 1.7% (fine-tuning on initial summarics), 28.9 + 1.7% (fine-tuning on human summarios). and 22.5 + 1.6% (FeedME). It is surprising that ILF outperforms fine-tuning on human summaries across all sample sizes, despite human-written summaries generally being of higher quality (see Fig. 15 top). Further evalustion (see App. C.8 Fig. II shows that the model fine-tuned on 1K refinements (ILF) exhibits significantly lower loss when evaluated on the validation dataset of refinements compared the model fine- tuned on human summaries when evaluated on the validation dataset of human summaries, suggesting that the model is more adept at approximating the distribution of refinements. Overall, these results demonstrate the effectiveness of our proposed ILF approach in accurately incorporating feedback and improving model performance, even outperforming fine-tuning on human summaries.
Schourer et al. 2022) found that ILF with 100 foodback samples outperformed FeedME, while here we find it underperforms FeedME with 100 feedback samples. Prior work uses author-written feedback that often conveys what the refinement should include, while our work includes more varied, crowdsourced feedback. As a result, we observe that embedding similarity does not properly rank refinements on our human feedback dataset (Table 5) and believe the difference in feedback may be significant source of differences in results in this section as well; see Appendix C.8.4 for more discussion.
Combining ILF With Learning From Binary Feedback Our results demonstrate that using OPT-RM for best-of-64 sampling on FeedME summaries outperforms all fine-tuning methods and sampling approaches across all sample sizes. The improved performance of OPT-RM best-of-64 FeedME comes at the cost of added inference time for best-of-N sampling. Combining ILF and learning from binary feedback (ILF + OPT-RM (best-of-64)) achieves human-level summarization performance with win rate of 50.8 = 1.9% using 5K samples for training. This suggests that both methods independently learn valuable information about human preferences that can be cunulative when used logether. It should be noted that the result for ILF + OPT-RM (best-of-64) is obtained through a separate human evaluation with different comparison summarics (see App. Fig. 12 and was added to Fig. for reference.
Fine-tuning on 100 refinements tied with fine-tuning 100 initial summaries.
17

---

Published in Transactions on Machine Learning Research (02/2024)
Lastly, we provide further analyses in Appendixes C.8.5 and C.9 demonstrating that the feedback is important for high-quality refinements and that the refinements do frequently incorporate the most important point in the feedback.
5 Related Work
Learning from Human Feedback Our algorithm is inspired by a number of past works that have trained models to learn from leedback. A common lechnique reinforcement learning from human feedback (RLHF Christiano et a 2017: Ziegler et al., 2019 Stiennon et al.) 2020 Ouyang et al., 2022). which trains models to satisfy human preferences. However, our algorithm is closer to works that use natural language feedback, rather than comparisons between different choices. Elgohary et al. (2020): : Austin et al. [202)) ; Nijkamp
et al. 2022 all demonstrate that code LLM performance generally improves when prompted with natural language feedback, though Nijkamp et al. (2022) observes that the feedback is more effective when it is given one step at a time. Our work differs from these in that ILF learns from the fcedback at training time, not al inference time.
Bai et al. (2022a) also uses natural language feedback during the training process, but as part of an RLHF algorithm instead where the feedback is used to solicit different responses from the digital assistant, the responses are ranked by crowdworkers, and the rankings are used to train the preference model. However, they note that this form of learning from natural language feedback does not measurably improve their code generation model more than simply prompting.
Other work aims to use natural language as a learning signal in other ways. Radford et al. 2019); Raffel ct al. (2020), and Brown al. (2020) investigate using explanations for gold labeled outputs of classification tasks, whereas our work addresses the more general text generation setting. Furthermore, explanations are notably different from feedback explanations describe why labeled output. is correct, whereas feedback describes how to improve candidate output. Explanations can also be used to train text classification
models, with mixed results (Camburu et al.
2018
Stacey et al. 2021 Pruthi et al. 2021 Wiegreffe et al. inter alia). In addition, some work learns from language
2021
Hase & Bansal 2021 Lampinen et al.
2022
feedback for ranking rather than generating outputs (Weston 2016 Li et al. 2016 Hancock et al.) 2019
ett al. 2022b Xu et al.] 2022b). Matiana et al.
(2021) learn text embeddings of language feedback, where improvements could benefit the refnemeet--coring step of our algorithm.
Language has also been widely used in RL settings for various purposes (sec Luketina et al.
2019 for
overview), such specifying tasks ("instruction following"; Wei et al. 2022a Ouyang et al.
2022 inter
alia), driving exploration (Tam et al., 2022), inferring reward functions (Lin et al 2022 Sumers et al. 2021
Fidler et al. 2017] inter alia), and training a model via strong supervision Andreas et al. (2017): Kaplan et al. (2017). reward shaping Goyal et al. (2019), or by providing descriptions of trajectories (Nguyen et al. 2021 contrast, we use language to correct faulty behavior. Other work uses language feedback at test time to correct mistakes in model's behavior, e.g., image segmentation (Rupprecht al. (2018) or code generation Elgohary et al. (2020); Austin et al. (2021). In contrast, we use feedback to train models, and our approach does not require human intervention al lest lime.
Lastly, our work builds upon our previous report (Scheurer et al. 2022), which showed that LLMs can refine outputs with language foodback. There, we introduced the same threc-step algorithm that ILF builds upon
and applied the algorithm only to text summarization. Our present work extends (Scheurer et al. 2022 in
a number of ways, including: (1) formalizing the theoretical justification of ILF (Section 2 (2) adapting
ILF to multiple tasks (e.g. both code generation and text summarization); summarization experiments analyzing the best methods for ranking refinements and selecting the best final summary; and (4) using larger-scale human crowdsoureing to both rate output summaries and generate feedback and refinements.
(3) conducting more extensive
LLMs for Program Synthesis Our work also builds on a large body of literature that explores the use of pre-trained LLMs for neural program synthesis. Many general purpose LLMss, although not pre-trained specifically for code generation, have demonstrated impressive proficiency at solving code challenges since they are pre-trained on large corpora of text such THE PILE (aoo et al. 2020) that contain small
18

---

Published in Transactions on Machine Learning Research (02/2024)
percentage of code content
Austin et al.
2021 Wang & Komatsuzaki 2021 Black et al. 2022 Nijkamp
et al. 2022h. Yet other recent L.L.s for program hesis are trained on solely source code files (Wang
et al. 2021 Zan et al. 2022 Li et al. 2022a; Xu et al. 2022a), or on both text and source code documents
sometimes either in succession (Chen et al.
2021; Nijkamp et al. 2022 Bai et al. 2022a), in a mixed corpus WWrkshop el al. 2022.. or on mixed natural language- pragllaggggogggmminggg language documents (Feng el al. 2020
Bayesian Inference for LLMs Several other works draw connections between Bayesian Inference and learning algorithms for LMs. Korbak et al. (2022) show that KL-regularised RL is equivalent to variational inference: approximating a Bayesian posterior which specifies how to updale a prior LM to conform with evidence provided by a reward function. Dohan et al. (2022) further argues that the process of generat- ing output through multiple rounds of interaction between prompted LMs and other agents (c.g. humans providing language leedback) can be seen as execuling probabilistic programs.
6 Conclusion
We have shown that ILF can significantly improve the quality of LLM generations for both code generation and text summarization, even with just small sample of human-written feedback and refinements. Combin- ing ILF with learning from binary preferences further improves the quality of the LLM's outputs. We have also shown that this approach is theoretically justified as minimizing the expected KL divergence between Te and a target ground-truth distribution, where we acquire signal from the latter via human-written natural language feedback.
This approach is also appealing because it is not model-specific (in the sense that ILF can be used with any type of base model ma, assuming the existence of sufficiently capable LLM to act as TRefine), and can be conducted in multiple rounds to continuously improve the model. ILF also does not involve training separate policies for modeling the reward signal and advantage values, as RI-based feedback learning algorithms often do (Stiennon et al. 2020). Furthermore, it is notable that our approach generates training data that is not only correct, but targets the specific kinds of weaknesses that are more likely in the given model's generations. In essence, provides an online and on-policy training signal that is missing from the offine pre-training and fine-tuning set-ups of modern LLMs. Our approach is also remarkably sample-efficient, improving the pass@1 rate from 22% to 36% compared to the baseline, despite fine-tuning we train on only 5000 examples to achieve approximately human-level summarization quality.
only 78 examples. For summarization,
Our work opens up multiple avenues for promising future work. For instance, ILF can be applicd iteratively over the course of multiple rounds whenever new information arrives (e.g. new Python syntax or world knowledge) or new bugs/quality issues are discovered. Our preliminary experiments in Appendix C.8.2 indicate that ILF benefits from the data gathered over multiple rounds, although continual fine-tuning may cause the model to occasionally forget past feedback. More extensive experimentation is required to study how to evfectively apply numerous rounds of ILF. Additionally, the tasks we study in this paper require only shorter outputs, and it remains to be seen how ILF generalizes to longer output sequences. It would also be useful to study how to generalize the implementation of ILF across a variety of domains and tasks. In this work, we designed separate reward functions for each task, but it would be beneficial to explore more generalizable reward functions that are easier to transfer between domains.
As the pace of progress of modern LLM research continues to accelerate, it may soon be feasible to par- tially or fully automate the generation of natural language feedback (similar to 'supervised learning from constitutional AI' (SL-CAT:: Bai et al. 2022b) and our experiments in Section 3.4 greatly reducing both the time and cost necessary for collecting feedback. This direction of work is also particularly appealing because the learning signal is process-basel rather than outcome-based, which has been shown to mitigate reward hacking and improve the correctness of intermediate reasoning steps (Uesato et al. 2022
Although further work required to extend our method, ILF represents an exciting step forward in training LLMs with feekback that is rich, interactive, and sample-efficient
19

---

Published in Transactions on Machine Learning Research (02/2024)
7 Acknowledgements
We are grateful to Nitarshan Rajkumar, Jason Phang, Nat McAleese, Geoffrey Irving, Jeff Wu, Jan Leike, Cathy Ych, William Saunders, Jonathan Ward, Daniel Ziegler, Scraphina Nix, Quintin Pope, Kay Kozaronek, Peter Hase, Talia Ringer, Asa Cooper Stickland, Jacob Pfau, David Lindner, Lennart Heim, Kath Lumpante, and Pablo Morena for helpful discussions and feedback about the design and implementation of this work. We are additionally thankful to Scott Heiner and Edwin Chen for extensive help with setting up our human annotation workflow and interface. We also thank the TMLR reviewers who provided valuable feedback that meaningfully improved this paper.
EP thanks the National Science Foundation and Open Philanthropy for fellowship support. JAC is supported by a doctoral grant from the Spanish MRCD. AC, SB, and KC are supported by National Science Foundation Awards 1922658 and 2046556. Any opinions, findings, and conclusions or recommendations expressed this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. KC is additionally supported by 42dot., Hyundai Motor Company (under the project. Uncertainly in Neural Sequence Modeling) and the Samsung Advanced Institute of Technology (under the project Next Generation Deep Learning: From Pattern Recognition to AI). This project has also benefited from financial support to SB by Eric and Wendy Schmidt (made by recommendation of the Schmidt Futures program), Open Philanthropy, and Apple. We also thank the NYU High-Performanue Computing Center for in-kind support and OpenAl for providing access to and credits for their models via the API Academic Access Program
References
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety, 2016.
Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with policy sketches. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 175- PMTR, 06-11 Aug 2017. URL pttps::/pproceednngs mlr www.sprwesssweseeeeeeee www.///////daaaa...m hhmml
Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carric J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. Program synthosis with large language models, 2021. URL
07732
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, T. J. Henighan, Nicholas Joseph, Saurav Kadavath, John Kernion, Tom Conerly, Sheer FllSShok,, Nelson Elhage, Zac Halfield-Doddss, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Christopher Olah, Benjamin Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback. ArXiv, abs/2204 hbs/2204.05862, 2022a.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatffeld-Docks, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional ai: Harmlessness from ai feedback, 2022b.
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. GPT-NcoX-2000 An open-source autoregressive language model. In Proceedings of the ACL Workshop on Challenges & Perspectives in Creating Large Language Models, 2022. URL httpps///artt arg/abs///n 2674.06745
20

---

Published in Transactions on Machine Learning Research (02/2024)
Samuel R. Bowman, Jecyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamilé Lukoštūté, Amanda Askell, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Christo- pher Olah, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Jackson Kernion, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Liane Lovitt, Nelson Elhage, Nicholas Schiefer, Nicholas Joseph, Noemí Mercado, Nova DasSarma, Robin Larson, Sam McCandlish, Sandipan Kundu, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Timothy Telleon- Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatlield-Doddss Ben Mann, and Jared Kaplan. Measuring progress scalable oversight for large language models. ArXiu, abs/2211.035400, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Berbert-Voss,, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few- shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Newral Information Processing Systems, volume 33, pp. 1807-1901. Curran Associates, Inc., 2020. URL https: Paper. Papeer.pp
//proceedings.nevie https//aww...aaaoooo///////////a
Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. e-snli: Natural language inference with natural language explanations. Advonces in Neural Information Processing Systems, 31, 2018. URL httts::/aaxii: wwwpexiv.org//df.. /181/ 1812. 011.2. pdd
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Eliza- both Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021. URL
2117 03374
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforce- ment learning from human preferences. Advances in neural information processing systems, 30, 2017. URL
org/pdf// https://arxiv http://aaxxvv...gppddoo hdf///df//// /1706 017411100600771...1111 pdf]
David Dohan, Winnie Xu, Aitor Lewkowycz, Jacob Austin, David Bieber, Raphael Gontijo Lopes, Yuhuai Wu, Henryk Michalewski, Rif A Saurous, Jascha Sohl-Dickstein, et al. Language model cascades. arXiv preprint ar Xrt::2207. 2022.
Ahmed Elgohary, Saghar Hosseini, and Ahmed Hassan Awadallah. Speak to your parser: Interactive text- to-SQL with natural language feedback. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2077, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1 10066633/11200.cl-mmm. aww.0.000mmmmii.ma.aaaa URL https://acllanthooogy) org.://0...gg/0000. acl-main....... 187.
Alexander R. Fabbri, Wojciech Kryšciński,, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. SummEval: Ro-ovaluating Summarization Evaluation. Transactions of the Associ- ation for Computational Linguistics, 9:391-409, April 2021. ISSN 2307-387X. doi: 10.1162/tacl. a_
00373. URL https: //do:.://di.... http///i..orgg////// 116./tacc..11 a_00373] eprint: https:/ /ttp///eretttt.coooo ceiiitdduuttat htt///ta//aatcc a 00373.pdf.
pdf/doi/10.1162// tacl
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. CodeBERT: A pro-trained model for programming and natural lan- guages. In Findings of the Association for Compulational Linguistics: FMNIP 2020, pp- 1536 1536-1547, On-
21

---

Published in Transactions on Machine Learning Research (02/2024)
line, November 2020. Association for Computational Linguistics. doi: (0.18653/v1/2020 www....innnngsnnnnl.p.. http:///w............/....ioieea findings-emnll 139
139. URL.
Sanja Fidler et al. Teaching machines to describe images with natural language leedback. Advances in Neural Information Processing Systems, 30, 2017.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027 2020.
Samuel Gchman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. Realtoxicityrrompt::: Evaluating neural toxic degeneration in language models. arXiv preprint or arXin:2009. 1 1146, 2020. URL
attps://acllnntoolog..oorg wtt../..oor//22....ooin.s...
301.pp.....pp.f
Mor Geva, Yoav Goldberg, and Jonathan Berant. Are we modeling the task or the annotator? an in- vestigation of annotator bias in natural language understanding datasets. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference Empirical Methods in Nat- ural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 1161 1166, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 11188/1///1111VVVV
URL htatp:://aclannhoooogyy htw.aaannhoooogooor/////-1-10000 org/D19-1107
Prasoon Goyal, Scott. Niekum, and Raymond J. Mooney. Using Natural Language for Reward Shaping in Reinforcement Learning, 2019.
Braden Hancock, Antoine Bordes, after deployment: Feed yourself, chatbot! arXiv preprint or Xiv: 1901:100100000 00..05545, 2019.
Mazare, and Jason Weston. Learning from dialogue
Jochen Hartmann, Jasper Schwenzow, and Maximilian Witte. The political ideology of conversational ai: Converging evidence on chatgpt's pro-envirommental, left-libertarian orientation, 2023.
Peter Hase and Mohit Bansal. When can models learn from explanations? a formal framework for understanding the roles of explanation data. arXiv preprint arXiv:2102.02201 2021. URL https:
22102 22201.2.2222.... 02201.222...
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. Advances in neural information processing systems, 28, 2015.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yojin Choi. The curious case of neural text degener- ation. arXiv preprinl ar Xrx:::1904. 099511, 2019. URL
1tt.://1994000091... 09751.pdf
Sungmin Kang, Juyeon Yoon, and Shin Yoo. Large language models are few-shot testers: Exploring llmeb general bug reproduction, 2022. URL https:://rxiv.org//aat
11515
Russell Kaplan, Christopher Sauer, and Alexander Sosa. Beating Atari with Natural Language Guided Reinforcement Learning, 2017.
Tomasz Korbak, Ethan Perez, and Christopher L Buckley. Rl with kl penalties is better viewed as bayesian inference. arXiv preprint arXiv:2205. 11275, 2022.
Sumith Kulal, Panupong Pasupat. Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy Liang. Spoc: Search-bascr basod pscudocode code. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alch-Buu. E. Fox; and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32, Cur- ran Associates, Inc., 2019. URL https::/prooceeee wewpeiinngnneurirpi www.nnuunniccmm http:////..aap...eee//////////// 726533200c000000444c6666666 Paper .df.
Andrew K Lampinen, Ishita Dasgupta, Stephanic CY Chan, Kory Matthewson, Michael Henry Tessler, Antonia Creswell, James L McClelland, Jane X Wang, and Felix Hill. Can language models learn from explanations in context? Xix preprint or arxi:2200 023229 2022.
22

---

Published in Transactions on Machine Learning Research (02/2024)
Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Saško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, An- gelina McMiilan-MMaoo, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. Datasets: A community library for natural language processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 175-184, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. URL https: fttps://aallanthologgs org/2021. emnlp-demo 211
Jiwei Li, Alexander H Miller, Sumit Chopra, Mare' Aurelio Ranzato, and Jason Weston, Dialogue learning with human-in-the-loop- arXiv preprint arxiv:161..m 09889, 2016.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Fe- cles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Mas- son d' d'Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-level code generation with alphacode. Science, 378 (062)) 1097, 2022a. doi: 10.1 00.118//scienceaabg15555 URL hhttps://wwat http::///ww.sccenceee science. org/ doi doi/abs/100 10. wwt.:/cie/.ccce...... abggi..iccnnne.....ge
Zichao Li, Prakhar Sharma, Xing Han Lu, Jackie CK Cheung, and Siva Reddy. Using interactive leedback to improve the accuracy and explainability of question answering systems post-deployment. arXiv preprint arXiv:2204.03022 2022b.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga Yian Zhang, Decpak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Alexander Cosgrove, Christopher D Manning, Christopher Re, Diana Acosta-Navas, Drew Arad Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue WANG, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksckgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri S. Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Andrew Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Teard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. Holistic evaluation of language models. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL Featured Certification, Expert Certification.
https://openrevieet http:://peerrrree.eeetf.rorrrrep mett:forummiddi001222bbaa
Jessy Lin, Daniel Fried, Dan Klein, and Anca Dragan. Inferring rewards from language in context. arXiv preprint ar Xrv::22004.000 02515, 2022.
Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring How Models Mimic Human False- hoods, 2021.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity, arXiv preprint 210:::1044007866, 2021.
Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefenstette, Shimon Whiteson, and Tim Rocktäschel. A survey of reinforcement Iearning informed by natural language. In Proceedings of the EirenyyFFighth International Joint Conference on Arlificial Indelligence, IJCAI-
19, pp. 6309 6317. International Joint Conferences on Artificial Intelligence Organization, 7 2019. doi: 10.24963 1224166/ijcai.2200 /ttp://aaa..../////// URL
Zohar Manna and Richard J. Waldinger. Toward automatic program synthesis. Commun. ACM, 14(3): 151-165, mar 1971. ISSN 0001-0782. doi: 362566.362568
URL htt:::///oo..org100/
Shahbuland Matiana, JR Smith, Ryan Techan Louis Castricato, Stella Biderman, Leo Gao, and Spencer Frazier. Cut the carp: Fishing for zero-shot story evaluation. arXiv preprint 3rXiv:2110.031 2021.
23

---

Published in Transactions on Machine Learning Research (02/2024)
Retichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet. Kosaraju, William Saunders, et al. WebGPT: Browser-assisted question-answwringg with human feedback. arXic preprint arXi::1122.09332, 2021. URL http:://axxxv.rr///aa//21111 org/paf/2112.
09332. pdf
Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, Johannes Heidecke, Pranav Shyam, Boris Power, Tyna Eloundou Nekoul, Girish Sastry, Gretchen Krueger, David Schnurr, Felipe Petroski Such, Kenny Hsu, Madelcine Thompson, Tabarak Khan, Toki Sherbakov, Joanne Jang, Peter Welinder, and Lilian Weng. Text and Code Embeddings by Contrastive Pre-Training, 2022.
Khanh X Nguyen, Dipendra Misra, Robert Schapire, Miroslav Dudík, and Patrick Shafto. Interactive learning from activity description. In International Conference on Machine Learning, pp. 8096- 8108. PMLR, 2021.
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint, 2022.
Augustus Odena, Charles Sutton, David Martin Dohan, Eilen Jiang, Henryk Michalewski, Jacob Austin, Maarten Paul Bosma, Maxwell Nye, Michael Terry, and Quoc V. Lc. Program synthesis with large language models. In n/a, pp. n/a, n/a, 2021. n/a.
OpenAL. Openai finetuning documentation. http::///oaaaaaoaaaooocccccrccct 2022a.
O OpenAy AL. Model index for researchers, 2022b.
URL httpss//plaafoor.oopma....
model -esee--iine
OpenAL. Gpt-4 technical report, 2023.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Preprint, 2022. URL https: //cdn. openai wwt.coom/pppe to |to_fooolooooo wwl.fooooosstteeeeee instructions with hwmannuaaaaaeeaaaaaaan fea.anffeebbbcckeee
language Language_models_ models
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High-Performan Deep Learning Library. In Advances in Neural Information Processing Systems 32, 2019.
Ethan Perez, Douwe Kiela, and Kyunghyun Cho. True few-shot learning with language models. Advances in Neural Information Processing Systems, 84:11054-11101 2021.
Ethan Perez, Sam Ringer, Kamile Lukosiute, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Benjamin Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Guro Khundadze, Jackson Kernion, James Landis, Jamie Kerr, Jared Mueller, Jecyoon Hyun, Joshua Landau, Kamal Ndousse, Landon Goldberg, Liane Lovitt, Martin Lucas, Michael Sellitto, Miranda. Zhang, Neerav Kingsland, Nelson Elhage, Nicholas Joseph, Noemi Mercado, Nova DasSarma, Oliver Rausch, Robin Larson, Sam McCandlish, Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Telleen-Lawton, Tom Brown, Tom Henighan, Tris- tan Hume, Yuntao Bai, Zac Hatfield-Dodds, Jack Clark, Samuel R. Bowman, Amanda Askell, Roger Grosse, Danny Hernandez, Deep Ganguli, Evan Hubinger, Nicholas Schiefer, and Jared Kaplan. Dis- covering language model behaviors with model-written evaluations. In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki (ods.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 13387-13434 Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 00.16633v112000 http://w1/222..nnnngeeeee findings-cll.e 847. URL httpss//aclaattalog...ooo http//223..fnddngs-aada. findings-acl..ee
24

---

Published in Transactions on Machine Learning Research (02/2024)
Danish Pruthi, Rachit Bansal, Bhuwan Dhingra, Livio Baldini Soares, Michael Collins, Zachary C. Lipton, Graham Neubig, and William W. Cohen. Evaluating Explanations: How much do explanations from the teacher aid students?, 2021.
Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. Stanza: A Python Natural Language Processing Toolkit for Many Human Languages. In Proceedings of the 58th Annual Meeting of the Associalion for Compulational Linguislics: System Demonstrations, 2020.
Alec Radford and Karthik Narasimhan. Improving Language Understanding by Generative Pre-Training, 2018. URL httpp:://openai-assette s3, amazonaws. http://www......ooo/oo/////////a pdf
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and 1lya Sutskever. Language
Models are Unsupervised Multitask Learners, 2019. URL https:///4muccppssyww. cloudfront, net/
hww.r-unguugeemddddss/llaaaaaee- _ateeeemmmll___eeeeeeeeeeeeeeeee are_
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffimann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Meth- ods, analysis & insights from training gopher. arXiv preprint Xiw:2112.114444 2021. URL https:
7/arxiv wttp:oogg/p////eeee 11446. pdf
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Trans- former, 2020.
Christian Rupprechl, Tro Laina, Nassir Navab, Gregory D Hager, and Federico Tombari. Guide me: In- teracting with deep networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8551-8561, 2018.
Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lec, Percy Liang, and Tatsunori Hashimoto. Whose opinions do language models reflect? In Proceedings of the 40th International Conference on Machine Learning, ICML/23 JMLR.org, 2023.
William Saunders, Catherine Ych, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike, Self-critiquing models for assisting human evaluators. arXiv preprint 22x::2206005800 2022.
Jérémy Scheurer, Jon Ander Campos, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. Training language models with language feedback. ACL Workshop on Learning with Natural Language Supervision, 2022. URL
absp:///b///222/// 14146
Gabriel Simmons. Moral mimicry: Large language models produce moral rationalizations tailored to political identity. In Vishakh Padmakumar, Gisela Vallejo, and Yao Fu (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), pp. 282-297, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/200 acl-srw.40. URL https://aclanthoooggy wwcplloogyoorgg//// org/2023. acl-srw.44
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonse Agnieszka Kluska, Aitor Lewkowyez, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Am- brose Slone, Ameet Rahane, Anantharaman S. Iyer, Anders Johan Andreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhlmüller, Andrew M. Dai, Andrew La, Andrew Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassun, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabhar- wal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakaş, B. Ryan Roberts, Bao Sheng Loe, Barret Zoph, Bartlomicj Bojanowski, Batuhan Ozyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin In- den, Benno Stein, Berk Ekmekei, Bill Yuchen Lin, Blake Howald, Bryan Orinion, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Argueta, Cesar Ferri, Chandan Singh, Charles Rathkopf, Chenlin Meng,
25

---

Published in Transactions on Machine Learning Research (02/2024)
Chitta Baral, Chiyu Wu, Chris Callison-Burhh, Christopher Waites, Christian Voigt, Christopher D Man- ning, Christopher Potts, Cindy Ramirez, Clara E. Rivera, Clemencia Siro, Colin Raffel, Courtney Ashcraft, Cristina Garbacea, Damien Sileo, Dan Garrette, Dan Hendrycks, Dan Kilman, Dan Roth, C. Daniel Free- man, Daniel Khashabi, Daniel Levy, Daniel Moseguí González, Danielle Perszyk, Danny Hernandez, Danqi Chen, Daphne Ippolito, Dar Gilboa, David Dohan, David Drakard, David Jurgens, Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dicuwke Hupkes, Diganta Misra, Dilyar Buzan, Dimitri Coelho Mollo, Diyi Yang, Dong-Ho Ho Lee, Dylan Schrader, Ekaterina Shutova, Ekin Dogus Cubuk, Elad Segal, Eleanor Hagerman, Elizabeth Barnes, Elizabeth Donoway, Ellie Pavlick, Emanuele Rodolå, Emma Lam, Eric Chu, Eric Tang, Erkut Erdem, Ernie Chang, Ethan A Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Eunice Engefu Manyasi, Evgenii Zheltonozhskii, Fanyue Xia, Fate- meh Siar, Fernando Martínez-Plumed, Francesca Happé, Francois Chollet, Frieda Rong, Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Germán Kruszewski, Giambattista Parascandolo, Giorgio Mariani, Gloria Xinyue Wang. Gonzalo Jaimovitch-Lopoz, Gregor Betz, Guy Gur-Ari, Hana Galijasovic, Han- nah Kim, Hannah Rashkin, Hannaneh Hajishirzi, Harsh Mehta, Hayden Bogar, Henry Francis Anthony Shevlin, Hinrich Schuetze, Hiromu Yakura, Hongming Zhang, Hugh Mee Wong, Ian Ng, Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kernion, Jacob Hilton, Jaehoon Lee, Jaime Fernández Fisac, James B Simon, James Koppel, James Zheng, James Zou, Jan Kocon, Jana Thompson, Janelle Wing- field, Jared Kaplan, Jarema Radom, Jascha Sohl-Dickstein, Jason Phang, Jason Wei, Jason Yosinski, Jekaterina Novikova, Jelle Bosscher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel, Jesujoba Alabi, Jiacheng Xu, Jiaming Song, Jillian Tang, Joan Waweru, John Burden, John Miller, John U. Balis, Jonathan Batchelder, Jonathan Berant, Jörg Frohberg, Jos Rozen, Jose Hernandez- Orallo, Joseph Boude- man, Joseph Guerr, Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan, Katerina Ignatyeva, Katja Markert, Kaustubh Dhole, Kevin Gimpel, Kevin Omondi, Kory Wallace Mathewson, Kristen Chiafullo, Ksenia Shkaruta, Ku- mar Shridhar, Kyle McDonell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang, Liam Dugan, Lianhui Qin, Lidia Controras-Ochando. Louis-Philippe Morency, Luca Moschella, Lucas Lam, Lucy Noble, Lud- wig Schmidt, Luheng He, Luis Oliveros-oolón, Luke Metz, Lütfi Kerem Senel, Maarten Bosma, Maarten Sap, Maartje Ter Hoeve, Maheen Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, Maria Jose Hamirez-Quintanee Maric Tolkiehn, Mario Giulianclli, Martha Lewis, Martin Potthast, Matthew L Leavitt, Matthias Hagen, Mátyás Schubert, Medina Orduna Baitemirova, Melody Arnaud, Melvin McFlrath, Michael Andrew Yee, Michael Cohen, Michael Gu, Michael Ivan- itskiy, Michael Starritt, Michael Strube, Michat Swedrowski, Michele Bevilacqua, Michihiro Yasunga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Mitch Walker, Mo Tiwari, Mohit Bansal, Moin Amin- naseri, Mor Geva, Mozhdeh Gheini, Mukund Varma T, Nanyun Peng, Nathan Andrew Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nicole Martinez, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar, Niveditha S. Iyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Pu Liang, Paul Vicol, Pegah Alipoormo- labashi, Peiyuan Liao, Percy Liang, Peter W Chang, Peter Eckersley, Phu Mon Htut, Pinyu Hwang, Piotr Mitkowski, Piyush Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, Qing Lyu, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ramon Risco, Raphaël Millière, Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymackers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan Le Bras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Russ Salakhutdi- nov, Ryan Andrew Chi, Seungjae Ryan Tee, Ryan Stovall, Ryan Techan, Rylan Yang, Sahib Singh, Saif M. Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel R. Bow- man, Samuel Stern Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebastian Gehrmann, Sebastian Schuster, Sepideh Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixiang Shane Gu, Shubh Pachchigar, Shubham Toshmiwal, Shyam Upadhyay, Shyamolima Shammie Debnath, Siamak Shakeri, Simon Thorneye,, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo-Hwan Lee, Spencer Torene, Sriharsha Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Stella Biderman, Stephanie Lin, Stephen Prasad, Steven Piantadosi, Stuart Shieber, Summer Misherghi, Svetlana Kiritchenko, Swa- roop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq Ali, Tatsunori Hashimoto, Te-Lin Wu, Théo Desbordes, Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili, Timo Schick, Timo-
26

---

Published in Transactions on Machine Learning Research (02/2024)
fei Kornev, Titus Tunduny, Tobias Gerstenberg, Trenton Chang, Trishala Necraj, Tushar Khot, Tyler Shultz, Uri Shaham, Vedant Misra, Vera Demberg, Victoria Nyamai, Vikas Raunak, Vinay Venkatesh Ra- masesh, vinay uday prabiu, Vishakh Padmakumar, Vivek Srikumar, William Fedus, William Saunders, William Zhang, Wout Vossen, Xiang Ren, Xiaoyu Tong, Xinran Zhao, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadel, Yair Lakretz, Yanggiu Song, Yasaman Bahri, Yejin Choi, Yichi Yang, Yiding Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yufang Hou, Yuntao Bai, Zachary Scid, Zhuoye Zhao, Zijian Wang, Zijie J. Wang, Zirui Wang, and Ziyi Wu. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL net/ meet/forum?ideee http://///peeeerreeeeea/e/////// teee//forumm1dddy/1SSBeeee
Joe Stacey, Yonatan Belinkov, and Marek Rei. Supervising Model Attention with Human Explanations for Robust Natural Language Inference. arXiv preprint (tli::2114008..... 2021. URL ftttps://arxiv.oom org/
pdf /df4. 08142.
Nisan Stiemnon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human leedback. Advances in Newral Information Processing Systems, 33:3008 3021, 2020. URL hattps///axxv.co
/tt.//.00........... pdf
Theodore R Sumers, Mark K Ho, Robert D Hawkins, Karthik Narasimhan, and Thomas L Griffiths. Learning rewards from linguistic foedback. feedback, 122):3, 2021.
Allison C Tam, Neil C Rabinowitz, Andrew K Lampinen, Nicholas A Roy, Stephanie CY Chan, DJ Strouse, Jane X Wang, Andrea Banino, and Felix Hill. Semantic exploration from language abstractions and pretrained representations. Xiv preprint arXiv:2204.0500 2022.
Rohan Taori and Tatsunori Hashimoto. Data feedback loops: Model-driven amplification of dataset biases. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volune 202 of Proceedings of Machine Learning Research, pp. 13883-33922 PMLR, 23 29 Jul 2023. URL https:// html
proceedings mlr. prress/v02/taaeeee
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang. Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process- and outcome-basse feedback, 2022.
Michael Völske, Martin Potthast, Shahbaz Syed, and Benno Stein. TL;DR: Mining Reddit to learn auto- matic sunmarivation. In Proceedings of the Workshop on New Frontiers in Summarization, pp. 59 63, Copenhagen, Demmark, September 2017. Association for Computational Linguistics. doi: 10.18653/v1/ W17-4508. URL
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github www./tingooollo//eeh---oooooo
jax, May 2021.
Yue Wang, Weishi Wang, Shafiq Joty, and Steven C.H. Hoi. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, 2021.
Jason Wei, Maarten Bosma, Vineent Zhao, Kelvin Gun, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V Le. Finetuned Language Models are Zero-Shot Learners. In International Conference on Learning Representations, 2022a. URL https::/tpppptt: httpa:///ppprrreenneeeeeeeeeeeee net./forum.dddgg2rGooaa
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoe V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alckh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022b. URL https://openreview wttpe/ewwwwwwwwtwteeeeeeee net/forum?id= ViQlMeSB
Jason E Weston. Dialog- Dislog-basod language learning. Advances in Neural Information Processing Systems, 29, 2016.
27

---

Published in Transactions on Machine Learning Research (02/2024)
Sarah Wiegreffe, Ana Marasović, and Noah A. Smith. Measuring association between labels and free-text rationales. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 10266 10284, Online and Punta Cana, Dominican Republic, November 2021. Association for Compu- tational Linguistics. doi: 11188533v1/2000 w1t/2/21.mmln-m/maaa 804. URL https: .ow.g.......................r... org/ 002/2021. emnlp-main.80 804
Thomas Wolf, Lysandre Debut,, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrutions, pp. 38 45, Online, October 2020. Association for Computational Linguistics. URL https: http:://www.wu wwtp:://wwwwwwwwwbbbbbb aclweb.wwbb........ waabboogg//ann hwngganthology//// /2020. emmlp-demos.e .
BigScience Workshop, :, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili,, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert. Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighoff, Albert Villanova. del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulmier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm,, Colin Leong, Daniel van Strien, David Ifeoluwa Adelani, Dragomir Radev, Eduardo González Ponferrada, Efrat Levkovizh, Ethan Kim, Eyal Bar Natan, Francesco De Toni, Gérard Dupont, Germán Kruszewski, Giada Pistilli, Hady Elsa- har, Hamza Benyamina, Hieu Tran, Ian Yu, Idris Abdulmumin, Isaac Johnson, Itziar Gonzalez-Dios Javier de la Rosa, Jenny Chim, Jesse Dodge, Jian Zhu, Jonathan Chang, Jörg Frohberg, Joseph Tobing, Joydeep Bhattacharjee, Khalid Almubarak, Kimbo Chen, Kyle Lo, Leandro Von Werra, Leon Weber, Long Phan, Loubna Ben allal, Ludovic Tanguy, Manan Dey, Manuel Romero Muñoz, Maraim Masoud, María Grandury, Mario Saško, Max Huang, Maximin Coavoux, Mayank Singh, Mike Tian-Jian J Jiang, Minh Chien Vu, Mohammad A. Janhar, Mustafa Ghaleb, Nishant Subramani, Nora Kassner, Nurulaqilla Khamis, Olivier Nguyen, Omar Espejel, Ona de Gibert, Paulo Villegas, Peter Henderson, Pierre Colombo, Priseilla Amuok, Quentin Lhoest, Rheza Harliman, Rishi Bommasani, Roberto Luis López, Rui Ribeiro, Salomey Osei, Sampo Pyysalo, Sebastian Nagel, Shamik Bose, Shamsuddeen Hassan Muhammad, Shanya Sharma, Shayne Longpre, Somaich Nikpoor, Stanislav Silberberg, Suhas Pai, Sydney Zink, Tiago Timponi Torrent, Timo Schiek, Tristan Thrush, Valentin Danchev, Vassilina Nikoulina, Veronika Laippala, Vio- lette Lepereq, Vrinda Prabhu, Zaid Alyafeai, Zeerak Talat, Arun Raja, Benjamin Heinzerling, Chenglei Si, Davut Emre Tagar, Elizabeth Salesky, Sabrina J. Mielko, Wilson Y. Lec, Abheesht Sharma, Andrea San- Gilli, Antoine Chaffin, Arnaud Stiegler, Debajyoti Datta, Eliza Szezechla, Gunjan Chhablani, Han Wang, Harshit Pandey, Hendrik Strobelt, Jason Alan Fries, Jos Rozen, Leo Gao, Lintang Sutawika, M Saiful Bari, Maged S. Al-shaibani, Matteo Manica, Nihal Nayak, Ryan Teehan, Samuel Albanie, Sheng Shen, Srulik Ben-David, Stephen H. Bach, Taewoon Kim, Tali Bers, Thibault Fevry, Trishala Necraj, Urmish Thakker, Vikas Raunak, Xiangru Tang, Zheng-Xin Yong, Zhiqing Sun, Shaked Brody, Yallow Uri, Hadar Tojarieh, Adam Roberts, Hyung Won Chung, Jaesung Tae, Jason Phang, Ofir Press, Conglong Li, Deepak Narayanan, Hatim Bourfoune, Jared Casper, Jeff Rasley, Max Ryabinin, Mayank Mishra, Minjia Zhang, Mohammad Shoeybi, Myriam Peyrounette, Nicolas Patry, Nouamane Tazi, Omar Sanseviero, Patrick von Platen, Pierre Cornette, Pierre François Lavallée, Rémi Lacroix, Samyam Rajbhandari, Sanchit Gandhi, Shaden Smith, Stéphane Requena, Suraj Patil, Tim Dettmers, Ahmed Baruwa, Amanpreet Singh, Anasta- sia Cheveleva, Anne- Ligozat, Arjun Subramonian, Aurélie Névéol, Charles Lovering, Dan Garrette, Deepak Tumuguntla, Ehud Reiter, Ekaterina Taktasheva, Ekaterina Voloshina, Eli Bogdanov, Genta In- dra Winata, Hailcy Schoelkopf, Jan- Chr-Chrittoph Kalo, Jekaterina Novikova, Jessica Zosa Forde, Jordan Clive, Jungo Kasai, Ken Kawamnra, Liam Hazan, Marine Carpuat, Miruna Clinciu, Najoung Kim, New- ton Cheng, Oleg Serikov, Omer Antverg, Oskar van der Wal, Rui Zhang, Ruochen Zhang, Sebastian Gehrmann, Shachar Mirkin, Shani Pais, Tatiana Shavrina, Thomas Scialom, Tian Yun, Tomasz Lim- isiewicz, Verena Rieser, Vitaly Protasov, Vladislav Mikhailov, Yada Pruksachatkun, Yonatan Belinkov, Zachary Bamberger, Zdeněk Kasner, Alice Rueda, Amanda Pestana, Amir Feizpour, Ammar Khan, Amy
28

---

Published in Transactions on Machine Learning Research (02/2024)
Faranak, Ana Santos, Anthony Hevia, Antigona Unldreaj. Arash Aghagol, Arezoo Abdollahi, Aycha Tam- mour, Azadeh HajiHosseini, Bahareh Behroozi, Benjamin Ajibade, Bharat Saxena, Carlos Muñoz Ferran- dis, Danish Contractor, David Lansky, Davis David, Douwe Kiela, Duong A. Nguyen, Edward Tan, Emi Baylor, Ezinwanne Ozoani, Fatima Mirza, Franklinc Ononivu, Habib Rezanejad, Hessie Jones, Indrani Bhattacharya, Trene Solaiman, Trina Sedenko, Isar Nejadgholi, Jesse Passmore, Josh Seltzer, Julio Bonis Sanz, Livia Dutra, Mairon Samagaio, Maraim Elbadri, Margot Mieskes, Marissa Gerchick, Martha Akin- Iolu, Michael McKenna, Mike Qiu, Muhammed Ghauri, Mykola Burynok, Nafis Abrar, Nazneen Rajani, Nour Elkott, Nour Fahmy, Olanrewaju Samuel, Ran An, Rasmus Kromann, Ryan Hao, Samira Alizadeh, Sarmad Shubber, Silas Wang, Sourav Roy, Sylvain Viguier, Thanh Te, Tobi Oyebade, Trieu Te, Yoyo Yang, Zach Nguyen, Abhinav Ramesh Kashyap, Alfredo Palasciano, Alison Callahan, Anima Shukla, An- tonio Eacaladd,,, Ayush Singh, Benjamin Beilharz, Bo Wang, Caio Brito, Chenxi Zhou, Chirag Jain, Chuxin Xu, Clémentine Fourrier, Daniel León Periñán, Daniel Molano, Dian Yu, Eurique Manjava- cas, Fabio Barth, Florian Fuhrimann, Gabriel Altay, Giyasoddin Bayrak, Gully Burns, Helena U. Vrabec, Imane Bello, Ishani Dash, Jihyun Kang, John Giorgi, Jonas Golde, Jose David Posada, Karthik Rangasai Sivaraman, Lokesh Bulchandani, Lu Liu, Luisa Shinzato, Madeleine Hahn de Bykhovetz, Maiko Takeuchi, Marc Pàmies, Maria A Castillo, Marianna Nezhurina, Mario Sänger, Matthias Samwald, Michael Cul- lan, Michael Weinberg, Michiel De Wolf, Mina Mihaljeic, Minna Liu, Moritz Freidank, Myungsun Kang, Natasha Seelam, Nathan Dahlberg, Nicholas Michio Broad, Nikolaus Muellner, Pascale Fung, Patrick Haller, Ramya Chandrasekhar, Renata Eisenberg, Robert Martin, Rodrigo Canalli, Rosaline Su, Ruisi Su, Samuel Cahyawijaya, Samuele Garda, Shlok Deshmukh, Shubhanshu Mishra, Sid Kiblawi, Simon Ott, Sinee Sang-aroonsiin Srishti Kumar, Stefan Schweter, Sushil Bharati, Tanmay Laud, Théo Gigant, Tomoya Kainuma, Wojeiech Kusa, Yanis Labrak, Yash Shailesh Bajaj, Yash Venkatraman, Yifan Xu, Yingxin Xu, Yu Xu, Zhe Tan, Zhongli Xie, Zifan Ye, Mathilde Bras, Younes Belkada, and Thomas Wolf. Bloom: A 176b- parameter ocen-aceess multilingual language model, 2022. URL https: //tp///aaxvvvv/a////////aa herg/abs/22111 05100
Frank F. Xu, Uri Alon, Graham Neubig, and Vincent Josua Hellendoorn. A systematic evaluation of large language models of code. Tn Proceedings of the 6th ACM SIGPLAN International Symposium Machine Programming, MAPS 2022, pp. 1 1-10, New York, NY, USA, 2022a. Association for Computing Machinery. ISBN 9781450392730. doi: 10.1445/5520311.3334883 3520312. 3534862
URL https:///oi. www.:oi..rg/.... 1145/
Jing Xu, Megan Ung, Mojtaba Komeili, Kushal Arora, Y-Lan Boureau, and Jason Weston. Learning new skills after deployment: Improving open-domain internet-driven dialogue with human feedback. arXiv preprint Xrx::2200.. 08270, 2022b.
Daoguang Zan, Bei Chen, Dejian Yang, Zeqi Lin, Minsu Kim, Bei Guan, Yongji Wang. Weizhu Chen, and Jian-Guang Lou. CERT: Continual pre-training on sketches for library-oriented code generation. In The 2022 International Joint Conference on Artificial Intelligence, 2022.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Lú, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXin preprint ar Xf0: 01068, 2022.
Albert Ziegler, Firini Kalliamvakou, X. Alice Li, Andrew Rice, Devon Rifkin, Shawn Simister, Ganesh Sittampalam, and Edward Aftandilian. Productivity assessment of neural code completion. In Proceedings of the 6th ACM SIGPLAN International Symposéum on Machine Programmingg MAPS 2022, pp. 29, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450392730. doi: 10.1145/
3520312.3534864. URL https://doi org//....rg/101 10. 1145/3520312 3534864
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Chris- tiano, and Geoffrey Irving. Fine-tuning language models from human preferences. ar preprint arXiv:1999. 08593, 2019. URL httpp::/ //arxiiv..vi.v///// org/////// htt////b///19..95.11
29

---

Published in Transactions on Machine Learning Research (02/2024)
A Appendix
A.1 Code Generation Prompt Formats
Prompt
Expected completion
OLD CODE:
...
Write python function to find the sum of the three lowest
Write python function find the sum of the three lowest
positive numbers from given list of numbers.
positive numbers from given list of numbera.
Example: sum three smallest nums ([10, 30, 20,50,60,777 37
Example: three smallest nums ([10 20,0,22,,00 40, 40,50,60 60, 71) 37
smallest numa (lat): def sor.
sum smallest (1st): 1st for in 1st if 0]
return sum (lat(:3))
FEEDBACK:
lst.sort()
This code finds the sum of the smallest
return sum (1at[:3))
numbers, not the smallest 3 positive numbers.
needs disregard negatives and 0.
REFINEMENT:
Figure An example of zero-shot LLM prompt for repairing incorrect code based on human-written foedback.
A.2 Code Generation Training Details
For the experiments Section 3.3 run hyperparameter sweep for all methods except for ILF. The hyper- parameter value ranges that we sweep include learning rale E (1.0-6, 5.00- 5,.0-9,1.0 1.0-5), batch size E {32, 64, 128), and number of epochs {1,2, 5}. The tasks for the training and validation datasets are from MBPPTrais and MBPP respectively, while the programs are sourced from the method (e.g. InstructGPT, MBPP, hriteewrilten, or zero-shot CoDEGEN-Mows 6.1B). For ILF, we use the best hyperparameters obtained for the sweep over MBPP programs instead of sweeping over 1LF-geneet programs, since the tasks in MBPP'Refie Refine are already used to train TRefine- All pass rates reported Table 4 are obtained by evaluating each method on MBPP Test using the best hyperparameters found during the sweep on MBPPReeine
For the experiments in Section 3.4 other experiments, train and validate using the tasks from MBPPTrain and MBPP respectively, cou- pled with the refinements generated by InstructGPT that pass the unit test suites. We sweep the same hyper- parameter value ranges as the experiments the previous section (i.e. learning rate E [1.0-6 5.0-6 1.0-5), batch size E {32, 64, 128}, and number epochs {1,2,5)).
separately tune hyperparameters for each size of dataset. As in
We implement all experimental pipelines with the HuggingFace transformers (v4.12.5) (Wolf al. 2020 Huggingface datasets (v2.7.1) (Lhoest al. 2021), and Pytorch (v1.11) (Paszke al. 2019) libraries.
30

---

Published in Transactions on Machine Learning Research (02/2024)
A.3 Code Generation Annotator Instructions
Given natural language description of Python programming challenge and its accompanying unit tests, you will be shown 10 sample model-generated Python solutions that do not pass the tests. Please do the following:
1) Select one model-generater code sample that seems relatively easy to correct (such that can be minimally corrected to pass the unit tests). no such code sample exists (ie every code sample would require extensive correction, select the corresponding option and move on to the next task.
2) Write 1-4 sentences of natural language feedback for the code sample that does two things: (a) describes what is wrong with the code sample, and (b) how it can be fixed. You can use individual variable or method names, but please do not include entire lines of code. Try to describe the necessary logic using mostly natural language, not Python expressions. Below are some examples of good versus bad feedback:
Good:
"The current code wrong because returns items using the heappop method the hq module, which gets the smallest items even though the task description actually asks for the largest items. Instead, the code should the nlargest function in hq to get the largest integers."
Bad (because it only describes what to change, and not what was originally wrong with the code):
"The code should use the nlargest function in hq to get the largest integers."
Bad (because it gives the actual code needed to fix the function, instead of describing the desired fix using words only):
"The current code is wrong because it returns items using the heappop method in the hq module, which gets the smallest items even though the task description actually asks for the largest. items. Instead, the function should return hea.poappop(mnn (nums) for in range(n))."
3) Write a refinement for the code that consists of copying the original model sample and making minimal edits to the code such that it correctly solves the challenge and can pass all the unit tests.
Here an example. We only show couple of model samples here for brevity, but practice you will be shown up to 10 model samples per task.
Example 1:
Prompt:
Write function find the similar elements from the given tuple lists.
Example:
4, 5, 6) (5, 10)) (4,
def similar, ceatllar_eeemente (teat_tupl, test_tup2)
Unit. Tests:
31

---

Published in Transactions on Machine Learning Research (02/2024)
(candidate):
(5, 7, 4, 10)) (4, 5), "candidate ((3, 4, 5, assert candidate ((3, 4, 5, 6) (5,
6) (5, 7, 4, 10)) (4, 5)"
assert candidate ((1, 2, 3, 4) (5, 4, 3, 7)) (3, 4) "candidate ((1, 2, 3, 4) (5, 4, 3, 7)) = (3, 4)
assert candidate ((11, 12, 14, 13) (17, 15, 14, 13)) - (13, 14). "candidate ((11, 12, 14, 13), (17, 15, 14, 13)) == (13, 14)
Model Sample 0:
Write function find the similar elements from the given tuple lists.
Example: similar (1e 4, 5, (5, 10)) (4,
def similar, (eellarrelementtttet_etutt test_tup2);
Similar to sets, want to get the values that are in both the given lists.
if len (test_tupl) == len Lentss__ttup))
return set (test_ttupl & set (test_tup2)
else:
raise Exception Exception("Tuple lists different lengths")
Unit test results for model sample 0: failed: candidate (3, 4, 5, (5, 7, 4, 10)) (4, 5)
Model Sample 1:
Write function find similar elements given tuple lists.
Example: similar_olements 4, 5, 6), (5, 7, 10)) (4,
def similar Sinilar_elemente eltereeeeeeesteeeeeeeeeee (test_tupi, test_tup2)
Args:
test_tup1 and test_tup2 tuo list of tuples of sane length
Returns:
tuple similar elements first second
32

---

Published in Transactions on Machine Learning Research (02/2024)
res []
for x,y in sip(test_tupl, test_tup2).
res. append (x if = else (x,y))
return tuple(res)
Unit test results for model sample 1: failed:
candidate 4, 5, 6), (5, 7, 4, 10)) == (4, 5)
Example annotator response:
1) Which model completion are you choosing to write feodback and a refinement for?
Completion 0
2) What feedback do you have for this model sample? Write 1-4 sentences of natural language feedback for the code sample that does two things: (a) describes what is wrong with the code sample, and (b) how it can be fixed. You can use individual variable or method names, but please do not include entire lines of code. Try to describe the necessary logic using mostly natural language, not Python expressions.
The logic is mostly correct, but the function returns the wrong type. It should always return a tuple, not a set or Exception. The return statement should convert the output tuple prior to returning, and the raise Exceplion stotement should relurn on emply tuple instead.
3) How can you minimally edit the code such that it correctly solves the challenge and pass all the unit tests? (Please copy over the original completion and edit it only as necessary.)
Write a function to find the similar elements from the given two tuple lists.
>>> Example: similar similar_elemente ((3, 4, 5, 6) (5. 7, 4, 10)) (4, 5)
def Beailltr_lementsseee test_tup2):
Similar want get the values that in both the given lists.
*..
len (test_tup1) (test_tup2):
return tuple(set (test_tup1) & set (test_tup2))
else:
return tuple ()
33

---

Published in Transactions on Machine Learning Research (02/2024)
A.4 Alternative Reefiee Models
We also explore prompting pre-trained LLMs such as gpt-3.5-turbo and gpt4 as Teefine instead of fine- tuning CODEGEN-MONO 6.1B model. Although such models are closed-source and require API credits access, the use of few-shot prompting removes the need to acquire separate training dataset and computa- tional resources for fine-tuning custom Refine model. We use the same prompt format as in Appendix A.1 but with two in-context examples added. The results are shown in Table 6 Although the custom-traines Refine model still yields the strongest results, both gpt-3. b-turbo and gpt-4 still yield significantly higher pass@1 and pass rates compared to both the zero-shot baseline and ablations Table 4 As expected, all the variations on ILF perform comparably (the recond-to-lass row in Table D. This demonstrates that ILF effectively improves the base model even when different kinds of Reefine model are used, and does not necessarily require extra data for training Teefine
the gold standard of training on human-written refinements
Table 6: Final performance of wg- (trained with ILF) MBPP Tests compared across multiple types of Refine models.
Method Feedback Source
Source of Refinements
Pass Rates of ITa-
Pass@1 Pass@
Humans
Fine-tuned CODEGEN-MONO 6.1B
36%
68%
ILF
Humans
2-shot gpt-3.5-tuue 5-turbo
34%
68%
Humans
2-shot gpt-4
32%
65%
A.5
Human Feedback More Informative Than InstructGPT Feedback
Table 7: The proportion of the feedback that addressed each type of bug, for feedback sourced from humans and IPT.. Each sample of feedback can be tagged with multiple categories, the quantities in each column do not necessarily add up 100%.
% of Feedback
Feedback Calegory
Human Instruct
Logic
30%
46%
Formatting
36%
14%
Missing step
10%
6%
Algebra
10%
8%
Recursion
1%
14%
Regex
6%
6%
Function semantics
2%
4%
Dynamic
program-
2%
0%
ming
Extra step
0%
12%
No feedback needed
0%
14%
Unrelated
0%
8%
To better understand why human feedback produced greater improvements in pass rate than InstructGPT feedback, we randomly selected 50 samples of feedback for each source (i.e. human InstructGPT) and annotated the number and types of bugs that each feedback sample addressed. The results are shown in Tables 7 and 8 We observed that InstructGPT often gave no feedback (e.g. "The code is correct" or "Great job!"), provided leedback that was irrelevant or incorrect, or restated the task description instead of addressing what should be repaired about the code. Despite this, InstructGPT's refinements were often correct even the feodback itself wasn't. Hommnn-rritten feedback addressed more bugs on average and did not contain irrelevant feedback. We provide further examples of the differences between human and InstructGPT feedback in Appendix A.7 A.7
34

---

Published in Transactions on Machine Learning Research (02/2024)
Table 8: Descriptive statistics for the human- versus Instruct Gerggeertte feedback. The * indicates that the metric was computed on the random sample of 50 that we manually inspected, whereas the other metries are computed from the full dataset.
Source of Feedback
Instruct-
Human
GPT
Avg. num. of bugs ad- dressed*
1.8
1.1
Avg. of words
68.9 + 48.2
24.2 + 28.6
80%
Pass rate of refinement
40%
20%
0%
3
4
Number of bugs addressed feedback
Figure 8: The number of bugs addressed in the feedback versus the pass rate of TRefine 's refinements.
A.6
TRefine Struggles To Incorporate Feedback Addressing Many Bugs
Lastly, we explored whether the number of bugs addressed in the feedback affected Teefnee 's ability to repair the original code sample. The results are shown in Figure The greater the number of bugs addressed, the lower the average pass rate of TRefine's refinements. This suggests that a promising direction for future work might consist of antomatically decomposing the feedback into multiple steps and having Teefnn incorporate the feedback one step at time. Indeed, Nijkamp al. (0022) show that the CODEGEN models often more effective at following instructions when the instructions given across multiple turns, and recent Chain-of-Thoughe work (Wei et al. 2022b) illustrates a similar prompting technique.
35

---

Published in Transactions on Machine Learning Research (02/2024)
A.7 Examples of Human Versus InstructGPT Feedback Generated Programs
Table 9:
Examples of natural language feedback written by humans versus InstructGPT for incorrect pro grams generated by CODEGEN- 61.B.
Incorrect program generated by CODEGEN- Human feedback MONO 6.1B
Instruct GPT feedback
"The only issuc with this code that does not dereference the key, When you set temp = (key, val), the key is entire tuple instead of the values in the tuple. Thus, you need the asterisks to get the values'
"The function works as in- tended."
Write function assign frequency the given tuple list.
tuple
Example: assign freq ([(6, 5, 8), (2, 7). (6. 5, 8). (6, 6, 8), (9, ) (2, 7)] (66, 5, 8, 3), (2, 7, 2), (9, 1)]
from collections import Counter def
>>> assign freq ([(6,
8), (2, 7). (6, 5, 8)
(6. 5. 8), (9. (2. 7)]
3), (2.
2). (9, 1)1*
enn
counter Counter (x
for in test_list)
res []
for key, val in counter. itens :):
temp = (key,
val)
res append
temp)
return str (res)
"Only one thing is wrong in the code that complex method can take the com- plex number as argument example complex(a+tb) not like complex(a,b) we have to replace, by and the code is fine."
"The code should use the atan2 function to get the angle of a complex nun- ber."
Write function to get the angle of a complex number.
>>> Example: angle_ comle_ (0,1 j) 6779966777963287998886
inport cmath
def angle_con complem b):
return cmath. phase (a)
36

---

Published in Transactions on Machine Learning Research (02/2024)
Table 9: Continued from previous page
Incorrect program generated by CODEGEN- MONO 6.1B
Human feedback
InstructGPT feedback
"One extra for loop for list2 is causing the issue, it finds more matching clements between 1,,, The pos- sible solution for the code is to remove the unneces- sary for loop of list2 itera- tion, and check all the lists same index element at the time, and return the count of this element."
"The code above is very inefficient. It is check- ing every possible combina- tion of pairs of elements in the three lists. Tt would be much more efficient to use
Write function to count the same pair in three given lists.
>>>
Example: count_samepar ([1
to keep track of the pairs that have already been seen.'
[2,2,3, 1,,,,,,,,,777,,,,,1,,,,,,,,,,,,, ,99 [2.1.3,1,2.6,1 93
def count count_samppair (list1, list2 :istt)):
count
range (list1))
for j range(1es (len 1ist2)):
if 1ie
[j] and [est2[]===1settti)
count+-1
return count
"The code works for tu- ples of more than one list, but not in case of one sin- gle list. Python goes one level down and reads the elements when given sin- gle list and does not see an instance of list. I would just check for the case of 'type(Input) == list' and turn the input into a 1- sized list of lists in this case."
"The function find lists needs to return the num- ber of lists present in the given tuple, not the mum- ber of items in the tuple."
Write function find number lists present in the given tuple.
>>> Example: find lists ((I1, 2, 3, 4], [5, 6, 7, 8)))
2
def find lists (Input):
result 0 for in Input: if isinstance 1ist)
result result
37

---

Published in Transactions on Machine Learning Research (02/2024)
Table 9: Continued from previous page
Incorrect program generated by CODEGEN- MONO 6.1B
Human feedback
InstructGPT feedback
"The logic behind check- ing whether the sum of rows, columns, and diago- nals is a magic number or not not correct, the code checks the sum of squares of itself in the is is_square method is wrong. stead of checking all the sums of rows, columns, and diagonals at every point, the logic should be like saving all the sums for rows, columns, and diag- onals in one array(sides) and then checking if there are all same values present in this array because for magic square test all the sums should be equal."
"The Lagic_ssqartt peet should return whether may_matri magic square."
function
Write function to calculate magic square.
>>> Example: =agic_aquara aquarquuueeeeeeeee teat (Il7, 12, 1, 14]. 12, 13, 8, 11], [16, 3, 10, 5], [9,
6, 15, =Tue
In-
def bagic_square_tes my_matrix): "*"
magic_ test ([[7,
12, 1, 14], [2, 13, 8, 111, [16, 3, 10, 61, [9, 6, 15, 4]])=True
n*e
def 1s ix_square
return x--x=x
def is it_magic_square my_matrix):
sides-[]
for i in range (len( my_matrix)):
addes..ppeeddccc my my_matrix(1)))
if not 1s_square (sun (sum
sides)):
False
sides=[]
range (len
my my_matrix)):
sides. append (sun my my_matrix [i][j] for j in range (len (ny_ (ay_matrix)) not is_square (sun (sun
sides)):
return False diagonals-[]
for 1 in range (len( my mymmatrix)):
diagonals. ( my_matrix(1](1))
if not is_square (sun diagonals)):
return False diagonals=[]
range (1en(
my my_matrix)):
aegoonals.appende
my my_matrix [ie (ny_matii))))
-1-1])
not is_square(sur diagonals)):
return False return True return is_magic_square
my_matrix)
38

---

Published in Transactions on Machine Learning Research (02/2024)
Table 9: Continued from previous page
Incorrect program generated by CODEGEN- MONO 6.1B
Human feedback
InstructGPT feedback
"a) The first thing that the code sample gets incorrect is that it increments the counter when each charac- ter the string is an up- per case character. The prompt is asking for how many characters in strl have the same numerical position as their respective alphabetic counterpart and the code does not check for this at all. b) The way to fix it would be to create separate j variable that keeps track of the ordi- nal value that corresponds with the alphabetic charac- ter that we would expect to be compared to the respec- tive character in strl. j and i would increment at the same time so that the code is both iterating through the alphabet AND iterat- ing through strl simultane- ously. The other fix would be to automatically vert strl to lowercase that we don't have to worry about checking the ordinal codes of both the lowercase and uppercase versions the same alphabetic char- acter."
*Great job!
Write python function to count characters at same position in a given string (lover and uppercase characters) as in english alphabet.
>>> Example:
count_oke char posiiton ("xboefg
") =
def count_ count_char_spa char (os__ppositionsttt))) counter
for rOnge (en
strili) [i] stri
upper ():
counter return counter
39

---

Published in Transactions on Machine Learning Research (02/2024)
B ILF for Text Summarization Prompts
B.1
Summarization Prompts
We report all prompt templates used generate INTIAL SUMMARIES, REFINEMENT WITH FREDBACK, and REFINEMENT WITHOUT FEEDBACK in Table 10
Table 10: Prompt templates used for summarization.
Methods
Format
INETIAL SUMMARY
Write an excellent, summary of the given text.
Title: {title}
Text: {text)
TL;DR:
REFINEMENT FEEDBACK
WITH
Write excellent summary that incorporates the feedback the given summary and is better than the given summary.
Title: {title}
Text: [text)
Summary: (summary]
Feedback on Summary: [feedback)
Improved TL,DR:
REFINEMENT
WITHOUT
Write an excellent summary that is better than the given summary.
FEEDBACK
Title: (title)
Text: (text)
Summary: {summary]
Improved TL:DR:
40

---

Published in Transactions on Machine Learning Research (02/2024)
B.2 InstructRM Prompts
We instructed one of the authors of this paper (who at the time had not been involved in the research project) to write 5 prompts that would achieve the goal of selecting high-quality summaries, i.e., refinements. The author did not have any domain knowledge or prior information on what kinds of prompts would work. The instructions provided to the author can be viewed here We report all 5 prompt templates in Table 11
Table 11: Prompt templates used for InstructRM Ensemble.
InstructRM Prompts
Format
PROMPT 1
Here's summary of a Reddit post, feedback on the summary, and a new summary. You will be asked to determine whether the new summary incorporates the feedback provided.
A good summary is a short piece of text that has the essence of the original text. A good summary tries to accomplish the same purpose and conveys the same information as the original text.
Post title: (title}
Below, there's the content of the post that was sum- marized.
Original post: (text} }
Original summary: {summary)
hurnan then provided feedback on the above sum- mary.
Feedback: {feedback)
Based on this feedback, summary was written.
New summary: (refinement)
Does this summary incorporate the feedback provided? Answer Yes or No.
Answer:
PROMPT 2
Post title: (title)
41

---

Published in Transactions on Machine Learning Research (02/2024)
Original post: {text) }
Original summary: (summary) }
Feedback: (feedback)
New summary: (refinement)
Question:
Does the new summary incorporate the feedback provided? Answer Yes or No.
Answer:
PROMPT 3
You will be given a Reddit post title, its content, an original summary of that post, and feedback for that summary. Then, your goal will be to determine whether the new sunmary improves upon the origi- nal with respect provided feedback.
Post title: (title)
Post content: (text)
Original summary: (summary)
Feedback: {feedback)
New summary: (refinement)
Question: Docs the new summary incorporate the leedback provided? Answer True or False.
Answer:
PROMPT 4
Here's summary of a Reddit post, foedback on the summary, and new summary. You will be asked to determine whether the new summary incorporates the feedback provided.
42

---

Published in Transactions on Machine Learning Research (02/2024)
A good summary is short piece of text that has the essence of the original text. A good summary tries to accomplish the same purpose and conveys the same information as the original text. Remember, you will be asked to determine whether the new summary incorporates the leedback provided.
Post title: (title)
Below, there's the content of the post that was sum- marized.
Original Post: (text)
Remember, you will be asked to determine whether the new summary incorporates the feedback pro- vided. Here's the original summary.
Original summary: (summary)
Remember, you will be asked to determine whether the new summary incorporates the feedback pro- vided. A human then provided feedback on the above summary.
Feedback: {feedback)
Based on this feedback, summary was written.
New summary: (refinement)
Does this new summary incorporate the feedback provided? Answer Yes or No.
Answer:
PROMPT 5
Here's a summary of a Reddit post, feedback on the summary, and new summary. You will be asked to determine whether the new summary incorporates the feedback provided.
The feedback was:
Feedback: feodback
43

---

Published in Transactions on Machine Learning Research (02/2024)
Here's the post that was summarized the first place.
Post title: (title)
Original Post: (text}
Remember, you will be asked determine whether the new summary incorporates the feedback pro- vided. Here's the original summary.
Original summary: (summary)
Remember, you will be asked to determine whether the new summary incorporates the feedback pro- vided. A human then provided feedback on the above summary. Here's the feedback again.
Feedback: (feedback)
Based on this fcodback, new summary was written.
New summary: (refinement)
Does this new summary incorporate the feedback provided? Answer True False.
Answer:
B.3 Fine-tuning Prompts
In Table 12 we report the prompts we use for fine-tuning on summaries and fine-tuning on feedback + refinements. The completion for fine-tuning on summaries indicates that we can have completions generated from various sources, i.., either initial summaries from FeedME, refinements generated with our method, ideal human written summaries. For fine-tuning feedback + refinements, we first generate the feedback and then the refinement.
Table 12: Prompt templates used for Fine-tuning on Summaries and Feedback + Refinement.
Methods
Prompt
Completion
44

---

Published in Transactions on Machine Learning Research (02/2024)
FINE-TUNING ON SUMMARIES
Write an excellent summary of the given text.
(surmaryyreatinnene///mna summary?
Title: (title)
Text: {post)
TL;DR:
[eedbbckk
FINE-TUNING ON
FEEDBACK
Write an excellent summary that incorpo- rates the foedback on the given summary and is better than the given summary.
Improved TL;DR: {ref inmment)
+ REFINEMENTS
# ##
Title: (title)
Text: [post}
Summary: (summary)
Feedback summary:
B.4
Reward Model Prompts
Table 13:
Prompt templates used for training the reward model with the language model loss. Both classi- fication and comparison prompts are shown.
Reward
Model
Prompt
Complction
Type
BINARY RM
Title: {title}
{" Yes"/" No"}
Text: (post)
TL:DR: (summary. A/summary_B
Question: Is the above an excellent sum- mary of the given text? An excellent. sum- mary is coherent, accurate, concise, and de- tailed. Answer with Yes or No.
45

---

Published in Transactions on Machine Learning Research (02/2024)
Answer:
COMPARISON RM
Title: {title}
/" B"]
Text: {post)
Summary A: {summary_A)
Summary B: (summary_
Which summary the better one? An excellent summary coherent, ac- curate, concise, and detailed. Answer with A B.
Question:
Answer:
B.5 Summarization Ranking Procedure Details
We standard ranking scheme where each of K summaries is given rank between and K (inclusive). Sometimes refinements are exact copies of the initial summaries is why we allow for summaries to be tied. When calclating the win rate we assign 0.5 wins for tied samples. We assign the rank r' to all summaries ranked in tie, where r' r+(r+n-1), is the rank of the tied elements, and n is the number of ties at the rank. For example, we map ranking of (1,2, 4, + (1, 2.5, 2.5, 4, 5) and a ranking of (1, 2,,,3,3,3) + (1, 2,4,
very similar in terms of quality, which
B.6 Dataset Collection and Analysis
Annotation process To ensure the high quality of our human annotations, we employ experienced anno- tators sourced through the data-labeling company Surge AI. During an ouboarding and evaluation process, calculate author-amooatoo agreement the binary comparison task and manually review the quality of the written feedback and ideal summaries to ensure their high quality. Then we select. 31 qualified an- notators for all annotation tasks, though they can choose which tasks to participate in and for how long, To further ensure the quality of our annotations, we provide detailed instructions, which we provide to the annotators, and update throughout the process to ensure continuous improvement (these instructions can be found in Appendix Cj. To measure the agreement rate between the annotators and the authors, we select a sample of 10 Reddit posts from the training dataset a gold standard and have 17 annotators label them. When comparing the binary comparison annotations with agreement rate of 81.0%. We also calculate the average agreement rate between all the possible annotalor combinations, yielding an annotator-anottaee agrecment of 70%. By utilizing these thorough processos and evaluations, we can ensure the accuracy and reliability of our human annotations.
ones, this results in an author-annotator
Dataset Analysis The feedback we collect typically addresses the most critical shortcomings of the sum- maries. In 92.0% of our train samples, the annolators' feedback was complete and addressed all important shortcomings of the summary, as reported by the annotators. Across our train dataset, we observe that the majority of the feodback pertains to coverage (77.0%), with smaller percentages relating to accuracy (16.0%), coherence (5.0%), and other categories (2.0%). We also analyze the length of the various sunmaries and feedback, measured in the average number of tokens. Our human-writted summaries have average length
46

---

Published in Transactions on Machine Learning Research (02/2024)
of 0.1 tokens, the extracted human summaries from Reddit had an average length of 32.5 tokens, the initial summaries generated by FeedME have an average length of 29.3 + 0.1 tokens, and the feedback written by annotators on these initial summaries has an average length of 20.4 + 0.2 tokens.
In addition to these analyses, we also measure the time it takes annotators to complete various tasks (i.e., binary comparison, feedback writing, and ideal summary writing) on our development dataset. We ignore outliers and consider only samples with annotation times of least 20 seconds and at most 420 seconds (7 minutes). Annotators take 61.5 5.3 seconds on average the binary comparison task, 182.5 6.3 seconds the feedback task, and 195.5 = 6.1 seconds on the ideal summary task. We plot the annotation times on the development dataset for the tasks of annotating binary comparisons, writing feedback, and writing ideal summaries as histograms in Fig. g The annotators are much faster at annotating binary comparisons than feedback or ideal summaries. Writing feedback takes less time than writing ideal summaries, which is expected, as critiquing a task usually casier than solving it. These comprehensive evaluations demonstrate the high quality and thoroughness of our dataset and annotation processes.
Foodback Armotation 192.50
30-
|M3356 $1.50
12
Human Meaiin 135.50
25
Bin.o.m
20
10
200
330
400
100
(a) Binary comparison lask.
(b) Feedback annotation Lask.
(c) Summary writing lask.
Figure 9: Histogram Plot of annotation times (in seconds) of the (9a) binary comparison task, (9b) feedback
annotation task, and (9c) the human summary writing task. The evaluation is conducted the development dataset. We observe that annotators are much quicker at the binary comparison task, which is expected. The results also show that writing feedback takes less time than writing an ideal summary.
C Annotator Instructions
Overall we completed many annotations to create datasets and evaluate our algorithm. The instructions were task-specific and also contimuously updated. In the following, we provide the instructions we used to create our train dataset and the instructions we provided for evaluating the summary quality (of 6 summaries). We will not share more instructions for brevity but can provide them upon request.
C.1
Train Dataset Annotation Instructions
Task Overview
You are given a Reddit Post, which you first need to read carefully. You then need to complete 5 subtasks which consist of comparing two summaries, writing feedback on a summary, classifying the type of feedback, indicating whether there is additional Feedback, and writing an ideal summary. When doing these tasks, please adhere to the guidelines below.
What makes for good summary? Roughly speaking, a good summary is short piece of text that has the essence of the original text. A good summary tries to accomplish the same purpose and conveys the same information as the original text. We would like you to consider these different dimensions of summaries:
Essence: Is the summary a good representation of the post? How well does the summary cover the important information in the post?
Clarity: Is the summary reader-friendly Does it express ideas clcarly?
Accuracy: Does the summary contain the same information the post?
47

---

Published in Transactions on Machine Learning Research (02/2024)
Purpose: Does the summary serve the same purpose as the original post?
Concise: Is the summary short and to the point?
Style: Is the summary written in the same style as the original post?
Generally speaking, we give higher weight to the dimensions the top of the list. The evaluation can be complicated though, since none of the above dimensions are simple yes/no matters, and there aren't hard and fast rules for trading off different dimensions. Use your best judgment and common sense to make these trade-offs. In case the subreddit, title, and Reddit post leave open some ambiguity about what happened, it is important to accurately reflect that in your annotations and not just interpret the text in a certain way. Always look at all the subreddit, title, and Reddit Post and use all information given to make your judgments (sometimes the title may contain crucial information that does not appear in the post but should nevertheless be used).
First, read the Subreddit calegory, title, and post carefully. A Subreddit is forum dedicated to a specific topic the website Reddit. Take your time with this step and re-read the parts that you might not have understood at first. Below is detailed description of the task you will need to complete for each Reddit post.
Below is a detailed description of each task you will need to complete for each Reddit post:
Comparison Task: Given a pair of summaries, indicate which is better.
Delails: Use the above description of what makes a good summary. It is alright to choose either summary both summaries are identical copies of each other or if there is no distinguishing feature that makes one summary superior to the other. However, if there is a small detail that makes one summary better than the other, that is enough reason to select that summary.
2. Feedback Task: Write short and simple feedback the given summary about the single, most important shortcoming of the summary. The feedback should NOT mention what category (Aceu- racy, Coverage, Coherence, other) the feedback belongs to, nor should it assume knowledge about the definitions of "Coverage", "Accuracy", or "Coherence" (see below). Otherwise, the feedback should be as short and simple as possible while still addressing the most important shortcoming of the summary.
Delavls: You can write the feedback in one or several sentences, but it should only address the single, most important shortcoming of the summary and be as short as possible. There are no other restrictions as to how you write the feedback and what exactly it addresses. If there are no shortcomings in the summary, the feedback can also mention a positive thing about the summary. Use the description of what makes a good summary to trade off the various dimensions that make for a good summary. Often the feedback will (but does not have to) address one of the following axes.
Coverage: For this axis, answer the question, "how well does the summary cover the important information in the post?" A summary has good coverage if it mentions the main information from the post that's important to understand the situation described in the post. A summary has poor coverage if someone reading only the summary would miss several important pieces of information about the situation in the post. A summary with good coverage should also match the purpose of the original post (e.g., to ask for advice).
Accuracy: For this axis, answer the question, "does the factual information in the summary accurately match the post?" A summary is accurate if it docsn't 't say things that aren't in the article, doesn't mix up people, and is generally not misleading. If the summary says anything at all that not mentioned in the post or contradicts something the post, is NOT accurate.
Coherence: For this axis, answer the question, "how coherent is the summary on its own?" A summary is coherent if, when read by itself, it's easy to understand and free of English errors. A summary is not coherent it's difficult understand what the summary trying to say. Generally, it's more important that the summary understandable than being free of grammar errors.
48

---

Published in Transactions on Machine Learning Research (02/2024)
The feedback should NOT mention what category (Accuracy, Coverage, Co- herence, other) the leedback belongs to, nor should it assume knowledge about the definitions of "Coverage", "Accuracy", "Coherence", or "other" (as defined above). Example: One should NOT write 'This is missing in the area coverage', or "This summary lacks the category of accuracy, because ... The leedback should be understandable to person who has never read the definition of "Coverage", 'Accuracy', and 'Coherence'. You are, however, ALLOWED to use those words if they make sense on their own, e.g,, you CAN say, "This summary does not cover the important parts of the text because', "This summary is inaccurate as it states ...,, , or 'This is not coherent summary because...
Additional Rules:
3. Feedback Type Task: If your feedback falls into the categories Accuracy-related, Coherence- related, or roletege-related, mark it as such by checking the corresponding checkbox for the (single) category is related to. If your feedback not related to any of these three categories, then check the "Other' checkbox.
More Feedback Task: Answer with Yes if there is additional Feedback about an important short- coming of the summary that you would want to mention and No otherwise.
5. Ideal Summary Task: Ideal Summary Task: Write short summary for the Reddit post that is ideal in your view.
The ideal summary should be ideal in terms of all the criteria mentioned above, ie., essence, clarity, accuracy, coverage, purpose, concisoness, coherence, and style. In other words, you should not be able to find an obvious critique of the ideal summary that you write. It is okay to reuse parts of previous summaries but only if those parts should be part of an ideal summary. The ideal summary should maximally be 48 tokens long (otherwise, you can't submit your annotation). Tokens are generated by taking your ideal summary and splitting up certain words into individual pieces (this necessary to train our AT). The interface will show you how many tokens your ideal summary has already taken up.
Details:
C.2 Summary Quality Evaluation Instructions
Task Overview
You will be given Subreddit category, title, and Reddit Post, which you first need to read carefully. Your lask is then to compare 6 summaries and rank them according to quality.
What makes for good summary? Roughly speaking, a good summary is short piece of text that has the essence the original text. A good summary trics to accomplish the same purpose and conveys the same information as the original lext. We would like you to consider these different dimensions of summaries:
Essence: Is the summary good representation of the post? How well does the summary cover the important information in the post?
Clarity: Is the summary reader-friendly? Does it express ideas clearly?
Accuracy: Does the summary contain the same information as the post?
Purpose: Does the summary serve the same purpose as the original post?
Concise: Is the summary short and to the point?
Style: Is the summary written in the same style as the original post?
Generally speaking, we give higher weight to the dimensions at the top of the list. The evaluation can be complicated though, since none of the above dimensions are simple yes/no mallers, and there aren't hard and fast rules for trading off different dimensions. Use your best judgment and common sense to make these tradc-offs. In case the subreddit, title, and Reddit post leave open some ambiguity about what happened, it is important to accurately reflect that in your annotations and not just interpret the text in a certain way. Always look at all the subreddit, title, and Reddit Post and use all information given to make your
49

---

Published in Transactions on Machine Learning Research (02/2024)
judgments (sometimes the title may contain crucial information that does not appear in the post but should nevertheless be used).
First, read the Subreddit category, title, and post carefully. A Subreddit is a forum dedicated to a specific topic on the website Reddit. Take your time with this step and re-read the parts that you might not have understood at first. Below is a detailed description of the task you will need to complete for each Reddit post.
Given 6 summaries, indicate which is better by ranking them according to quality. Rank 1 is considered the highest rank, and Rank 6 is considered the lowest rank. The summary with the best quality should be ranked highest, ie., as Rank 1, and the summary with the worst quality should be ranked lowest, i.e. Rank 6. Use the above description of what makes a good summary. Ties between summaries are allowed, but only if summaries are exact copies of each other or if there is no distinguishing feature that makes one summary superior to the other. However, there small detail that makes one summary better than the other, that is enough reason to rank that summary better than the other summary. We use Standard Competition ranking (i.e., example rankings of 122456). In standard competition ranking, items that compare equally receive the same ranking number, and then a gap is left in the ranking numbers. The number of ranking numbers that are left out in this gap is one less than the number of items that are compared equally. Equivalently, each item's 's ranking number is 1 plus the number of items ranked above it.
Comparison Task:
C.3
Reward Model
Here we describe the various RMs that we evaluate in more detail. We evaluate the final RM that we use, which produces a language output (e.g., " Yes" or " No") and a standard reward model that produces a scalar output.
Standard RM. Akin to Stiennon et al. 22020), we remove the last embedding layer of a language model and train it to output a scalar value. This scalar value predicts which summary, E {ao } is better as judged by a human, given a context c. We use the OPT 13B LM, introduced in (Zhang et al. 2022; ; as the base model for our RM and fine-tune it on the human preference comparisons that we collected. is worth noting that it not possible to add linear layers on top of GPT-3 models provided via the API, which is why we use the OPT model.
Reward Model with Language Output. In addition to the classic RM (Stiennon et al. 2020 train an R.M to output language tokens instead of a scalar value. To do so, we finc-tune an LM to classify whether a summary vn is high quality or not, by training it to predict a label E {Mgood: Dbad): }- We then fine-tune the LM to maximize A Alogp(zo) + log p(y|zo), where X e [0, 1], chosen using the development dataset. The complete loss can also be written as:
|2|
l81
C(pg, x, y) = - Σ
log po (2))))))
log (Ve x Vee).
where the subscript t indicates the token index. We evaluate the fine-tuned LM on given summary xo by computing P(Ugood(zo) The best RM overall uses the following instruction Is the aboue an excellent summary of the given text? An excellent summary is coherent, accurate, concise, and detailed. Answer with Yes or No., which we refer to as the OPT-RM (when fine-tuning OPT-13B) and GPT-3 Binary (when fine-tuning GPT--3175B). We also explore fine-tuning on another prompt, where we provide both summaries A and B to the LM and instruct it to indicate which summary is preferred, i.c. Question: Which summory the betler one? An excellend summary is coherent, accurale, concise, and delailed. Answer with A or B. We then fine-tune the LM on the label of the preferred summary (according to binary human feedback), i.e. y E (MA, )n). We evaluate the fine-tuned LM on a given summary In by computing p(yal (zo). We refer to this RM as Comparison RM. We explore two RMs, namely, OPT-13B Zhang et al. (2022), and GPT-3-175B and refer to Appendix C.4 for the hyperparameters we use and to Appendix B.4 for the prompt templates).
Results We evaluate all RMs on our validation dataset, and calculate the accuracy of predicting the preferred summary out of two, based on human preferences. Table 15 the complete results, and here
50

---

Published in Transactions on Machine Learning Research (02/2024)
report on some of the RMs trained on 5K samples. The OPT model with the standard RM loss achieves accuracy of 71.8 + 2.0% on the validation dataset. The results further show that both of our methods for training OPT with the LM loss outperform the standard RM loss, with OPT comparison achieving an accuracy of 72.6 = 1.9%, and OPT-RM an accuracy of 73.4 + 1.9%. We obtain similar results with fine- tuning GPT-3-175B, achieving an accuracy of 71.2 + 2.0% with the GPT3 Comparison, and an accuracy of 74.2 2.0% with GPT-3 Binary, which outperforms the OPT-RM.
Based on these results, we further evaluate the OPT Binary and GPT-3-175B Binary models the devel- opment dataset that
to evaluate the scoring functions in $4. We calculate the fraction of times the refinement selected by an RM is better than a randomly-selected refinement ("win rate"), according to a ranking given by human evaluators (see App. B.5 for more details). The results can be found in Table [4] OPT-RM achieves a win rate of 63.3 + 2.7%, and the GPT-3-175B Binary model achieved a win rate of 61.8 + 2.9%. In this evaluation, OPT-RM outperforms GPT-3 Binary. When considering the results from both the validation and development datascts, both OPT-RM and GPT-3-Binary seem to perform similarly, Given that we have more control over the training process of OPT, the possibility of releasing the model, and the cost involved in training using OpenAl's API, select OPT-RM model as our reward model for comparison with ILF. In Figure 10] we show the validation accuracy of OPT-RM trained on 100, 1K, and 5K samples on a log-log plot. The figure shows scaling when increasing the dataset size.
We further evaluate results for fine tun-tuning OPT-RM on the dataset of Stiennon et al. (2020), and also eval- uating their model with 1.3B parameters on our dataset. We observe that the binary preference distribution of the training dataset has a significant impact on the performance of the reward model. For example, OPT-RM trained on 5K samples of our own train dataset (i.c., our final reward model) achieves an accuracy of #1.9 0.2% on the test set from Stiennon et al. (2020) (not shown in Table 15) When this same model is trained on 90K samples from the train dataset of Stiennon et al. (2020), it achieves an accuracy of 0.23% on their test set (also not shown in Table 15) In contrast, this same model trained on 90K samples from their train datasct achieves an accuracy of only 68.6 + 2.0% on our validation dataset, which is significantly lower than the accuracy of 1.99% achieved by the model trained on 5K samples of our own train dataset. Similar patterns can be observed when comparing the OPT Binary model with 1.3B parameters trained on 5K samples of our own train dataset to the released 1.3B reward model trained by Stiennon et al. (0000) approx. 64K samples of their own train dataset. The former model achieves accuracy of 69.6 = 2.0% our validation dataset, while the latter only achieves an accuracy of 63.8 + 2.1% (note, though, that the RMs are trained with different loss functions). These results highlight two important considerations: (1) preference distributions can vary significantly and have a strong effect on what a reward model learns, and (2) the sample elliciency of a reward model depends heavily on the train and test distributions. If the test distribution differs from the train distribution, reward models may be very sample inefficient and fail accurately learn the true distribution, even when given significantly more samples.
Table 14: In human evaluation, we compare reward models and ranking methods on the development dataset (in the same way as in Fig 5 Both RMs are trained on 5K samples and outperform the zero-shot methods.
Scoring Function
Win Rate vs Random Selection (in %)
Task Specific Heuristic
Max Length
65.0 + 2.7
Embedding Similarity
48.3 + 3.0
Zero-Shot
InstructRM Ensemble
56.0 + 3.0
OPT Binary
63.3 + 2.7
Fine-tuning on 5K samples
GPT-3 Binary
61.8 + 2.9
51

---

Published in Transactions on Machine Learning Research (02/2024)
Table 15: In a human evaluation, we evaluate various RMs on the development dataset and validation dataset. We also report the results of training on the train dataset of Stiennon al. (2020) and evaluating on our development and validation datasets. We calculate the accuracy of predicting which of two summaries is preferred by human.
Models
No. Params
Training
Dev. Acc. (%)
Val. Acc. (%)
Dataset Size
OPT
13B
5K
66.5 = 3.3
72.6 + 1.9
Comparison
OPT RM
1.3B
5K
70.0 3.2
69.6+2.0
LM Loss / Our dataset
OPT RM
13B
100
54.5 - 3.5
53.4 2.2
OPT RM
13B
1K
68.5 3.2
2.1
OPT RM
13B
5K
69.5 + 3.2
73.4 + 1.9
GPT-3
5K
68.0
2.0
Comparison
GPT-3
5K
74.2 + 2.0
Binary
RM Loss / Our dataset
OPT
13B
5K
68.5 3.2
71.8 + 2.0
RM Loss
Stiennon et al.
Stiennon
1.3B
64K
58.0 + 3.4
63.8 + 2.1
2020
train dataset
et al. 2020
RM
LM Loss / Stiennon et al.
OPT Binary
13B 90K
69.0 + 3.2
68.6 + 2.0
2020
train dataset
C.4 ILF for Text Summarization Hyperparameters
C.4.1
Generating Refinements
For all summarization experiments sample up to 48 tokens (as in Stiennon et al. 2020 with nucleus sampling Holtzman et al. (2019) with 0.95 and temperature characters (c.g., newlines) from the beginning of sampled summaries. We further remove empty white spaces in the generated summaries and remove all text that comes after a new line token / n. Due to the maximum token length, sampled summaries sometimes end with incomplete sentences. Thus, we remove ending sentences that do not end in an up" or "?". The described temperature and post-proccssing are applied all summary generations, ie., for generating initial summaries, refinements, and test summaries.
1.0. We strip non-alphamumeric
C.5 Fine-tuning on Summaries
We conduct independent hyperparameter optimization sweeps with three dataset sizes of human summaries of 100, 1K and 5K samples, and then use the same hyperparameters for fine-tuning refinements (ILF) and fine-tuning since this will not give an unfair advantage to our algorithm that fine-tunes on refinements. For the sweep, we utilize the train dataset of human summaries (consisting of 100, 1K, and 5K samples) and evaluate on the development dataset. Unfortunately, the OpenAl API only provides validation loss and token accuracy for batches of the development datasct, making impossible evaluate the model on the full development dataset during training. As result, we utilize the model API to evaluate on the full development dataset after fine-tuning and calculate the perplexity of the generated summaries performance measure.
initial summaries. We choose to the hyperparameter sweep Human summaries
52

---

Published in Transactions on Machine Learning Research (02/2024)
Table 16: We report the chosen hyperparameters of fune-tuning on 100, 1K, and 5K samples of HUMAN SUMMARIES. A symbol indicates that this hyperparameter is optimal but used only for fine-tuning on HUMAN SUMMARIES. For fine-tume fine-tniig on REFINEMENTS and INITIAL SUMMARIES we inadvertently use the prompt loss weight 0.
Samples
Epochs
Prompt Loss Weight
Learning Rate
100
1
0.05
1K
1
0.05*
0.02
5K
1
0.1
0.2
To determinc the optimal hyperparamcters, we perform sweep over a range of values for the following parameters: epochs {1, 2, 4), prompl loss weight {0, 0.01, 0.05, 0.1}, and learning rales {0.02, 0.05, 0.1, 0.2). We first sweep over epochs and select the best value, then perform sweep using that value for the prompt loss weight, and so on. Our empirical observations indicate that the number of epochs has the greatest impact on perplexity, with training for more than one epoch resulting in overfitting. The selected hyperparameters can be found in Table 16
During the fine-tuning phase for the REFINEMENTS and INITIAL SUMMARIES datasets with 1K samples each, we made an error in our hyperparameter selection. Instead of using a prompt loss weight of 0.05, we mistakenly used value of 0, when fine-tuning on human summaries. While this error may have slightly impacted our results, the difference in perplexity between the two settings is minimal, with a value of 6.68 for a prompt loss weight of 0.05 and 6.71 for prompt loss weight of O. Despite this mistake, our method still outperforms fine-tuning on human summaries for 1K samples, as well as fine-tuning on initial summaries using suboptimal hyperparameters.
C.6 Multiple Iterations of ILF for Text Summarization
To evaluate multiple iterations of ILF, i.c., multiple iterations of reining-and--ine-tuunng we fine-tune GPT- 3-175B on refinement dataset with 200 and 300 samples. Thus conduct hyperparameterr optimization on a train dataset of 200 and 300 refinements and evaluate on a development dataset of 200 refinements (instead of human summaries). To determine the optimal hyperparameters, we perform a sweep over range of values for the following parameters: epochs 21,2,344 2,3,4),, 4), prompt loss weight {0, 0.01, 0.05, 0.1), and learning rales (0.02, 0.05, 0.2). We first sweep over epochs and select the best value, then perform sweep using that value for the prompt loss weight, and so on. For fine- aue-tuning on 200 refinements we select the following hyperparameters: epochs 1, prompt loss weight 0.05, lcarning rate multiplicr 0.1. For finc-tuning 300 refinements select epochs 1, prompt loss weight 0, and learning rate multiplier 0.2. The results are reported in Section C.8.2
C.7 Fine- tine-tnning Reward Models
OPT Reward Model. For fine-tuning the OPT Reward Model, we perform bayesian hyperparameter
optimization for each of the three different types of reward models: Standord, Comparison and Classéfication (see section C.3 We sweep over the learning rate in the range of [le 1e 16 and the batch size {32, 64) for all the models. For the reward models using the language loss, also optimize the prompt-loss weight f0.0, 0.01, 0.05, 0.1, 0.5, 1.0). We run 10 iterations per model and evaluate all the sweeps with the 200 development examples. We use a linear learning rate scheduler and a weight decay of 0.1 for all the runs. The optimal batch size is 32 for all the models. The best prompt loss weight is 0.01 for both the Comparison and Classification RMs. As for the learning rate, we use 9.3c-6 for the Standard RM, 5. for the Classification RM and 1e-6 for the Comporison RM. In the final fine-tuning, we select the best RM in the validation split over 10 epochs.
53

---

Published in Transactions on Machine Learning Research (02/2024)
OPT-RM, LM Loss/Binry
70
Accuracy %)
60
50
100
1000
5000
Train Dataset Size
The validation accuracy versus training dataset size of OPT-RM trained on 100, 1K, and 5K samples on a log-log plot.
Figure 10:
GPT-3 Reward Model. In order to fine-tune GPT-3-175B as an RM, we utilize the OpenAl API. We fine- tune two types of RMs: the Comparison RM, which learns to predict which of two summaries superior, and the Classification RM, which predicts whether a given summary is of high quality or not. For cost considerations, we conduct hyperparameter tuning on a training dataset of 1K samples (instead of 5K) and evaluate on development dataset of 200 samples. We use dataset with 1K samples for cost reasons. We then apply the same hyperparameters when fine-tuning terms of epochs. Due to the binary nature of the human preference annotations in the classification reward model, the effective train dataset size for this model is doubled to 2K samples.
5K samples while implementing carly stopping in
In order to determine the optimal hyperparameters, we perform sweep over a range of values for the number of epochs 21,2,3,4) and the prompt loss weights {0, 0.001, 0.005, 0.01, 0.05, 0.1, 0.5). The OpenAT APT provides classification accuracy (for both the comparison and classification tasks) for the full development dataset after each epoch, allowing 5K samples, utilize early stopping to prevent overfitting, using epoch and prompt loss weight of ( for the comparison model and 4 epochs and a prompt loss weight of 0.001 for the classification model. We use default values for all other hyperparamcters, which may vary depending on the dataset size.
select the appropriate number of epochs and prompt loss weight.
When fine-tuning
C.8
Additional Text Summarization Results
C.8.1 Results: ILF + OPT-RM
In this section, we present the full results of our best-performing method ILF + OPT-RM and other additional methods (see $4.4.1 for a description of ILF + OPT-RM and $4.4.3 for a discussion the results). We conduct the same evaluation as described test summaries based on quality. We then calculate the win rate against human written summaries, which use evaluation metric. Importantly, all methods evaluated here trained datasets with 5K samples. Note that the methods compared here not exactly the same the methods compared in Fig. G Concretely, the test summarics generated by the methods fine-tuning on refinements (ILF), fine-tuning on human summaries, and OPT-RM best-of-64 FeedME are the same as in Fig, D for the test. summaries generated by corresponding methods trained on 5K samples. Here, however, we don't evaluate FeedME and fino-tuning on initial summarics. However, we evaluate ILF + OPT-RM (best-of-64), our best-performing model, which we also added Fig. 0 for reference. We also evaluate a new method called Fine-tuned on Feedback Refinements, which we describe below.
$4.4.2 i.c. a human evaluation, annotators rank various
54

---

Published in Transactions on Machine Learning Research (02/2024)
Fine-tuned on Human Summaries
Fine-tuned on Refinements
Fine-tuned on Iuitial Summaries
1.5 x 10
x 10
Negative Log-Likelihoon Liikel
x.3 10
1.2 x 10
1.1 100
10
9 10-1
102
10a
Train Dataset Size
Figure 11: Evaluation of models fine-tuned on 5K initial summaries, refinements, and human summaries on 500 samples from the corresponding validation datasets. For example, the model fine-tuned on human summarics is evaluated on 500 human summarics from the validation dataset. The model fine-tuned on refinements has a significantly lower negative log-likelihood than the model fine-tuned on human summaries.
For fine-tuning on feedback + refinements, we use title, post, and summary input and the model is trained to predict the corresponding feedback and refinement. Our motivation for this approach is that generating feedback first may improve the quality of the resulting refinements, similar to the findings of previous work on self-prompting methods Sethhos(Suunders Saunders et al. 2022 : Bai et al. (022b)) and the Chain of Thought (CoT) prompting technique Wei et al. (2022b). CoT has been shown to improve the performance of models across various tasks Wei al. (2022b) when allowing the model to reason before answering a question. For fine- fnn-tuning on feedback and refinements, we utilize the initial summaries that were used to gather human feedback, as well the refinements generated by our method. We use the loss log p(11, prompp)) + Å Algg p(prompt))) i.c. we learn to predict the refinement and the feedback. We employ the same hyperparameters refinements algorithm (including the prompt loss weight). During testing, we require initial summaries, from which we generate feedback and refinements. As initial summaries, we use the test samples generated by FeodME (as evaluated in Figure 6). To ensure compatibility with the 48-token length restriction of the test summaries, we append the special end loken /n n ## # to the end of the feedback and refinements during training. At test time, we the maximum number of tokens to generate 300, and terminate generation when the stop-word / ### appears. We then apply the same postprocessing procedure outlined in Appendix C.4.1 to shorten the refinements to 48 tokens. We refer to Appendix B.3 for the exact prompt templates we used.
in the fine-tuning
We present all the results in Fig. 12 We find that fine-tuning on a set of 5K refinements achieves a win rate of 36.0 1.8%, while ILF + OP RRM (best-of-64) has a win rate of #19% achieving human-level summarization performance (sce $4.4.3 for a more detailed discussion). OPT-rM best-of-64 FeedMe achieves win rate of 45.1 + 1.9%, fine-tuning on a set of 5K lumaan-enerratd summaries achieves a win rate of 35.4 1.8%, and fine-tur fin-tuning It is worth noting that the performance of fine-tuning fine-tuning on refinements alonc. We attribute this to the increased difficulty of generating both feedback and refinements and believe that this discrepancy may be due to limitations in our models, dataset size, or hyperparameters. Previous work has demonstrated the feasibility of training models to generate feedback Saunders et al. (2022): ; Bai al. (2022b), so we believe that further optimization and experimentation may improve the performance of this method. We further want to note that the results for fino-tuning on 5K refinements, 5K human summaries, and best-of-64 FeedME. deviate from the results in Fig 6 This is because
combination of 5K feedback and refinements has win rate of 26.1 1.7%.
feedback and refinements is lower than that of
55

---

Published in Transactions on Machine Learning Research (02/2024)
compare different methods with each other, and human annotations generally contain some amount of noise (given that different, people annotate the same samples).
50
Human Summaries (%)
40
Win Rate vs.
30
20
10
Finetuned on Reffnements
OPT-RM (best-of-64)
best-af-64 FoodMe
Iummn
Summaries
Feedback
Refinements
How often human evaluators prefer summaries from ILF: Fine-tuned on Refinements, OPT- RM best-of-64 FeedME, ILF + OPT-RM (best-of-64), fine-tuning on human summaries, and fine-tuning on feedback refinements (all methods fine-tuned on 5K samples). ILF + OPT-RM (best-of-64) generates summaries of a similar quality human summaries. Fine-tuning on feedback refinements performs worse than fine-tuning on refinements (ILF).
Figure 12:
C.8.2 Multiple Iterations of ILF
Our experiments suggest that ILF is an effective method for leveraging language feedback in the training of LMs. Here we explore ILF its most general form by doing multiple iterations of
Dataset Improvement. In this experiment, we evaluate the effectiveness of iterative refinement of the dataset distribution using ILF. To this end, we first fine-tune GPT-3-175B on 100 refinements from iteration of ILF (i.e. doing one iteration of refining initial summaries, we did in the main results of our paper, see $14.2) and refer to this tnnettued model as M, The notation use here is that the subscript indicates the iteration of ILF that the refinements were generated in, and the superscript indicates the number of overall samples the model is fine-tuned on. We also refer to the dataset of 100 refinements from iteration 1 as 11000 As a baseline, we fine-tune M on an additional 100 refinements from ILF iteration 1, resulting in M i.e., a model trained on 200 refinements from ILF iteration We then compare this baseline to iterations of ILF. Specifically, we use M| to generate summaries for an additional 100 samples (the same Reddit posts for the baseline) and collect human feedback those summaries. We then this feedback to generate 5 refinements using the FeodME and then select the best refinement using InstructRM method. We refer to these 100 selected refinements from the second ileration of ILF as Di00, Finally, we fine-tune 10000 on D100 to obtain the model M which has been trained on a total of 200 refinements generated in both the first and second itcrations ILF. All fine-tuning was performed using the same hyperparameters as described in Appendix C.4 for fine-tuning on 100 refinements, We refer to Table 17 for an overview of all models and train datasets.
In this human ovaluation, compare the performance of the summarics generated by the bascline model (M1300) with those generated by two iterations of ILF (M232) indicate their preferred summary for each comparison, and the win rate of M200 against. .2200 is calculated and plotted in Fig. 13 (left) Our results show that two iterations of ILF outperform one iteration with a win rate of 53.2 + 1.9% indicating that applying multiple rounds of ILF can improve the data distribution. However, we also want. to investigate whether multiple rounds of IL. lead to better models than directly
test set. Human evaluators asked to
Idcally, would the model M0 to generate the refinements. However, in case, this is not possible since fine-tuned GPT-3-1P2 175B, which 11 Note, the win manually 50% at samples, since the baseline is equivalent to one iteration ILF.
instruction-fe fine- tuned model.
56

---

Published in Transactions on Machine Learning Research (02/2024)
Table 17: Datasets (refinements) over which the models M are trained, and which they generate. The superseript indicates the number of samples, whereas the subscript indicates the ILF step. In this figure we do not show Fe which is used to generate the refinements given feedback. these samples are new samples from the interval [100, of D700
Fine-tuning Dataset
Initial Model
Fine-tuned Model
Produces Datasct
ILF Iter. 1
ILF Iter. 2
ILF Iter. 3
GPT-3-175B
M
D100
D
M200
D,000
GPT-3-175BB
M M20
scratch,1
D200
GPT-3-175B
M300
D4000
scratch,1
M
M,.00
D100
D
D
M
M 1,2.3
1100
D200
Dj00
GPT-3-175B
M 200
1100 + D
s.ratch,1.2
GPT-3-175B
M
11000 + D200 + D
scratch,,1,233
fino-tuning on the same number of refinements from the first round from scratch. In other words, while our current baseline consists of further fine-tuning Mf on an additional 100 samples, it is also possible to directly fine-tune GPT-3-175B on 200 refinements from the first iteration of TLF from scratch, i.e. M200 Mcratek.. We aim to determine the relative effectiveness of these two approaches in improving model performance on the text summarization task.
Model Improvement In this experiment, we aim to compare the performance of multiple rounds of ILF to directly fine-tuning on comparable number of refinements from the first iteration of ILF. As a baseline, we fine-tune GPT-3-175B on 200 and 300 refinements from the first iteration of ILF and conduct hyperparancter tuning as described in the Appendix C.4 We then compare these basclines to two and three rounds of ILF. For the two-round ILF model, we use the previously described M To obtain the three-round ILF model, we use M to generate summaries for an additional 100 samples (on the same Reddit posts as for the bascline), gather human foedback, generate refinements with GPT-3-175B using the feedback, and select the best refinement using InstructRM, resulting in D We then fine-tune M2200 on Dj00 to obtain the model M Tt is important to note that while our baselines fine-tune GPT-3-175B from scratch on 200 and 300 refinements, the models M and M are obtained by continuously fine-tuning model iteratively on additional refinements. This difference in approach may introduce a discrepancy in the results, as we use different hyperparameters, and the dataset size may affect the learning dynamics. To control for this potential difference, we also fine-tune GPT-3-175B from scratch on the refinements generated through various iterations of ILF. Specifically, as an alternative to M we fine-tune GPT-3-175B from scratch on a concatenation of 100 refinements from the first round of ILF (i.e., D100) and 100 refinements from the second round of II.F (Le.,, D200), and refer to the resulting model as M200 scratch1,2- Similarly, as alternative to M1.233 we fine-tune GPT-3-175B from scratch on concalenation of 100 refinements from the first round of ILF (DI"0), 100 refinements from the second round of ILF D and refinements from the third round of ILF (i.e. DJ and refer to the resulting model as M330 Mcrrtck112233 It is worth noting that the refinements from the second and third rounds of ILF (i.e. D and D2 are based on summaries generated using models that were continuously finc-tuned (i.c. MM000 and M2.00) As such, the models 2200 M000 M
scratch1,2 and scratch1,2.33 are not a direct application of ILF,, but rather an approximation of the distribution induced by ILF. We refer to Table 17 for an overview of all models and train datasets.
Using a human evaluation, we compare the performance of the three methods on the test dataset: the baseline, ILF with continuous fine-tuning, and ILF approximated by fine-tuning from scratch. The results
57

---

Published in Transactions on Machine Learning Research (02/2024)
ILF (M122)
ILF (continuous fine-tuning) (M1 /M
Approximating ILF (M /M acratchh1
55
Finetuning on Refinements
54
Finetuning on Refinements
Win rate vs.
54
53
Win rate vs.
52
52
51
48
100
200
Train Dataset Size
100
200 Train Dataset Size
300
Figure 13: Left: Win rate of 2 iterations of ILF against fine-tuning on the same number refinements from the first iteration ILF. Right: Win rate of iterations of ILF, and approximating iterations of ILF by fine-tuning from scratch, against fine-tuning the same number of refinements from the first iteration of ILF.
are shown in Fig. 13 (right). With this more realistic baseline, find that directly applying ILF does not improve upon the basclines, with win rates of 49.4 + 1.9% and 50.9 + 1.9% for 200 and 300 samples, respectively. However, approximating ILF by fine-tuning from scratch on the distributions induced by ILF significantly improves upon the baseline for 300 samples, with a win rate of 55.6 + 1.9%. The method is slightly worse than the baseline for 200 samples, with a win rate of 48.9 + 1.9%. We currently hypothesize that continuous fine-ture -ee-tttinn may lead catastrophic forgetting, while fine-tuning from scratch may not have this problem. This could explain why M300
scratch1,2,3 performs significantly better than M300 for 300 samples. Specifically, M may actually generate an improved distribution in the third iteration of ILF. However, when further fine-tuning M22 on this improved distribution D1000 the model may forget what learned previously. On the other hand, the model M300 all datasets produced by ILF may actually benefit from the improved dataset distribution because it does not unlearn anything. It is, however, unclear why M200 seratch1,2 does not benefit from the improved dala distribution D1000 It also possible that the hyperparameters play a significant role in the final performance of the various models and that the dataset size has strong influence model performance (e.g., fine-tuning more samples may be more stable than fine-tuning on fewer samples). In future work, plan to conduct more elaborate experiments to answer these questions and better understand the effects of the dataset size and number of iterations on ILF. Specifically, we aim to run multiple iterations of ILF and use 2000 as the model to generate summaries in the third round of ILF (instead of M7.00). This would be a direct implementation of ILF, rather than an approximation of it, as we would be fine-tuning the same model with which we are also generating an improved distribution. We also hope to investigate the effect of the dataset size and number of iterations on ILF. Overall, our results suggest that ILF has the potential to improve the performance of natural language processing systems by continuously incorporating human foodback into the training of language models, but further research needed to fully understand the best ways to leverage this approach.
scratch1,,23 that learns from scratch the concatenation of
scratch1,2
C.8.3 Part-of-Speech Distribution for Fine-tuning Datasets
We evaluate the negative log-likelihood of GPT-3-175B on the three fine-tuning datasets, i.e. on initial summaries, refinements, and human summaries. We use the training dataset with 1K samples and calculate the negative log-likelihood over different Part-of-Speoch tags. We use Stanza Qi et al. (2020) as the PoS tagger for this experiment and then we separate the words into three groups: function words, content words, and others. The function words are words that have little lexical meaning: articles, pronouns, adpositions,
58

---

Published in Transactions on Machine Learning Research (02/2024)
conjunctions, auxiliary verbs, particles and interjections. On the other hand, content words are words that contain semantic information: nouns, adjectives, adverbs and lexical verbs. We keep numbers and symbols under the group others. With this analysis, we want to spot different patterns between model-generated (initial summarics and refinements) and Haman-written summarics. Note that a high negative log-likclihood implies high loss. We present the results Fig [4] Since the average loss is higher for human summaries, we normalize all the loss values by transforming them have mean and standard deviation 1. Overall, the word distribution is very similar for all three fine-tuning datasets. In terms of normalized mean loss, it is interesting how the content words have bigger infiuence the refinements dataset. We believe that this is related to our results in section 4.4.3 where we obtain the best results when fine-tuning on refinements.
C.8.4 Comparison to Results of Scheurer et al. (2022)
Here we relate our results to our previous non-archival work, Scheurer et al. (2022). In Fig. 2 of Scchurer et al. [2022), they compare their method of fine-tuning on refinements against various baselines, such as fine- tuning on initial summaries, sampling from FeedME (called InstructGPT), and sampling from GPT-33175B.
They calculate the win rate of all methods against human written summaries (Völske et al. 2017) that are automatically extracted from Reddit. As shown in $4.2 and App our human summaries are preferred 72.3 + 3.2% to the human summaries of Völske et al. (2017). This implies that the win rates in Scheurer et al. 2022 are much higher than in our case since we use much stronger baseline.
We now present three differences between the results found in Scheurer et al. (2022) and the results found in our paper. Then we will provide various potential reasons that could explain the differences. First, when comparing the results (in relative terms) in Scheurer et al. (2022) Fig. 2 to our results in Fig. g where we fine-tune on 100 samples, we see differences in performance. Scheurer al. (022)) reports that fine-tuning on refinements outperforms fine-tuning sampling from FeedME (i.e., Inst InstructGPT). In our experiments fine-tuning on 100 refinements achieves a win rate of 19.6 + 1.5% against human summaries, fine-tuning on initial summaries a win rate of 19.6 and FeodME a win rate of 20.8 + 1.5%. Thus both fine-tuned mothods perform equally and are worse than sampling from FeedME.
initial summaries. And both methods outperform
Second, we compare the results of refining a summary with feedback. Note that Scheurer et al. 2022 uses an embedding-basee scoring function to select refinements, whereas we use InstructRM. In Scheurer el al.) (2022) Fig. 3 (left) REFINE WITH FEEDBACK + BEST OF N achieves win rate of 67.0 = 3.1% against initial summarics (sampled from FeedME), REFINE WITH FEEDBACK achieves a win rate of 60.5 + 3.0%, REFINE WITHOUT FEEDBACK achieves 50.3 + 2.6% and Human Summaries have win rate of 60.8 3.4. In our Fig. 15 (left) Refine with Feedback + Best-of-5 achieves a win rate of 69.1 + 1.9%, Refine with Feedback achieves a win rate of 63.9 + 2.0%, Refinement without Feedback achieves a win rate of 59.4 + 2.0% and Human Summarics a win rate of 83.2 + 1.7%. The difference in the human summarics is expected, given that use better human summaries. The Refinement without Feedback method achieves higher results in our work than in Scheurer et al. (022)).
Third, it is also noteworthy that using the embedding similarity as scoring function worked well in Scheurer et al. 2022), while it does not work in our setting (see Table 5 and 44... for a discussion of the results). We believe this is because the foodback we collect is written by many annotators and is thus much more diverse, while in Scheurer et al. (2022), the authors themselves wrote the feedback.
Here we now list various differences in the setup of Scheurer et al. (2022) and our paper, which could all account for the different results.
1. Scheurer et al. (2022)
embedding similarity a scoring function, while we use InstructRM
Ensemble. Looking at Tab. 5 and the corresponding discussion in $4.3.1 already shows that the methods are very different.
The weittten-rittenneee summaries are of much higher quality in our paper than in Scheurer al. (2022) (see 44.2 and App. B.6
59

---

Published in Transactions on Machine Learning Research (02/2024)
Word distribution
Normalized mean loss
Intal Bummary
Refinamnt
Human Burmary
Intal Surmary
Refirement
Human Sunmary
1.5
1.0
0.0
-05
Fundion
Dhees
Fundian
Figure 14: Distribution of tokens of various fine-tuning datasets with 1K samples in terms of content, and function words. We only evaluate the various completions, i.e, summaries, since the prompts are the same for all distributions.
In Scheurer al. (2022), the annotation instructions specifically state that the feedback should mention how to improve a summary. In our work, we collect much more unrestricted and diverse feedback. This difference is also apparent in the fact that the embedding similarity does not work well as scoring function in our setting.
In Scheurer et al. 2022), the authors themselves annotated the data, i.e., they wrote the feedback and evaluated the final summaries. In our case, we use independent evaluators who are trained on this task. Using 31 annotators overall also gives us a more diverse and less biased estimate of our methods. Also, doing human evaluations is inherently noisy and will never lead to the exact results.
The evalnation in Scheurer et al. 2022 was done on a different dataset than this work. Specifically, they used only 100 samples to evaluate their method, while we use a test set of 698 samples.
6. The hyperparrameters in Scheurer et al. (022) used for sampling and fine-tuning different from the hyperparameters used in our work.
7. Overall, we use different prompts than Scheurer el. al. 2022 (see App.
B.3 and App.
B.1
C.8.5 Does Language Feedback Improve Refinements of Summarizations?
The improvements from ILF suggest that the refinements used for fine-tuning are high-quality, so here we investigate whether language feedback is responsible for the high quality. To do so, we have human evaluators rank Refinement with Feedback + Best of N summaries against summaries from several other methods, similar to $4.3 We use the human ranking to compute a win rate between each method and the initial summary. We compare against Refinement with Feedback, which randomly chooses refinement E =t =,. 25 This ablation helps evaluate the importance of choosing a refinement with our scoring function R, i.e., InstructRM Fasemble. We also evaluate Refinement without Feedback, which instructs the LM to refine the initial summary but without feedback. This ablation helps to evaluate the importance of using language feedback. Lastly, we evaluate Human Summaries and Initial Summaries i.c., the initial summary To generated by FeedME. We evaluate all methods on the validation dataset.
Results Fig. 15 (left) shows the win rates of summaries from various methods against initial sunmaries.
Surprisingly, instructing a model to improve its output without feedback already leads to a significant improvement (win rate of 59.4 + 2.1% over the initial summarics). Refinements with Feedback achieve an improved win rate of 63.9 2.0% showing that language feedback is useful for improving refinement quality. Refinement with Feedback Best of N achieves
better win rate of 69.1 + 1.9%, highlighting that
60

---

Published in Transactions on Machine Learning Research (02/2024)
Refine w/
Foedback +
Refine
Best-of-5
Feedback
Best-of-5
Refine w/
Refine w/ Feedback
Feedback
Refinement
Refinement
(w/o
(w/o
FeedbacK)
Feedback)
Human
Human
Summaries
Summartes
60
65
70
75
80
85
% summaries incorporating most important feedback point
Win Rate vs.
Initial Summaries (%)
Figure 15: Left: Human evaluators prefer summaries from all refinement methods the initial summaries (FeodME). Refine with Feedback best-of-5 is rated highest. Right: Refine with Feedback + best-of-5 generally does incorporate the most important feedback point.
Best-of-N with the InstructRM Ensemble further improves the refinements. Overall, language feedback is important for high-quality refinements, especially when using Best-of-N sampling.
C.9 Do Refinements of Summarizations Incorporate the Feedback?
To determine whether refinements are of higher quality due to incorporating feedback rather than improving the summary in other ways, we conduct a study on the validation dataset in which crowd workers evaluate how often the most important point of the foedback is incorporated methods. As shown in Fig. 15 right, our method Refinement with Feedback Best of N incorporates the most important point in the feedback most frequently (57.4 = 2.2% often). Refinement with Feedback incorporates foedback 49.6 + 2.2% of the time, showing that Best-of-N sampling improves how often the feedback is incorporated. For reference, Refinement without Feedback fixes the most important point in the leedback 30.8 = 2.1% of the time, despite the model not receiving the language feedback. Human Summaries address the most important point in the feedback 74.0 1.9% of the time when writing the summary from scratch despite not receiving the foedback explicitly. Our results suggest that refinements are high-quality in part because they incorporate the most important point in the feedback.
the refinements produced by various
61

---

Published in Transactions on Machine Learning Research (02/2024)
D Broader Impacts
Our work, like many other human foedback-based machine learning works, relies upon hand-labelled human annotations. Since these annotations are often provided by small groups of expert annotators, the resulting model may exhibit social [Liagg et al., 2023: Srivastava et al., 2023). political (Hartmann et al. 2023 Perez
et al. 2023 or moral biases (Simmons 2023 learned from the human annotations. Past work has also shown that such human (cedback-basod models may have difficulty modcling the full spectrum and diversity of human opinions (Santurkar al. 2023), and may even rely upon modeling the most productive annotators rather than modcling the task itself (Geva et al., 2019). Continued training multiple rounds of feedback may also result in further amplification of biases present in the dataset (Taori & Hashimoto 2023). As such, the deployment of feedback- based models warrants careful attention to the diversity, biases, and steerability of the model.
62


Reviews:
[{"review_id": "70fd75lEqM", "reviewer": "Reviewer_EjWp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths of the paper\n---\n* The paper contributes a novel finetuning technique that takes advantage of language feedback and demonstrates it on two diverse tasks.\n* I liked the ablation to identify whether the feedback is actually used by the refinement model and discussion on difficulty of fixing multiple bugs pointed out in the feedback.\n* The human evaluation of quality of text summarization is valuable and the authors have indicated that they will open source their code and data (including human feedback).\n\nWeaknesses of the paper\n---\nExperimentation\n* Code generation: Can $\\pi_\\theta$ and $\\pi_{\\text{Refine}}$ be different code generation? The datasets are small and it should be possible to use an off-the-shelf LLM capable of repair as $\\pi_{\\text{Refine}}$. This would substantiate the claim that \"the approach ... appealing ... not model-specific\".\n* Text summarization: The authors use GPT3-175B as the base generative model and finetune it. Surprisingly, they do not compare the finetuned model with the base (unfinetuned) GPT3-175B model in Figure 6; they use FeedMe instead. The athours should include zero-shot and OPT-RM setups with GPT3-175B.\n* The output sequences for both tasks are short (MBPP are small programs and text summarization is designed to be short). The paper does not offer any evidence that ILF can work for longer output sequences.\n* ILF is presented as an iterative approach but the tasks don't use multiple iterations. The authors should either show utility of multiple iterations or mark it explicitly as future work. The claim \"the approach ... multiple rounds to continuously improve ...\" is an over-statement in the absence of experiments.\n\nPresentation\n* Both algorithms have mistakes: Algorithm 1, subscript k, line 2. Algorithm 2, line 11, are you proposing to create a separate $\\pi_{\\theta}^*$ in each iteration (then you need to select among the k-copies produced in k-iterations) or is the LHS a typo (meaning, you continue to finetune $\\pi_\\theta$ across iterations)?\n* Insights and limitations: The paper offers interesting results but lacks insightful discussion about why the technique works. The limitations of the approach (e.g., only two tasks with small output length) etc. are not discussed adequately. The authors have sprinkled some insights (e.g., the last para in Sec 3.3 and conclusions section) and limitations here and there. I recommend that the authors create a new section to discuss some insights (if possible backed with some eval) and limitations.\n* There are typos (e.g., extra \")\" and missing Table ref in Sec 4.3) and citations appear without \\citep in conclusions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a novel two-step procedure (imitation learning from language feedback, ILF) to incorporate language feedback to improve the base generative model. The feedback is first used by a refinement model whose outputs are ranked and used for finetuning the base model. ILF is instantiated on two problems: code generation and text summarization.\n\nFor code generation, CodeGen-6B model is used as the base model for both refinement and generation. The paper shows that finetuning with model-generated refinements outperforms/matches finetuning with InstructGPT generated refinements, gold answer and human written refinements on pass@k metrics. \n\nFor text summarization, GPT3-175B model is the base model and the FeedME model is the refinement model. This setup does better than FeedME, and finetuning with initial and human summaries, as judged by human evaluators. Up on using a further ranking method OPT-RM with 64 samples, ILF achieves parity with human summaries.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the weaknesses listed above with suggested changes.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "KsVH9nw5O1", "reviewer": "Reviewer_MVCT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## What I like about this paper\n\n- Well written and structured paper with many experiments that serve to demonstrate the validity of their approach. \n- I appreciate the effort the authors have taken into collecting language feedback data for the two tasks and their commitment to open source it.\n- The formalization of the approach and its connection to minimizing the KL divergence to the target distribution.\n- My main take away is in Section 3.3, which shows that ILF performs better for the code generation task than fine-tuning directly on gold data or even human-writte programs. I believe this is an important result that can help understanding how to better self-improve LLMs in the future.\n\n## Concerns\n- I was expecting to see a note about how ILF is more data efficient than RLHF approach since $\\pi_\\mathrm{refine}$ can leverage pretrained language model as opposed to learning a user preference models from scratch for RLHF. I'm curious to know if the authors have any intuition on that.\n\n- In the ablation experiment to test whether CodeGen-Mono 6.1B is good at incorporating feedback (which is needed for $\\pi_\\mathrm{refine}(x_1|t, x_0, f)$), the authors compared providing relevant feedback vs. unrelated feedback. I am concerned with that ablation. I could imagine the model to be performing worse because it is being distracted by the unrelated feedback, thus actually incorporating it. I think a better baseline would be to provide no feedback (or almost no feedback), e.g. \"Fix the code\". I don't expect the outcome to change (i.e., model + relevant feedback should be significantly better than no feedback) but that seems a more principle ablation to me.\n  - If I understood correctly, the authors seem to have done it for the text summarization task (see Fig. 15 in Appendix C.9). Why not for the code generation task? Was there a particular reason for ablating with unrelated feedback?\n\n- In section 3.3, one experiment is ablating the human annotations by using an LLM to generate feedback and refinements instead (i.e., the one the authors call InstructGPT). What was the prompt template use for this?\n\n- In Table 4, what can explain models fine-tuned on InstructGPT 1/2-shot refinements performing worse than the zero-shot baseline? I would expect it to be at least as good as the zero-shot baseline. Is it also because of the point mentioned in \"Analysis of Training Data Sources\" where InstructGPT's refinements would be too much out-of-distribution compared to the data CodeGen-MONO 6.1B was trained on? If so, any ideas on how to mitigate that?\n\n- In Section 4.1, according to equation (12), the reward function $R$ conditions on the context $c$, however I don't see it provided in the binary question used to approximate $R$: \"Does this new text [$x_1$] incorporate the feedback [$f$] provided on the initial text [$x_0$]? Answer Yes or No.\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper is about improving Large Language Models (LLMs) by leveraging natural language feedback that describes what is wrong with the model's output and how to fix it. The authors called this approach Imitation learning from Language Feedback (ILF). Specifically, the authors start by doing inference with $\\pi_\\theta$, i.e. the LLM they wish to improve on a given dataset. Next, they ask humans to annotate each output for which the model got it wrong according to some task-dependent metric. The annotation consists of 1) a natural language description of what is wrong with the output and 2) how it can be improved, and also 3) the expected improved output. With that annotated dataset in hand, they fine-tune a separate language model, $\\pi_\\mathrm{refine}$ such that it generate refinements given an incorrect output and natural language feedback. Equip with $\\pi_{\\mathrm{refine}}$, the authors can now generate refinements and use them as targets to fine-tune $\\pi_\\theta$. Experiments were conducted on two tasks: code generation and text summarization. Empirically, it was shown that this approach improves the LLM on those tasks compared to fine-tuning it directly on the human-written refinements.\n\nWhile a lot of work has been done in the space of improving LLMs, this paper presents a novel way of leveraging natural language feedback to do so. To my knowledge, I haven't seen other work that does this and certainly as thorough. The paper is well written and structured, and all claims are backed by empirical evidence. I look forward to see more work in this direction, especially with more ILF iterations.\n\nI recommend this paper for acceptance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "While addressing the questions and concerns listed above would strengthen the work in my view, they are not critical for my recommendation for acceptance.\n\n## Typos\n- p.2: Missing point between \"... see Section 3.3)\" and \"For text summarization, ...\"\n- p.16, Section 4.4.3. Missing Table numbers in the first paragraph.", "broader_impact_concerns": "I think a Broader Impact Statement can added to discuss the following points:\n- The authors mention that the feedback is collected from crowd workers. What are the potential biases that could arise from that.\n- At several occasion in the paper (e.g., Table 4), it is mentioned that language model could generate feedback and refinements (i.e. self-evaluate). What are the implication/bias of that?", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qIqVqvVRqa", "reviewer": "Reviewer_zchu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The method itself is interesting: natural language feedback is very high bandwidth compared to binary preferences or Likert scales.  I also appreciate that human evaluation is used for judging summaries.\n\nHowever, there are three significant weaknesses:\n\n1. The results are only slightly better than the baselines, but are presented to seem like major baseline improvements.  For example, the abstract says \"For code generation, ILF improves a Codegen-Mono 6.1B model’s pass@1 rate by 38% relative (and 10% absolute) on the Mostly Basic Python Problems (MBPP) benchmark, outperforming both fine-tuning on MBPP and fine-tuning on repaired programs written by humans.\"  The 10% number refers to an absolute improvement from 26% to 36%, but this is comparing their tuning method against against a zero-shot baseline.  It is true that the method also improves against fine-tuning on repaired programs written by humans, but fine-tuning on repaired programs (which is also data used by their method) gets 33% (and is tied at 68% with their method for pass@10).  Reading either the abstract or skimming Table 4 might miss the size of the improvement: the abstract doesn't mention it, and Table 4 explicitly calls \"zero-shot\" the baseline rather than normal fine-tuning.\n\n2. The results don't demonstrate a human feedback bandwidth improvement.  Indeed, the baselines they compare against use strictly *less* human data, since fine-tuning on refinements alone doesn't need the feedback data.  E.g., section 3.1 notes that \"ILF assumes the availability of feedback but not necessarily of the repaired code/refinements, for a variety of reasons\", but then notes that they train $\\pi_{\\textrm{Refine}}$ using human-written refinements.  Thus, the key code result is that using ILF over fine-tuning on demonstrations improves from 33%/68% (pass@1/pass@10) to 36%/68% using strictly more data.\n\n3. The mathematical discussion in section 2 is interesting, but is not very related to the method.  In particular, section 2.2 says they use equation (5) as an approximation, but the method does not actually use equation (5).  Importance sampling as shown in equation (5) requires *dividing* by the proposal distribution density $q_t(x_1)$, but the construction of $q_t$ is sampling only with intractable densities since it's a multi-stage sampling process.  Thus, it would be intractable to divide by $q_t(x_1)$ even if that were desired.  As far as I can tell this issue is not mentioned again in the paper, so the division is silently dropped.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a method for learning from human feedback which uses human-written natural language feedback to improve models, and applies it to code generation (Mostly Basic Python Problems) and summarization (Reddit TL;DR).  The method starts with both natural language feedback and natural language refinements based on that feedback, trains a refinement generator based on that data, then trains a policy on the refinements.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The major changes are for (1) and (3) above:\n\n## Fair presentation of results\n\nThe results should be presented more fairly, using correct baselines, easier to understand numbers, and a few table fixes.\n\n1. **Correct baselines:** The correct baseline for ILF on code seems to be \"gold standards, human refinements\", since this is the simple thing to do with the demonstrations gathered and used by ILF.  Thus, the abstract comparison should to this method, rather than using zero-shot as the baseline.\n2. **Easy to understand numbers:** The abstract claims \"38% relative (and 10% absolute)\" improvement on MBPP.  Ignoring that this is to zero shot, it is better presented as an improvement \"from 26% to 36% pass@1 rate\".  \"relative\" and \"absolute\" are confusing without knowing the raw numbers, even if it possible to back out the raw numbers.\n3. **Table fixes:** Several tables are misleading.  Table 3 is particularly unfortunate, since (1) it shows 0% as the baseline without noting that the $\\textrm{MBPP}_{\\textrm{Train}}$ dataset was explicitly constructed to have 0% by removing examples where the baseline passed and (2) is quoted in the text as being \"on the evaluation dataset\".  My guess is that the text is simply incorrect, as it would be surprising if the baseline got 0% on the evaluation dataset.  But (1) is also bad: captions should be designed to not mislead a reader who skims the tables, and this one definitely would.  Table 5 is another example: the caption explicitly says \"InstructRM Ensemble performs best\" and that row is bolded, but two other rows are better: the naive MaxLength baseline is way better, and Prompt 2 is slightly better.  It might still be reasonable to use the ensemble, but it's wrong to say it performs the best.  This is a nonexhaustive list of table problems, so a good through all of them to make them clearer is important.\n\n## Mathematical details\n\nThe big one is the importance sampling.  It's bad to say the method is importance sampling when it isn't, because it doesn't divide by the proposal density.  Instead, you should just say that the method approximates $\\pi^*(\\cdot | t)$ with $q_t$, then uses $q_t$ as a policy improvement operator.  As is, the method is presented as mathematically motivated in a way that it isn't, and the gap between the mathematical presentation and the algorithm is not discussed.\n\nThere are a variety of smaller notational issues:\n\n1. The paper says \"forming dataset $\\mathcal{D} = \\\\{t\\\\}$\".  This is bad notation, as it looks like a singleton set.\n2. It's bad notation to sample $t \\sim p(t)$: it should be $t \\sim p$ as $t$ isn't known before sampling itself.\n3. The $\\operatorname{FINETUNE}(\\pi_\\theta, \\mathcal{D})$ is confusing.  The notation for $\\mathcal{D}$ makes it seem like it contains task inputs but not outputs, since section 2.2 clearly thinks of $t$ is being the input.  But you can't fine-tune until you have the outputs.\n4. Eqns. (6) and (7) are harder to read than just writing out the multistep sampling algorithm, and somewhat misleading since only the sampling algorithm is used, not the density (because the density is intractable).  It would be better to just write the few-line algorithm showing each sampling step.\n5. The delta functions used are Kronecker delta functions, not Dirac delta functions.\n\n## Small notes\n\n1. Figure 1 was unclear when I first saw it, as I wasn't sure what the type signature of each component was.  E.g., I was initially expecting there to be a model that generated feedback, but it was missing, and I wasn't sure what the type signature of $\\pi_{\\textrm{Refine}}$ was.\n2. The method shows the model one of the unit tests.  Does it use the rest of the unit tests for rejection sampling?  It's likely I just missed this, but it seems important to state explicitly so that other compares compare to this one accurately.", "broader_impact_concerns": "I don't think a broader impact statement is necessary.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes Imitation Learning from Language Feedback (ILF) which involves enhancing a language model (LLM). First, they performing inference on a dataset and identify incorrect outputs based on a task-specific metric. Then, humans annotate the outputs with a description of the error, suggestions for improvement, and the expected correct output. This annotated dataset is used to fine-tune a separate model to generate refinements for incorrect outputs given feedback. These refinements are then used as targets for further fine-tuning the original LLM.

All reviewers agree that this paper makes interesting contributions, was very thorough in the experimental framework. One of the main contributions found by a reviewer is that ILF performs better for the code generation task than fine-tuning directly on gold data or even human-written programs. A particular interesting ablation was to identify whether the feedback was actually used by the refinement model and an analysis on the difficulty of fixing multiple bugs pointed out in the feedback. One of the weaknesses pointed out by a reviewer is that the length of the outputs in the tasks considered is rather small, this particular point might be highlighted in the camera ready version.

Overall this is a strong and sound paper. The release of the human feedback dataset also makes a strong contribution to the field.
Decision: Accept (poster)