RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputs
Year: 2023

Abstract:
Despite their unprecedented success, even the largest language models make mistakes. Similar to how humans learn and improve using feedback, previous work proposed providing language models with natural language feedback to guide them in repairing their outputs. Because human-generated critiques are expensive to obtain, researchers have devised learned critique generators in lieu of human critics while assuming one can train downstream models to utilize generated feedback. However, this approach does not apply to black-box or limited access models such as ChatGPT, as they cannot be fine-tuned. Moreover, in the era of large general-purpose language agents, fine-tuning is neither computationally nor spatially efficient as it results in multiple copies of the network. In this work, we introduce RL4F (Reinforcement Learning for Feedback), a multi-agent collaborative framework where the critique generator is trained to maximize end-task performance of GPT-3, a fixed model more than 200 times its size. RL4F produces critiques that help GPT-3 revise its outputs. We study three datasets for action planning, summarization and alphabetization and show relative improvements up to 10% in multiple text similarity metrics over other learned, retrievalaugmented or prompting-based critique generators.1

heading: 1 Introduction
text: Correcting model outputs is a pressing challenge in natural language generation (Ribeiro et al., 2018; Reid and Neubig, 2022), emerging across many use-cases such as style transfer (Mallinson et al.,
1A significant portion of this work was done while Feyza was an intern at Allen AI. Correspondences to Afra Feyza Akyürek (akyurek@bu.edu).
2020; Malmi et al., 2022), grammatical (Lichtarge et al., 2019) or factual error correction (Mitchell et al., 2022b), debiasing and detoxification (Schick et al., 2021). Unlike humans who can understand natural language feedback and improve using the information, most of the previous work relied on sequence tagging (Reid and Neubig, 2022), retraining from scratch (Sun et al., 2019) or parameter editing (Mitchell et al., 2022a) to repair model predictions.
7716
Recently, researchers show that large language models can correct their answer given more sophisticated feedback formulated in natural language (Schick et al., 2022; Saunders et al., 2022). For example, in Fig. 1, we present sample feedback for two tasks. Both of these examples exemplify the case where the initial model outputs ŷ have flaws. In topic-based summarization, an automatically generated summary of a story involves factually incorrect statements such as “... he was betrayed by his father Bill ...” where an appropriate critique is “Bill is not Michael’s father”. In action planning, given a goal x, the objective is to generate a set of steps y to achieve the goal. The initial sequence of actions in Fig. 1, denoted by ŷ, has a missing a step. The human-written natural language critiques c describe the ways in which ŷ’s are incorrect and ŷnew denotes the corrected prediction conditioned on the critique. Note that in many situations helpful critiques do not necessarily reproduce an entire answer—they may simply point out one way in which the answer could be improved.
Researchers use crowd-sourcing to collect critiques for model outputs (Saunders et al., 2022). However, collecting feedback from humans is infeasible in an online setting where a model is required to produce a rapid stream of outputs. The goal of this paper is to shed light on whether the task of critiquing language model predictions can be effectively passed on to an external agent while keeping the language model itself intact.
Our multi-agent collaborative framework involves two language models where one model’s job is to criticize the other as the latter performs a task of interest, such as summarization. This setting comprises a task model, denoted by LMtask, which learns the mapping from an input x (e.g. passage) to a ground truth output y (e.g. summary); and a critiquing model LMcritique which provides natural language critiques for LMtask’s outputs ŷ ∼ LMtask(x). The framework can additionally involve a separate model (say LMrefine) for repairing model outputs conditioned on critiques. We follow past work (Schick et al., 2022), and merge LMtask and LMrefine into a single model. Hence, in addition to predicting y given x, LMtask is also tasked to improve its initial output conditioned on a critique ĉ sampled from LMcritique(x, ŷ).
We introduce RL4F (Reinforcement Learning for Feedback Generation), a cascade (Dohan et al., 2022) of two language models for automatic cri-
tique generation and refinement. RL4F is trained to maximize target task performance of LMtask by learning to provide critiques for its outputs via LMcritique. RL4F advances retrieval-based methods with learned critique generation (Madaan et al., 2022). Unlike previous work which teaches LMtask to read a crowd-sourced set of critiques (Schick et al., 2022; Saunders et al., 2022), RL4F learns the particular set of critiques that will steer LMtask into improving its predictions without requiring any updates to LMtask parameters. Treating LMtask as fixed is especially important in era of limitedaccess large language models which are costly, if not impossible, to fine-tune.
RL4F is illustrated in Fig. 2(a,c). Previous work demonstrate that language models smaller than roughly 50 billion parameters lack the ability to understand and act upon a natural language critique (Saunders et al., 2022; Bai et al., 2022). Therefore, we chose GPT-3 as the LMtask model which is a clear example of an inaccessible LM that shows this ability. While RL4F is general enough to accommodate an ensemble of feedback generators, in this work we focus one single model as LMcritique for simplicity.
In summary, this work2:
• Presents a reinforced critique generator which advances simple supervision in improving the end-task performance without retraining the downstream model.
• Demonstrates effectiveness of RL4F on three tasks: topic-based summarization, action planning and alphabetization (e.g. sorting a list of words alphabetically) with relative improvements up to 10%.
• Showcases that RL4F exhibits promising scaling properties and remains to be useful when applied iteratively.

heading: 2 Related Works
text: Past works differ to a large extent with respect to what they call human feedback and how they make use of it. In this section, after elucidating the use of human feedback in previous works, we briefly describe connections of RL4F to the parameterefficient fine-tuning and discrete prompt learning literature.
2Code for the experiments is released under https:// github.com/feyzaakyurek/rl4f.

heading: 2.1 What kind of feedback is used and where does it originate?
text: Human feedback on model predictions come in different flavors. The most notable ones include (1) binary feedback, e.g. thumbs up/down and pairwise comparisons (Ouyang et al., 2022; Bai et al., 2022; Gao et al., 2022), (2) natural language critiques (Tandon et al., 2022; Schick et al., 2022; Madaan et al., 2022; Saunders et al., 2022; Murty et al., 2022; Chen et al., 2023; Madaan et al., 2023) and (3) direct textual refinements to outcomes (Scheurer et al., 2022; Shi et al., 2022).
Bai et al. (2022) introduce what they call Reinforcement Learning from AI Feedback (RLAIF) in which they replace human preference labels with those of the model’s itself; the model is prompted to evaluate its own predictions in consideration of human values and preferences. In a similar vein, Gao et al. (2022) use accuracy for extractive question answering as a reward signal when fine-tuning their policy model.
In another thread,Schick et al. (2022) use comments from forums and Wikipedia edit histories as natural language feedback. Scheurer et al. (2022) and Shi et al. (2022) collect human natural language critiques and associated refinements. They then fine-tune the task model on the refinements. Our work is similar to these works in that we also use human-generated critiques in the first stage of our algorithm. Aside from human-written critiques, we additionally use synthetically generated critiques in the absence of the former.

heading: 2.2 How is feedback used?
text: An overwhelming majority of past work simply fine-tunes their task model using human feedback; whether it is a general purpose language model (Ouyang et al., 2022; Bai et al., 2022) or a taskspecific model (Shi et al., 2022; Gao et al., 2022; Saunders et al., 2022; Scheurer et al., 2022; Schick et al., 2022). Tandon et al. (2022) differently finetunes a separate corrector model which takes in a retrieved critique utterance to correct initial outputs. Similarly, Madaan et al. (2022) retrieves from a memory of previous critiques to improve GPT-3 predictions via few-shot prompting.
Our work separates from existing work by focusing on critique generation and harnessing critiques that yield better final outcomes by LMtask. Similar to Schick et al. (2022), we effectively propose a multi-agent setup by disentangling critique gener-
ation and conditional refinement. Differently, we keep the latter model intact and only train the critique generator LMcritique via reinforcement learning. Moreover, we take a step forward by leveraging end task data for the first time and directly optimize the critique generation process to improve final task performance. In contrast to RLHF whose policy network (LMtask) is trained to maximize human alignment (Wiener, 1960), our policy network (LMcritique) is trained to bootstrap end-task success of LMtask. Our proposal RL4F is orthogonal to RLHF; in fact we use an RLHF fine-tuned checkpoint in our experiments. For further discussion, please refer to Fernandes et al. (2023) who catalogue different approaches on integrating natural language feedback to textual generations.

heading: 2.3 Adapters & Discrete Prompt Learning
text: A large body of existing work finds that parameterefficient fine-tuning, often referred to as adapters (Pfeiffer et al., 2020) is as effective as full finetuning while being computationally cheap. RL4F can also be interpreted as an alternative “adapter” under the strict setting where only textual access to task model is available. Furthermore, our work can also be viewed from the perspective of learning discrete prompts for language models. Past work propose to generate knowledge pieces (Liu et al., 2022) or arbitrary textual snippets (Deng et al., 2022) which they append to the input via reinforcement learning. These works are different than ours in that their policy is conditioned solely on the input x whereas in our case we sample critiques of machine-generated predictions based on x and ŷ.

heading: 3 Background
text: The problem of learning from critiques entails two major challenges: (1) generating critiques and (2) the task of refining initial answers based on a critique. In our experiments (Section 6), GPT-3 responds very well to ground-truth critiques. This observation suggests that given quality critique, GPT-3 is indeed able to improve a potentially erroneous prediction for the better. Hence, in this study we focus our efforts on (1). Our ultimate goal is to reach, and eventually exceed, human-level critique performance using machines.
Following Saunders et al. (2022), we identify four primary functions towards studying the problem of learning with natural language critiques.
First is PREDICT: the base task of predicting without using critiques to model x → y. As an example, if x is a passage, y is the summary (see Fig. 1). Moreover, we refer the task of learning to generate critiques x, ŷ → c where ŷ ∼ LMtask(x) as CRITIQUE. Lastly, we call the conditional refinement objective x, ŷ, c → y as REFINE and repairing an answer without a critique x, ŷ → y as DIRECTREFINE3. We use ŷ and ĉ notation to indicate an estimate of ground truth y, and similarly for c, from a respective model.

heading: 3.1 SUPERVISED: Supervised Learning for Critique Generation
text: We initialize LMcritique to be a pretrained encoderdecoder-type model and fine-tune it to generate critiques satisfying the CRITIQUE objective x, ŷ → c using natural language critiques. Namely, if LMcritique is parameterized by θ we maximize E [log pθ(c|x, ŷ)]. We delegate PREDICT and REFINE tasks to GPT-3 via in-context learning. The procedure is depicted in Fig. 2a-b.
The main difference of our implementation of SUPERVISED to that of Saunders et al. (2022)’s is that we rely on separate models for CRITIQUE and the rest of the tasks while they train a single GPT-3style model to collectively achieve PREDICT, CRITIQUE, REFINE and DIRECTREFINE; effectively merging LMcritique and LMtask into a single model. While this may seem parameter-efficient, our version has a few key advantages. First, leaving any
3Additionally, Saunders et al. (2022) put forth the tasks of critiqueability to gauge whether an output requires a critique in the first place and helpfulness referring to if a sampled critique is useful. We let the critique generator model to implicitly take care of these tasks.
LMtask model intact (parameters frozen) enables us to work with models that are already-deployed as LMtask and those with expensive training and inference processes. Moreover, our approach refrains from disturbing overall integrity of a generalpurpose language model by conditioning it to a specific task. Lastly, training LMcritique, which is multiple orders of magnitude smaller than GPT-3, is much more computationally efficient and therefore accessible to a broader range of users.

heading: 3.2 Direct-Refinement
text: Madaan et al. (2023); Chen et al. (2023) propose that using the critiques from the model itself via few-shot prompting results in improved performance. On the contrary, Saunders et al. (2022) and Bai et al. (2022) argue that direct refinement (as denoted with DIRECTREFINE in this work) i.e. the practice of prompting a language model without self-generated critiques to directly repair its own answers proves a stronger baseline, especially when the model size is large >50B. They hypothesize that this is primarily due to model’s initial answers getting increasingly more difficult to selfcritique as the model size grows. In fact, both Saunders et al. (2022) and Bai et al. (2022) showed that their largest model achieves superior end-task performance when performing DIRECTREFINE than refining using self-generated critiques. Hence, we use Direct-Refinement as a baseline and describe how we implement it via in-context learning in Section 6 while providing further discussions in Appendix B.5.

heading: 4 RL4F: Reinforcement Learning for Feedback Generation
text: SUPERVISED is straightforward to implement but it does not make use of any final task data (x → y) that is usually more abundant than natural language critiques. Moreover, it fails to provide ground for adaptation when the critiques in the train set are generally applicable but not entirely tailored to improving a target model. We describe RL4F where we follow supervised training with policy gradient learning using end-task data in order to generate critiques. We assume that the task model LMtask is already deployed and treat it as a fixed module. In all of our implementations we train the natural language critique generator LMcritique alone. In both SUPERVISED and RL4F, LMcritique takes in the input x and an initial prediction ŷ and produces a (natural language) critique ĉ:
LMcritique(x, ŷ) = ĉ . (1)
Fig. 2c provides an illustration of RL4F. We implement LMtask as GPT-3 given its adaptability into new tasks using few-shot prompting. Our implementation which is primarily based on the RL4LMs library4 (Ramamurthy et al., 2022) will be publicly available.
Learning via Policy Gradient We warm-start RL4F by first fine-tuning LMcritique for CRITIQUE which we defined as is the supervised objective of learning to generate natural language critiques c conditioned on x, ŷ. We continue fine-tuning the policy network (LMcritique) to maximize the reward using Proximal Policy Optimization (Schulman et al., 2017). We utilize the implementation of PPO provided by Ramamurthy et al. (2022) and refer the readers to the original work about the details for KL-regularized PPO objective. While any policy gradient approach could be used e.g. REINFORCE (Williams, 1992), our initial experiments showed that PPO works best in this setting.
Pseudocode for RL4F is provided in Algorithm 1 where we use two sets of in-context examples for prompting GPT-3. We define E to be a set of incontext-learning examples in the form of (x, y) to get GPT-3 solve PREDICT. Similarly, Ec contains in-context examples to prompt GPT-3 to fix an initial attempt ŷ into y conditioned on the natural language critique c which we termed as REFINE; Ec = {(x, ŷ, c, y)}. As per our reward function,
4https://github.com/allenai/RL4LMs
Algorithm 1 RL4F Pseudocode of the algorithm used to train feedback model.
Input: Dataset D = {(xi, ŷi, yi)}Ni=1 of size N Input: Initial LMcritique, LMtask Input: In-context examples for refinement Ec repeat
Sample mini-batch Dm = {(xm, ym, ŷm)}Mm=1 ∼ D Sample ĉ ∼ LMcritique(x, ŷ) for Dm in parallel Sample ŷnew ∼ LMtask(Ec, x, ŷ, ĉ) for Dm Compute KL-regularized rewards Rt ▷ Eq. (2) Compute the advantage estimate Ât Update the LMcritique by maximizing the PPO objective
until convergence and return LMcritique
we opt to use a lexical similarity metric ROUGE (1/2/L) (Lin, 2004) in Eq. (2) for planning and summarization datasets. Measuring ROUGE is computationally fast, making it easy to use in an online learning setting. Reward is only collected at a terminal stage i.e. either when the end of sentence token is produced or maximum number of tokens is reached.
R(ŷ, y) = mean (R1(ŷ, y),R2(ŷ, y),RL(ŷ, y)) (2)

heading: 5 Datasets

heading: 5.1 Topic-Based Summarization
text: Saunders et al. (2022) crowd-sourced natural language critiques for topic-based summarization. The train, validation and test sets contain 14230, 1150 and 2658 tuples of (x, ŷ, ĉ). The dataset provides multiple questions for a given passage each inquiring about a different aspect. Given a passage and question (x) multiple summaries are sampled from the model. Human annotators provide natural language critiques for the answers along with improved summaries. One example is provided in Fig. 1 and more are available in the Appendix (Table 8).

heading: 5.2 Interscript
text: Interscript (Tandon et al., 2021) is an action planning dataset for everyday tasks such as “put on a costume” or “play musical chairs”. Each goal x is associated with a sequence of ground truth actions y. Along with x, y pairs, it contains erroneous action plans ŷ and natural language critiques ĉ suggesting a fix. An example is provided in Fig. 1 for “put soap in dishwasher”. Other examples of critiques are “You need to have music to play musi-
cal chairs.” and “You need to pay for the costume before leaving the store”. More examples are available in the Appendix (see Table 6). Interscript represents a low-resource scenario: it contains 253, 45 and 169 examples for train, validation and test sets where each example contains 1-4 reference texts.

heading: 5.3 Synthetic Task: Alphabetization
text: We synthetically generate a task for alphabetically sorting a list of words with lengths ranging between 3-12. We use the lexicon #11 by Keith Vertanen (2018) which contains 43K unique English words. Given an unsorted list and a ground truth sorting of the list we identify 5 operations to sample a incorrect sorting of y denoted by ŷ and associated critique c articulating what is wrong about ŷ in natural language. One example is shown below:
x: mug, greek, book, house y: book, greek, house, mug ŷ: book, greek, house c: The word mug is missing.
The operations we use for distortion are REORDER, REPLACE, ADD, REPEAT and REMOVE (shown above). We also leave majority of sorted lists intact for which the ground truth critique is “The list is correctly sorted”. We use a total of 40K examples for warm-starting LMcritique for the CRITIQUE objective and another 10K, 1K and 1K examples for PPO stage, for train, dev and test splits, respectively. Examples delineating other operations in action and corresponding natural language critiques are provided in Appendix A.
In alphabetization, we use Inverse-Levenstein distance for the reward function R as defined in Eq. (3) where |·| measures length of the list. Levenstein distance is a form of edit distance for single character edit operations such as removal, insertion and substitution in a string. We count word-level operations rather than character-level. Note that the higher inverse-Levenstein score of a predicted ordering, the closer it is to the alphabetically sorted version. The sorted list gets the maximum reward of 1.
R(ŷ, y) = 1− Levenstein(ŷ, y) max(|ŷ|, |y|) (3)

heading: 6 Experiments and Results
text: Our experiments are designed to test effectiveness of RL4F, along with other sources of critiques, in
both natural and controlled settings. In our evaluations, we test the usefulness of critiques by looking at the final task performance rather than evaluating generated critiques themselves; as multiple critiques may lead to the same correct answer.
Sampling Critiques We sample critiques from LMcritique as in Eq. (1) by first concatenating the input and initial prediction. The specific input format for LMcritique we use for Interscript is given below and the other two can be found in Appendix B.1. We initialize LMcritique with pretrained T5-large which is a 0.77M parameter encoder-decoder type language model trained on large web text (Raffel et al., 2020).
Goal: {goal} Steps: {initial_answer}
Downstream Predictions In our experiments, we consider GPT-3 as the LMtask model. GPT-3 can handle a wide range of tasks with prompting— using a handful of task examples in the input and without requiring task-specific fine-tuning (Brown et al., 2020). GPT-3 is not only able to tackle numerous tasks conveniently but also can refine initial predictions when given a natural language critique (Madaan et al., 2022). Since our setting requires LMtask model to be able to model both the main task objective x → y and the refinement objective x, ŷ, c → y, GPT-3 is a suitable candidate that can adapt to both, using few-shot exemplars. The prompt template we use for the latter is shown in Fig. 3 where we provide the model with an initial attempt to the question initial_answer and re-sample a revised prediction conditioned on the question and critique for the summarization task. We use code-davinci-002 checkpoint via OpenAI API5 and 3, 1 and 6 hand-written incontext examples for planning, summarization and alphabetizations tasks, respectively as we exhaust the 4096 token input limit.
In action planning, instead of resampling entire plans, we prompt GPT-3 to produce an edit operation on the initial plan ŷ. The set of edit operations identified in the original dataset are Insert, Remove and Reorder where each critique comes with a corresponding edit operation. Note that, these operations can algorithmically be applied to ŷ. While Reorder and Remove are expected to refer to existing steps in ŷ, we expect Insert to introduce a
5We find that only the largest 175B parameter GPT-3 can handle REFINE.
novel action. Hence, we stick with a generic lexical similarity metric in calculating reward (Eq. (2)) for this task. In summarization, we compare humanwritten summaries with the repaired model summaries.
Baselines We compare effectiveness of RL4F to SUPERVISED which is described in Section 3.1. This is the closest baseline to the approach by Saunders et al. (2022) and Schick et al. (2022) that abides by our condition that LMtask should remain unchanged. We use the same set of initial predictions ŷ when comparing different critique generators.
In addition to SUPERVISED, we use a simple Direct-Refinement baseline where we ask LMtask to revise the initial prediction given a fixed critique “Improve the answer.” (DIRECTREFINE). The prompt template is otherwise the same as in other methods. We configure our in-context examples to show that not all ŷ need to be repaired. Hence, LMtask is free to update the prediction or leave
it as is when it is correct. Despite its simplicity, Direct-Refinement has been established as a strong baseline (Saunders et al., 2022).
Moreover, we compare to MemPrompt (Madaan et al., 2022). In their work, authors study a setup where LMtask generates an understanding along with the target output. For example, given a question “What sounds like good?”, the model generates an understanding of the question “The question is asking for homonym.” before saying “wood”. In their critique retriever, they train a mapping to model x into an understanding u. However understanding is redundant in particular tasks e.g. summarization where the question is no different than u, thus throughout our experiments, we replace the learned retriever in MemPrompt with BM25 (Harter, 1975).
Lastly, we use human-written critiques (gold feedback) for REFINE in getting LMtask to repair outputs and report this as an upperbound.

heading: 6.1 Planning and Summarization
text: Our main results for Interscript and topic-based summarization are provided in Table 1. Given the
free-from nature of the outputs, we evaluate planning and summarization tasks using text similarity metrics to capture semantic and lexical similarities. We utilize learned metrics such as BLEURT (Sellam et al., 2020) and BERTScore (Zhang* et al., 2020) along with ROUGE (Lin, 2004). We compare the performance achieved by using different sources of critiques to that of human-written critiques. Across all metrics, RL4F yields one of the closest outcomes to human-written critiques.6

heading: 6.2 Alphabetization
text: We initialize our LMcritique using the synthetic critiques as described Section 5.3. Our results are provided in Table 2. For alphabetization we compute exact match and inverse-Levenstein scores as defined in Eq. (3). As an additional baseline, we fine-tune davinci (Brown et al., 2020) on the same train set as our RL4F.
Because of the synthetic procedure to create x, ŷ, c triplets, the generated ŷ as well as c do not necessarily reflect the kinds of errors that LMtask would do. We observe this in the scores of SUPERVISED which fails to improve upon initial outputs. Nevertheless, RL4F procedure helps the policy network to capture a useful distribution of critiques, improving over SUPERVISED by more than 27 absolute points. In this simple task, DirectRefinement prompt also yields a competitive performance. Compared to full fine-tuning, we observe that despite training substantially fewer parameters RL4F achieves a significantly better accuracy. For a comparison to concurrent work Madaan et al. (2023), please refer to the appendix.

heading: 7 Analysis
text: Scaling Properties While we use T5-large as our main model for all of our experiments to initialize LMcritique, we inquire about different model sizes. In Fig. 4, we consider three different model sizes to tackle Interscript ranging from 60M to 770M parameters. On the y-axis we provide average of three ROUGE scores for the generated plans. RL4F greatly benefits from an increase in the model size where a similar trend in SUPERVISED is non-existent.
6We have identified a handful of examples where a pair of train and test examples differs by only a single concept e.g. all occurrences of “noodle” in the train sample was replaced with “food” to produce the test sample. The goal and steps are the same otherwise. MemPrompt does exceedingly well on these 7 cases, hence performing occasionally higher, yet fails in the rest of the test/val examples.
Semantic Drift In goal-oriented training, semantic drift occurs when the strings produced by the policy begin diverging from initial language (Lee et al., 2019; Blank, 1999). Although, RL4F does not guarantee that ĉ ∼ LMcritique will be natural language, we find minimal sign of semantic drift in the sampled critiques with respect to fluency and naturalness. In most cases, generated critiques are intelligible. We speculate that may be due to GPT-3 responding to natural language best than gibberish, though future work should look closely into this to make a more conclusive argument. We provide sample predictions from both models in Appendix C for all three tasks.
Iterative Improvement In Section 6, we provide results with applying only one round of critiques in alphabetization. Past work advocated for iterative editing (Reid and Neubig, 2022; Faltings et al., 2021) as opposed to one-shot editing. In Fig. 5, we sample and apply critiques from LMcritique to ŷ’s iteratively and see if the number of correctly
sorted lists increase or decrease. Note that critiques may also lead to deteriorating performance as we are not eliminating the solved examples and it is at LMcritique’s discretion to declare a solution as correct e.g. by saying “This list is correctly sorted.”. In fact, when we ask GPT-3 to simply improve its predictions iteratively (via Direct-Refinement as described in Section 3.2), it occasionally scrambles an already correct ordering while not scoring any new points. In contrast, RL4F leads to up to 7 more corrections (see Fig. 5).

heading: 8 Conclusion
text: We have described a collaborative framework involving two language models where one model, the critique generator, is trained to improve the performance of the task model. We train the former via policy gradient learning while treating the task model as a black-box. We show that RL4F leads to superior final performance across three domains compared to other strong baselines without resulting as the critiques remain fluent and natural. Future work might focus on generalizing the critique generator into a mixture of experts allowing humans and other models to contribute to critiqueing procedure.

heading: 9 Limitations
text: RL4F is primarily targeted at improving final performance. While we have found that the critiques learned by RL4F remain natural, we do not introduce any explicit restraints preventing semantic drift. As though it may raise end-task performance, semantic drift would also hinder interpretability. Future work might study datasets that are not covered by this dataset and quantify semantic drift along with proposing measures to prevent it, as necessary. Moreover, this work does not provide an explicit mechanism to incorporate new critique labels that might become available in the future nor it identifies a framework that could combine critiques from multiple experts such humans and other machines. Lastly, we limit our analysis to GPT-3 and focus on a scenario where it is inefficient or impossible to train the task model while this may be a conservative assumption for other settings.

heading: Acknowledgement
text: We thank Aman Madaan for sharing his insights throughout this project. We also thank Anna
Ivanova, Jacob Andreas, Zilu Tang, Shashank Gupta and Ashish Sabharwal for their valuable feedback on earlier drafts of this paper. Finally, we thank Rajkumar Ramamurthy and Prithviraj Ammanabrolu for helpful discussions on using their RL4LMs repository which facilitated the experiments of this work.
Afra Feyza Akyürek is supported in part by the U.S. NSF grant 1838193 and DARPA HR001118S0044 (the LwLL program). The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes. The views and conclusions contained in this publication are those of the authors and should not be interpreted as representing official policies or endorsements of DARPA and the U.S. Government. At MIT, Ekin Akyürek is supported by an MITAmazon ScienceHub fellowship and by the MITIBM Watson AI Lab.

heading: A Dataset Processing
text: Action Planning Interscript is larger but we are only using a subset, removing distractors. The scripts used for data cleaning will be released along with the codebase.
Alphabetization We sample initial predictions from GPT-3 for alphabetization some of which comprise multiple distortions simultaneously, yet we use one-step distortions to warm-start LMcritique.
Given the following a pair of unsorted and sorted word lists e.g.
x: mug, greek, book, house y: book, greek, house, mug
below are the operations we used to create our data:
REORDER ŷ: book, house, greek, mug c: The word greek is placed in an incorrect position.
REPLACE ŷ: book, greek, house, mud c: The word mug is replaced with mud
REMOVE ŷ: book, greek, mug c: The word house is missing
REPEAT ŷ: book, house, greek, house, mug c: The word house is repeated
ADD ŷ: book, hair, greek, house, mug c: The word hair is not in the original list
NOTHING
ŷ: book, greek, house, mug c: The list is correctly sorted.

heading: B Experiment Details
text: We use code-davinci-002 as GPT-3 unless otherwise specified. We compute ROUGE implementation in the datasets library and set use_stemmer=True for summarization and Interscript.
B.1 Data Formats We use the following input formats for LMcritique:
• Summarization:
{passage} Question: {question} Answer: {initial_answer}
• Planning:
Goal: {goal} Steps: {steps}
• Alphabetization:
{unsorted_list} ||| {initial_answer}
We train separate models for each of the datasets and evaluate individually. We use T5-large provided by transformers library.
B.2 Prompts for GPT-3 We provide prompt templates used for Alphabetization and Interscript when prompting GPT-3 for REFINE. Template for summarization is provided in the main text.
• Interscript:
Goal: {goal} Steps: {steps} Feedback: {critique} Edit:
• Alphabetization:
{unsorted_list} ||| {initial_answer} Feedback: {critique} Edit:
In Direct-Refinement, templates remain the same and critique’s are replaced with “Improve the answer.”. Exact prompt exemplars will be made available in the released code repository.
B.3 Standard Deviations
Standard deviations of R1/R2/RL scores across 5 runs in Interscript are 1.4/0.1/1.1 for SUPERVISED and 0.5/0.4/0.5 for RL4F.
B.4 Hyperparameters
In all of our experiments we use temperature 0 for prompting GPT-3 except when sampling initial predictions for alphabetization we set it to 0.5. We provide hyperparameters for RL4LMs (Ramamurthy et al., 2022) in Table 3, Table 4 and Table 5.
B.5 Results for Self-Refine (Madaan et al., 2023) on Alphabetization
Madaan et al. (2023) and Chen et al. (2023) propose that self-generated critiques (sampling critiques simply via few-shot prompting) is useful for a range of tasks. We examine if self-generated critiques are more useful for Alphabetization task than other techniques proposed in Table 2. Doing so, we curate a few-shot prompt:
Below is a given list of words which are supposed to be sorted in alphabetical order.
Describe what is wrong in the provided ordering.
---
Ordering: quirky whimsical bubbly joyous delightful melodic glimmering vivacious radiant lively zestful spontaneous Feedback: Whimsical should come in the end. Delightful should come before joyous.
---
Ordering: airy amiable animated ardent astute beaming blithe brilliant Feedback: This listed is correctly sorted.
---
Ordering: curious sprightly vivacious tenacious passionate vivacious Feedback: The list contains duplicates and passionate should come before sprightly.
---
Ordering: {ordering} Feedback:
After sampling critiques using the above prompt, we follow the same steps described in Section 6 for refinement. We obtain 21.6% exact match accuracy using these critiques which is a significant drop from code-davinci-002’s initial performance. When we sample critiques and refinements from text-davinci-002, the exact match score increases notably (to 58.6%) while still hurting the initial accuracy of 63.6%. SelfRefine may improve over initial performance when critiques are sampled from more capable models such as text-davinci-003 (Madaan et al., 2023). Nonetheless, having initialized as T5-large, RL4F’s critique model already produces useful feedback despite being significantly smaller than text-davinci-003.

heading: C Sample Predictions
text: Sample predictions for all three tasks are provided in Table 6, Table 7 and Table 8. A manual examination of the generated critiques revealed that close to 100% of the critiques generated for alphabetization and action planning are grammatical. However, critiques for the topic-based-summarization task often involve repetitions or generic calls to fix or improve the answer e.g. "The answer is not adequate. The answer is wrong, it is wrong and should be fixed."

heading: D Learning Curves for Reinforcement Learning
text: In Fig. 6, we provide how evaluation metrics progress as LMcritique in RL4F is trained.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
9
7 A2. Did you discuss any potential risks of your work? Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Left blank.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 6
3 B1. Did you cite the creators of artifacts you used? Left blank.
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Datasets we used are made publicly available by the authors for scientific use. Interscript is available via Apache 2.0.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Left blank.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Left blank.
7 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.
C 3 Did you run computational experiments? 6
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? 6, Appendix B
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? 6, Appendix B
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 6
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 6, Appendix
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Future work might study datasets that are not covered by this dataset and quantify semantic drift along with proposing measures to prevent it, as necessary. Moreover, this work does not provide an explicit mechanism to incorporate new critique labels that might become available in the future nor it identifies a framework that could combine critiques from multiple experts such humans and other machines. Lastly, we limit our analysis to GPT-3 and focus on a scenario where it is inefficient or impossible to train the task model while this may be a conservative assumption for other settings