Self-Refine: Iterative Refinement with Self-Feedback
Authors: Aman Madaan
Niket Tandon
Prakhar Gupta
Skyler Hallinan
Luyu Gao
Sarah Wiegreffe
Uri Alon
Nouha Dziri
Shrimai Prabhumoye
Yiming Yang
Shashank Gupta
Bodhisattwa Prasad Majumder
Katherine Hermann
Sean Welleck
Amir Yazdanbakhsh
Peter Clark
Venue: NeurIPS 2023 poster (2023)
Keywords: LLMs
Iterative Refinement
Feedback-driven Generation
TL;DR: Self-Refine is a novel approach that allows LLMs to iteratively refine outputs and incorporate feedback along multiple dimensions to improve performance on diverse tasks.

Abstract: Like humans, large language models (LLMs) do not always generate the best output on their first try. Motivated by how humans refine their written text, we introduce Self-Refine, an approach for improving initial outputs from LLMs through iterative feedback and refinement. The main idea is to generate an initial output using an LLMs; then, the same LLMs provides *feedback* for its output and uses it to *refine* itself, iteratively. Self-Refine does not require any supervised training data, additional training, or reinforcement learning, and instead uses a single LLM as the generator, refiner and the feedback provider.  We evaluate Self-Refine across 7 diverse tasks, ranging from dialog response generation to mathematical reasoning, using state-of-the-art (GPT-3.5, ChatGPT, and GPT-4) LLMs. Across all evaluated tasks, outputs generated with Self-Refine are preferred by humans and automatic metrics over those generated with the same LLM using conventional one-step generation, improving by $\sim$20\% absolute on average in task performance. Our work demonstrates that even state-of-the-art LLMs like GPT-4 can be further improved at test-time using our simple, standalone approach.

SELF-REFINE: Iterative Refinement with Self-Feedback
Aman Madaan', Niket Tandon', Prakhar Gupta', Skyler Hallinan 8 Luyu Gao', Sarah Wiegreffe Uri Alon', Nouha Dziri², Shrimai Prabhumoye', Yiming Yang', Shashank Gupta?, Bodhisattwa Prasad Majumder', Katherine Hermann", Sean Welleck 2.3 Amir Yazdanbakhsh Peter Clark?
Language Technologies Institute, Carnegie Mellon University
Allen Institute for Artificial Intelligence
University of Washington 'NVIDIA UC San Diego GGoogle Deepmind ielorrein@@goggeerroops...
Abstract
Like humans, large language models (LLMS) do not always generate the best output on their first try. Motivated by how humans refine their written text, we introduce SELF-REFINE, an approach for improving initial outputs from LLMS through iterative feedback and refinement. The main idea is generate initial output using an LLM; then, the same LLM provides feedback for its output and uses it refine itself, iteratively. SEI.F-REFINE does not require any supervised training data, additional training, or reinforcement learning, and instead uses a single LLM as the generator, refiner, and feedback provider. We evaluate SELF-REFINE across 7 diverse tasks, ranging from dialog response generation to mathematical reasoning, using state-of-the-art (GPT-3.5 and GPT-4) LLMS. Across all evaluated tasks, outputs generated with SELF-REFINE are preferred by humans and automatic metrics over those generated with the same LLM using conventional one-step generation, improving by ~20% absolute on average in task performance. Our work demonstrates that even state-of-the-ar LLMS like GPT-4 can be further improved at test-time using our simple, standalone approach.".
Introduction
Although large language models (LLMS) can generate coherent outputs, they often fall short in addressing intricate requirements. This mostly includes tasks with multifaceted objectives. such as dialogue response generation, or tasks with hard-to-define goals, such as enhancing program readability. In these scenarios, modern LLMS may produce an intelligible initial output. yet may benefit from further iterative refinement-i.e.. iteratively mapping a candidate output to an improved one-to ensure that the desired quality is achieved. Iterative refinement typically involves training refinement model that relies on domain-specific data (e.g., Reid and Neubig (2022); Schick et al. (2022a); Welleck et al. (2022)). Other approaches that rely on external supervision or reward models require large training sets expensive human annotations (Madaan et al., 2021; Ouyang et al., 2022), which may not always be feasible to obtain. These limitations underscore the need for an effective refinement approach that can be applied to various tasks without requiring extensive supervision.
Iterative self refinemefin is a fundamental characteristic human problem-solvin (Simon, 1962; Flower and Hayes, 1981; Amabile, 1983). Iterative self-refinement is process that involves creating an initial draft and subsequently refining it based on self-provided feedback. For example, when
Now at Google DeepMind
"Cdde and data at httpps//seelrrfinne info/
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Input
Feedback
Refine
Model M
Use M to get feedback on its output
Use M to refine its previous output, given its feodback
Figure I: Given an input (Q), RELF-REFINE starts by generating an output and passing it back to the same model M to get feedback (i). The feedback passed back M, which refines the previously generated output (2). Steps (i) and (2) iterate until stopping condition is met. SELF-REFINE is instantiated with language model such as GPT-3.5 and does not involve human assistance.
drafting an email to request a document from colleague, an individual may initially write direct request such "Send me the data ASAP", Upon reflection, however, the writer recognizes the the phrasing and revises to "Hi Ashley, could you please send me the data al your earliest convenience?". When writing code, programmer may implement an initial "quick and dirty" implementation, and then, upon reflection, refactor their code to a solution that is more efficient and readable. In this paper, demonstrate that LLMs provide iterative self-refinement without additional training, leading to higher-quality outputs on a wide range of tasks.
potential impoliteness
We present REFFNNE an iterative self-refinement algorithm that alternates between two gener- ative steps-FEEDBAR and REFINE. These steps work tandem generate high-quality outputs. Given an initial output generated by a model M, we pass it back to the same model M to get feedbuck. Then, the feedback is passed back to the same model to refine the previously-generanl draft. This process is repeated either for specified number of iterations or until M determines that no further refinement is necessary. We use few-shot prompting (Brown et al., 2020) to guide M to both generate feedback and incorporate the feedback into improved draft. Figure illustrates the high-level idea, that SELF-REFINE uses the same underlying language model to generate feedback and refine its outputs.
We evaluate REFFNE on generation tasks that span diverse domains, including natural language and source-code generation. We show that REFFER outperforms direct generation from strong and 5.0-33.-turro; OpenAI; Ouyang et al., 2022) and
LLMS like GPT-3.5 (text-davinci- GPT-4 (OpenAI, 2023) by 5-40% absolute improvement. In code-generation tasks, SELF-REFINE improves the initial generation by up to absolute 13% when applied to strong code models such as CODEX ccode-davinci-eeee
Chen et al., 2021). We release all of our code, which is easily extensible to other LLMs. In essence, our results show that even when an LLM cannot generate an optimal output its first try, the LLM often provide useful feedback and improve its own output accordingly. turn, SELF-REIE REFINE provides an effective way to obtain better outputs from a single model without any additional training, via iterative (self- feedbbcke and refinement.
Iterative Refinement with
Given an input sequence, REFINE generates an initial output, provides feedback on the output, and refines the output according to the feedback. SELF-REFINE iterates between feedback and refinement until desired condition met. SELF-REFINE relies on a suitable language model and three prompts (for initial generation, feedback, and refinement), and does not require training. SELF-REFINE shown Figure and Algorithm Next, we describe SELFEREFINE in more detail.
Initial generation Given an input x, prompt Pgen. and model M, SELF-REFING generates an initial output yo:
yo M (Pgen (ar).
(1)

---

(a) Dialogue: 2, Yt
(b) FEEDBACK fb
(c) REFINE Yt+1
User: am interested in playing Table tennis.
Engaging: Provides no information about table tennis or how to play it.
Response (refined): That's great to hear (...) It's fun sport requiring quick reflexes and good hand-eye coordination. Have you played before, or are you looking to learn?
Response: I'm sure it's a great way to socialize, stay active
User understanding: Lacks understanding of user's s needs and state of mind.
(d) Code optimization: Y,
(c) FEEDBACK fb
(f) REFINE Yt+1
Generate of 1, M def sum (
This code slow as uses brute force. A better approach is to use the formula (n(n+1))/2.
Code (refined)
res 0 for in range(n+1);
def sum_ sum_faster (n) return (n* (1+1))//2
return res
Figure 2: Examples of SELF-REAINE: initial output generated by the base L.M and then passed
back the same LLM receive feedback to the same illustrates this for dialog generation where an initial dialogue response can be transformed into more engaging one that also understands the user by applying feedback. The bottom illustrates this for code optimization where the code is made more efficient by applying feedback.
refine the output The top row
Algorithm 1 SELF-REFINE algorithm
Require: input x, model M, prompts {peen: Pro: Prefine stop condition stop(-) Initial generation (Eqn. 1)
1: yo = =M[geen] ||r)
2: for iteration E 0, ....dd
3: fb, = .M (Pro | (bb)
Feedback (Eqn. 2)
4: if stop(ff.., then
Stop condition
5: break
6: else
7: Vt+1 M
> Refine (Eqn. 4)
8: end
9:
end for
10:
return 1/c
Figure 3: The SELF-REFINE algorithm. See ($2) for discussion each component.
For example, Figure 2(d), the model generates functionally correct code for the given input. Here, Pgen is task-specific few-shot prompt (or instruction) for initial generation, and || denotes concatenation. The few-shot prompt contains outptt pairs (x), (k)) for the task.?
FEEDBACK Next, SELF-REFINE uses the same model M provide feedback fb, on its own output, given task-specific prompt Pn for generating feedback:
fbr M (pa||=//ee))
(2)
Intuitively, the feedback may address multiple aspects of the output. For example, in code optimiza- tion. the feedback might address the efficiency, readability, and overall quality of the code.
Few-shot prompting (also referred to "in- contect learning") provides a model with prompt consisting of k in-context examples of the target task, each in the form of input-output pairs (29,, gv) (Brown et al., 2020).

---

Here, the prompt Pn provides examples of feedback in the form of input-output- feepback triples (x(h), y(h), fb(k)) We prompt the model to write feedback that is actionable and specific via fó(() By 'actionable', we mean the feedback should contain a concrete action that would likely improve the output. By 'specific', we mean the feedback should identify concrete phrases in the output to change. For example, the feedback in Figure 2(e) is "This code is slow as it uses a for loop which is brute force. A better approach is to use the formula... (n (n+ 1) /2". This feedback is actionable, since it suggests the action 'use the formula...' The feedback is specific since it mentions the 'for loop'.
REFINE Next, SELF-REFINE uses M to refine its most recent output, given its own feedback:
yt+1 = .M (Pretine (toetine/=[lll]]]11 ||
For example, in Figure 2(f), given the initial output and the generated feedbuck, the model generates re-implementation that is shorter and runs much faster than the initial implementation. The
prompt Pretine provides examples of improving the output based on the feedback, in the form of (k)
(k) input-outppu-fecdaae--eeeeedd quadruples (1/(), ; Sb, 1/+1/
Iterating SELF-REFINE SELF-REFINE alternates between FEEDBACK and REFINE steps until a stopping condition is met. The stopping condition stop(fb,, t) either stops at specified timestep t, or extracts a stopping indicator (e.g. a scalar stop score) from the feedback. In practice, the model can be prompted to generate stopping indicator in Pm. and the condition is determined per-task.
To inform the model about the previous iterations, we retain the history of previous feedback and outputs by appending them to the prompt. Intuitively, this allows the model to learn from past mistakes and avoid repeating them. More precisely, Equation (3) in fact instantiated as:
1 M (Prennel|az)) I-
Finally, we use the last refinement Yt as the output SELF-REFINE.
Algorithm 1 summarizes SELF-REFINE, and Figure shows an example of SELF-REFINE in the Dialogue Response Generation (Mehri and Eskenazi, 2020) and Code Optimization (Madaan et al., 2023) tasks. Appendix V provides examples of the Pyen: Ptb. Prefine prompts for various tasks. The key idea is that SELF-REFINE the same underlying L.LM to generate, get feedback, and refine its outputs given its own feedback. It relies only on supervision present in the few-shot examples.
Evaluation
We evaluate SELF-REFINE on 7 diverse tasks: Dialogue Response Generation (Appendix P: Mchri and Eskenazi, 2020), Code Optimization (Appendix Q; Madaan et al., 2023), Code Readability Improvement (Appendix O; Puri et al., 2021), Math Reasoning (Appendix R; Cobbe et al., 2021), Sentiment Reversal (Appendix S; Zhang et al., 2015), and we introduce two new tasks: Acronym Generation (Appendix T) and Constrained Generation (a harder version of Lin et al. (2020) with 20-30 keyword constraints instead of 3-5; Appendix U)
Examples for all tasks and dataset statistics are provided in Table 4 (Appendix A).
3.1 Instantiating SELF-REFINE
We instantiate SELF-REFINE following the high-level description in Section 2. The FEEDBACK- REFINE iterations continue until the desired output quality or task-specific criterion is reached, up to a maximum of 4 iterations. To make our evaluation consistent across different models, we implemented both FEEDBACK and REFINE as few-shot prompts even with models that respond well to instructions, such CHATGPT and GPT-4.
Base LLMs Our main goal is to evaluate whether we can improve the performance of any strong base LLMs using SELF-REFINE. Therefore, we compare SELF-REFING to the same base LLMs but without feedback-refine iterations. We used three main strong base LLM across all tasks: GPT-3.5 Lext-davvici-O033 CHATGPT (gpt-3.5-turl 5-tubb)), and GPT-4 (OpenAI, 2023). For base- tasks, we also experimented with CODEX scode-davinci-000 GPT-4 is the previous ttate-of-the-art We used the same prompts from previous work when available
In all tasks, either GPT-3.5 or
A comparison with other sho-shot and fine-tuned approaches is provided in Appendix H

---

GPT-3.5
CHATGPT +SELF-REFINE
GPT-4
Task
Base
+SEIF-REFINE
Base
Base
+SELF-REFINE
Sentiment Reversal
8.8
30.4 (121.6)
11.4
43.2 (131.8)
3.8
36.2 (132.4)
Dialogue Response
36.4
63.6 (127.2)
40.1
59.9 (119.8)
25.4
74.6 (149.2)
Code Optimization
14.8
23.0 (18.2)
23.9
27.5(13.6)
27.3
36.0 (18.7)
Code Readability
37.4
51.3 (113.9)
27.7
63.1 (135.4)
27.4
56.2 (128.8)
Math Reasoning
64.1
64.1 (0)
74,8
75.0 (10.2)
92.9
93.1 (10.2)
Acronym Generation
41.6
56.4 (114.8)
27.2
(1100)
30.4
56.0 (125.6)
Constrained Generation
16.0
39.7 (123.7)
2.75
33.5 (130.7)
4.4
61.3 (156.9)
Table I: SELF-REFINE results various tasks using GPT-3.5, CHATGPT, and GPT-4 base LLM.
SELF-REFINE consistently improves LLM. Metrics used for these tasks are defined in Section 3.2.
(such as for Code Optimization and Math Reasoning); otherwise, we created prompts as detailed in
Appendix V. We generate samples using temperature of 0.7.
3.2 Metrics
We report three types of metries:
Task specific metric: When available, we use automated metrics from prior work (Math Reasoning: % solve rate; Code Optimization: % programs optimized%)
GPT-4-pref: addition human-pref, we use GPT-4 as a proxy for human preference following prior work (Fu et al., 2023; Chiang et al., 2023; Geng et al., 2023; Sun et al., 2023), and found high correlation (82% for Sentiment Reversal, 68% for Acronym Generation, and 71 % for Dialogue Response Generation) with human-pref. For Code Readability Improvement, we prompt GPT-4 to calculate fraction of the variables that are appropriately named given the context (e.g.. = 0 input_ _uffeer = (]). Additional details are provided in Appendix F, For constrained generation, we combine automated evaluation to quantify concept coverage and GPT-4-pref to ensure the commonsense correctness of generated sentences. A sentence is only doemed a winner if it maintains validity in commonsense reasoning and has greater coverage in terms of concepts.
Human evaluation: Dialogue Response Generation, Code Readability Improvement, Sentiment Reversal, and Acronym Generation, we additionally perform blind human A/B evaluation subset of the outputs to select the preferred output. Additional details are provided in Appendix C.
3.3 Results
Table shows our main results:
SELF-REFINE consistently improves over base models across all model sizes, and additionally out- improves
performs the previous state-of-the--a across all tasks. For example, GPT-4+SELF-REES the base GPT-4 by 8.7% (absolute) in Code Optimization, increasing optimization percentage from 27.3% to 36.0%. Confidence intervals are provided in Appendix M. For code-based tasks, we found similar trends when using CODEX; those results are included in Appendix H.
One of the tasks in which we observe the highest gains compared to the base models is Constrained Generation, where the model is asked generate sentence containing up 30 given concepts. We believe that this task benefits significantly from SELF-REFING because there are more opportunities to miss some of the concepts the first attempt, and thus SELF-REFINE allows the model to fix these mistakes subsequently. Further, this task has an extremely large number of reasonable outputs, and thus SELF-REFING allows to better explore the space of possible outputs.
In preference-based tasks such as Dialogue Response Generation, Sentiment Reversal, and Acronym Generation, SELF-REFINE leads to especially high gains. For example in Dialogue Response Generation, GPT-4 preference score improve by 49.2% from 25.4% 74.6%. Similarly, we see remarkable improvements in the other preference- based tasks across all models.
5

---

The modest performance gains in Math Reasoning can be traced back to the inability to accurately identify whether there is any error. In math, errors can be nuanced and sometimes limited single line or incorrect operation. Besides, a consistent-looking reasoning chain can deceive M.I.MS to think that "everything looks good" (e-g. CHATGPT feedback for 94% instances is "everything looks good"). ) In Appendix K.1, we show that the gains with SELF- on Math Reasoning are much bigger (5++) an external source can identify if the current math answer is incorrect. Although SELF-REFINE demonstrates limited efficacy in Math Reasoning, we observe gains with SELF-REFINE in a subset Big-Bench Hard (Suzgun et al., 2022) tasks that typically require combination of commonsense reasoning and logic, such date reasoning (Appendix D). This suggests that SELF-REFINE may be more effective in scenarios where the interplay of logical analysis and knowledge acquired through pre- pre-training facilitates self- velf-verification..
Improvement is consistent across base LLMS sizes Generally, GPT-4+ REFINE performs better than TTT-3.5+SSEF-RREEEE where the initial base results of GPT-4 were lower than GPT-3.5 or CHATGGT. We thus believe that SELF-REFINE allows stronger models (such GPT-4) to unlock their full potential, even in cases where this potential is not expressed in the standard, single-pas, passe, output generation. Comparison to additional strong baselines provided in Appendix H.
and CHATTGPI-SEF-RRENNE across all tasks, even tasks
4 Analysis
The three main steps of SELF-REFINE are FEEDBACK, REFINE, and repeating them iteratively. In this section, we perform additional experiments analyze the importance of each of these steps.
Task
SELF-REFINS feedback
Generic feedback
No feedback
Code Optimization Sentiment Reversal
27.5
26.0
24.8
43.2
31.2
Acronym Generation
56.4
54.0
48.0
Table 2: Prompting to generate generic feedback (or having the model generate no feedback at
all) leads to reduced scores, indicating the importance of the FEEDBACK step of SELF-REFINE These experiments were performed with CHATGPT (Code Optimization and Sentiment Reversal) and GPT-3.5 (Acronym Generation), and metries used are defined in Section 3.2.
The impact of the feedback quality Feedback quality plays a crucial role in DELF-REFINE To quantify its impact, compare SELF-REFINE, which utilizes specific, actionable feedback, with two ablations: one using generic feedback and another without feedback (the model may still iteratively refine its generations, but is not explicitly provided feedback do so). For example, in the Code Optimization task: actionable feedback, such as Avoid repeated calculations in the for loop, pinpoints an issue and suggests a clear improvement. Generic feedback, like Improve the efficiency of the code, lacks this precision and direction. Table 2 shows feedback's 's clear influence.
In Code Optimization, performance slightly dips from 27.5 (SELF- REFINE feedback) to 26.0 (generic feedback), and further to 24.8 (no feedback). This suggests that while generic feedback offers some guidance specific, actionable feedback yields superior results.
This effect is more pronounced tasks like Sentiment Reversal, where changing from our feedback to generic feedback leads to significant performance drop (43.2 to 31.2), and the task fails without feedback. In the "No feedback" setting, the model was not given clear instructions on changing the output. We find that the model tends to either repeat the same output in each iteration to make unrelated changes. Since the scores in this task are the relative improvement increase in human preference, a score of 0 means that "No feedback" did not improve over the base model outputs. Similarly, in Acronym Generation, without actionable feedback, performance drops from 56.4 to 48.0, even with iterative refinements. These results highlight the importance of specific, actionable feedback in our approach. Even generic feedback provides some benefit, but the best results are achieved with targeted, constructive feedback.
How important are the multiple iterations of FREDRACK-REFINE Figure 4 demonstrates that on average, the quality of the output improves the number of iterations increases. For instance, in

---

15
14
13.5
I C. Opt.
| C. Gen
|| S. Rev.
Task
yo
3/1
1/2
1/3
10
C. Opt.
22.0
27.0
27.9
28.8
6.5
S. Rev.
33.9
34.9
36.1
36.8
5
C. Gen.
12.1
26.1
39.6
46.1
0.9
1.2
0.9
0.7
Agyo +y1)
A(y1+ty))
Ayy2+tya)
Figure 4: Left: Iteration-win wise score improvements. Early iterations significantly improve output quality, and scores generally keep improving with more iterations. Right: SELF-REFINE Perfor- mance improvements with iterations. Most gains(A) are in the initial iterations for both Code Opt. and Sentiment Reversal. The numbers are averaged over CHATGPT, GPT-3.5, and GPT-4. Task abbreviations: C. Opt. (Code Optimization), S. Rev. (Sentiment Reversal), C. Gen. (Constrained Generation).
Slower code def solve solve :
Faster code
best_price (amount 199) // 200
def solve (amount):
- 380
coins [200, 300]
First loop
prices [380, 550]
for in range (amount // 200 1):
dp [float ('inf') ')] (amount +
... nested loops
dp[0]
for c1 in range(amount // 1500 1):
for i in range (len
for in range (coins amount+1);
if a*200 + b+300 = amount:
dp[j] min dp
price 3+380 b+550
- coins [i]] prices[i]]
if price best_price:
return dp [amount]]
best-price price
return best_price
Figure 5: Comparison of code generated by Madaan al. (2023) (left) and the output after applying SELF-REFINE (right). The initial code by the baseline, which is nearly identical to the slower input program, fails improve the efficiency and merely alters the logic for reading input. SELF-REFINE first generates feedback that diagnoses that This code is slow because it is using six nested loops to iterate through all possible combinations of coins to pay the amount, and suggests that a more efficient approach would be .... SELF-REFINE then uses this feedback to generate the revised code (right), reducing the time complexity O(amount coins). The full example provided in Appendix K
the Code Optimization task, the initial output (yo) has three iterations (ya). Similarly, in the Sentiment Reversal task, the initial output has a score of 33.9, which increases to 36.8 after three iterations. This trend of improvement is also evident in Constrained Generation, where the score increases from 26.1 to 46.1 after three iterations. Figure highlights the diminishing returns in the improvement the number of iterations increases. Overall, having multiple FEEDBACK R-RFFINE iterations significantly enhances the quality of the output, although the marginal improvement naturally decreases with more iterations.
of 22.0, which improves to 28.8 after
The performance may not always monotonically increase with iterations: multi aaltit feedback tasks like Acronym Generation, where the output quality vary during iteration with improvement

---

in one aspect but decline in another aspect. To counter this, SELF- generates numerical scores for different quality aspects, leading to a balanced evaluation and appropriate output selection.
Can we just generate multiple outputs instead of refining? Does SELF-REFINE improve because of the iterative refinement, or just because generates more outputs? We compare SELF-REFINE with CHATGPT, when CHATGPT generates A = 4 samples (but without feedback and refinmentt. Then, we compare the performance of SELF-REFINE against these initial outputs in a 1 vs. i evaluation. In other words, we assess whether SELF-REIIEE can outperform all k initial outputs. The results of this experiment are illustrated in Figure (Appendix K). Despite the increased difficulty of the 1 vs. k setting, the outputs of SELF-REFINE are still preferred by humans over all k initial outputs. This shows the importance of refinement according to feedback over the alternative of just generating multiple initial outputs.
Does SELF-REFINE works in an instruction only setup? In our main experiments, we use few-shot prompting to guide model ouiput into a more readily parseable format. Next, we experiment with SELF-REFINE under a zero-shot prompting scenario, where traditional few-shot examples are supplanted by explicit instructions at each stage of the SELF- process. For these experiments, we use CHATGPT. The results (Appendix E in Table 8) show that SELF-REFINE remains effective across diverse tasks, even in the absence of example prompts. Notably, in tasks such as Acronym Generation and Sentiment Reversal, SELF-REFINE, under zero-shot prompting, enhances performance from 16.6% to 44.8% and 4.4% to 71.4%. respectively. However, achieving optimal performance in this setting requires extensive prompt engineering for instructions.
For Math Reasoning tasks, SELF-REFINE improves the solve rate from 22.1% to 59.0% in an instruction-only setting. We find that much of this gain comes from fixing omitted return statements in 71% of the initial Python programs, despite clear instructions to include them. Subsequent iterations of feedback generation and refinement address this issue effectively, decreasing the error rate by 19%. Further, we find that when the initial programs are valid, SELF-REFFI does not improve the performance.
Does SELF-REFINE work with weaker models? The experiments in Section 3.3 were performed with some of the strongest available models: does SELF-REFINE work with smaller or weaker models as well? To investigate this, we instantiated SELF-REFINE with Vicuma-13B (Chiang et al., 2023), less powerful base model. While Vicuna-13B capable of generating initial outputs, it struggles significantly with the refinement process. Specifically, Vicuna-13B was not able to consistently generate the feedhack in the required format. Furthermore, even when provided with Oracle or hard-coded feedback, it often failed to adhere to the prompts for refinement. Instead of refining its output, Vicuna-133 13B either repeated the same output or generated a hallucinated conversation, rendering the outputs less effective. Example output and analysis is provided in Appendix I.
How does SELF-REFINE perform with strong open access models like 2TAMM22-70B? We conduct additional experiments using SEL.- on LLama-2 (Touvron et al., 2023), a state-of-the- art, open-access language model on Acronym Generation, Sentiment Reversal, Dialogue Response Generation, and Math Reasoning. Consistent with our primary findings, SELF-REFINE shows an improvement across all these tasks relative to the base model. The full results are shown in Appendix J. These promising results with LLAMA2-70B suggest that the applicability of SELF-REFINE might extend to a wide array of increasingly powerful open-source models in the future
Qualitative Analysis We conduct a qualitative analysis of the feedback generated by SEI.-RRFIIN and its subsequent refinements. We manually analyze 70 samples in total (35 success cases and 35 failure cases) for Code Optimization (Madaan et al., 2023) and Math Reasoning (Cobbe et al., 2021). For both Math Reasoning and Code Optimization, we found that the feedback was predominantly actionable, with the majority identifying problematic aspects of the original generation and suggesting ways to rectify them.
When SELF-REFINE failed to improve the original generation, the majority of issues were due to erroneous feedback rather than faulty refinements. Specifically, 33% of unsuccessful cases were due to feedback inaccurately pinpointing the error's location, while 61 % were result of feedback suggesting an inappropriate fix. Only 6% of failures were due to the refiner incorrectly implementing good feedback. These observations highlight the vital role of accurate feedback plays in SELF-REFINE
8

---

Supervision-
Supervision- free feedback
Multi-aspes feedback
Iterative
free refiner
Learned refiners: PEER (Schick et al., 2022b), Self-critique (Saunders ot al., 2022b), CodeRL (Le ct al., 2022b), Self-correction (Welleck et al., 2022).
X
or
Prompted refiners: Augmenter (Peng et al., 2023), Re (Yang al., 2022), Reflexion (Shinn et al., 2023).
X
X
X
SELF-REFINE (this work)
Table 3: comparison of SELF-REFIN to closely related prior refinement approaches.
In successful cases, the refiner was guided by accurate and useful feedback to make precise fixes to the original generation in 61% % of the cases. Interestingly, the refiner was capable of rectifying issues even when the feedback was partially incorrect, which was the situation in 33% of successful cases. This suggests resilience to sub-optimal feedback. Future research could focus on examining the refiner's robustness to various types of feedback errors and exploring ways to enhance this resilience. In Figure 5, we illustrate how FEEF-E significantly improves program efficiency by transforming a brute force approach into dynamic programming solution, result of insightful feedback. Additional analysis on other datasets such as Dialogue Response Generation is provided Appendix K.
Going Beyond Benchmarks While our evaluation focuses benchmark tasks, SELF- RER-EE is designed with broader applicability mind. We explore this real-world use case of website gen- eration, where the user provides high-level goal and RELF-REFINE assists in iteratively developing the website. Starting from a rudimentary initial design, SELF-REFHS refines HTML, CSS, and JS to evolve the website in terms of both usability and aestheties. This demonstrates the potential of REFFNE in roal-world, complex, and creative tasks. See Appendix L for examples and further discussion, including broader, societal impact of our work.
5 Related work
Leveraging human- and machine- -egeeraedd natural language (NL) feedback for refining outputs has been effective for variety of tasks, including summarization (Scheurer et al., 2022), script generation (Tandon et al., 2021), program synthesis (Le et al., 2022a; Yasunaga and Liang, 2020), and other tasks (Madaan et al., 2022; Bai et al., 2022a; Schick et al., 2022b; Saunders et al., 2022a; Bai et al., 2022b; Welleck et al., 2022). Refinement methods differ in the source and format of feedback, and the way that refiner is obtained. Table summarizes some related approaches; Appendix B for an additional discussion.
Source of feedback. Humans have been effective source of feedback (Tandon et al., 202 1; Elgohary et al., 2021; Tandon et al., 2022; Bai et al., 2022a). Since human feedback is costly, several approaches a scalar reward function as surrogate of (or alternative to) human feedback (c.g.. (Bai et al., 2022a; Liu et al., 2022; Lu et al., 2022; Le et al., 2022a; Welleck et al., 2022)). Alternative sources such as compilers (Yasunaga and Liang, 2020) Wikipedia edits (Schick al., 2022b) can provide domain- domain-specific feedback. Recently, LLMs have been used to generate feedback for general domains (Fu et al., 2023; Peng al., 2023; Yang al., 2022), However, ours is the only method that generates feedback using an LLM on its own output, for the purpose of refining with the same LLM.
Representation of feedback. The form of feedback can be generally divided into natural language (NL) and non-NL feedback. Non-NL feedback can come in human human- provided example pairs (Dasgupta et al., 2019) or scalar rewards (Liu et al., 2022; Le et al., 2022b). In this work, use NL feedback, since this allows the model to easily provide self relf-feedback using the same LM that generated the output, while leveraging existing pretrained LLMs such GPT-4.
Types of refiners. Pairs of feedback and refinement have been used to learn supervised refiners (Schick et al., 2022b; Du et al., 2022; Yasunaga and Liang, 2020; Madaan et al., 2021). Since

---

gathering supervised data is costly, some methods learn refiners using model generations (Welleck al., 2022; Peng al., 2023). However, the refiners are trained for each new domain. Finally, (Yang et al., 2022) use prompted feedback and refinement specifically tailored for story generation. In this work, we avoid training separate refiner, and show that the same model can be used as both the refiner and the source of feedback across multiple domains.
Non-refinement reinforcement learning (RL) approaches. Rather than having explicit refinement, an alternative way to incorporate feedback is by optimizing scalar reward function, e-g. with reinforcement learning (e.g., Stiennon et al. (2020); Lu et al. (2022); Le al. (2022a)). These methods differ from SELF-REFIN in that the model does not access feedback Second, these RL methods require updating the model's parameters, unlike SELF- REFFNE. Recently, in discrete-space setereett--pcceeeeeee simulated environments, LLMs have also been shown to iteratively shape and refine rewards and policies, thereby performing RL tasks without expert demonstrations or gradients (Kim al., 2023; Brooks et al., 2023). While we focus real-world code and language tasks in this paper, would be interesting to explore applications of self-refine in simulated environments.
intermediate generation.
Limitations and Discussion
The main limitation of our approach is that the base models need to have sufficient few-shot modeling or instruction-followin abilities, in order to learn to provide feedback and to refine in an in-context fashion, without having to train supervised models and rely on supervised data.
Further, the experiments in this work were primarily performed with language models that are not open-sourced, namely GPT-3.5, CHATGPT, GPT-4, and CODEX. Existing literature (Ouyang et al., 2022) does not fully describe the details of these models, such as the pretraining corpus, model sizes, and model biases. Nonetheless, of our work. In addition, initial results from our experiments with the open-access ILLAMA2-70B language model are promising, reinforeing the notion that SELF-REFINE has the potential to be widely applicable, even as open-source models continue to evolve and improve.
release our code and model outputs ensure the reproducibility
Another limitation of our work is that we exclusively experiment with datasets in English. In other languages, the current models may not provide the same benefits. Finally, there is possibility for bad actors to use prompting techniques to steer model to generate more toxic or harmful text. Our approach does not explicitly guard against this.
7 Conclusion
We present SELF-REFINE a novel approach that allows large language models to iteratively provide self-feedback and refine their own outputs. SELF-REFINE operates within single LLM, requiring neither additional training data nor reinforcement learning. We demonstrate the simplicity and ease of use of SELF-REFINE across a wide variety of tasks. By showcasing the potential of SELF-REFINE in diverse tasks, our research contributes to the ongoing exploration and development of large language models, with the aim of reducing the cost of human creative processes in real-world settings. We hope that our iterative approach will help drive further research in this area. To this end, we make all our code, data and prompts available at https: h/spp:://///feeeeee ine. intoo..fenn.........iiiio
Acknowledgements
We thank the anonymous reviewers for their useful comments and suggestions, and extend our thanks to James Laudon, Cliff Young, Graham Neubig, Daniel Fried, Rif A. Saurous, and the Google DeepMind team for their valuable feedback and comments. This material is partly based on research sponsored in part by the Air Force Research Laboratory (agreement number FA8750-19-22 19-2-0200). The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed implied, of DARPA the U.S. Government.
10

---

References
Teresa M. Amabile. 1983. A Theoretical Framework. The Social Psychology of Creativity, pages 65-96. Springer New York, New York, NY.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022a. Training helpful and harmless assistant with reinforcement learning from human feedback. ArXiv:2204.0586..
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022b. Constitutional ai: Harmlessness from feedback. arXiv preprint 12.00:22
Emery D Berger, Sam Stern, and Juan Altmayer Pizzomo. 2022. Triangulating Python Performance Issues with SCALENE. ArXiv preprint, abs/2212.075597.
Ethan Brooks, Logan Walls, Richard L. Lewis, and Satinder Singh. 2023. Large language models can implement policy iteration.
Lawrence D Brown, T Tony Cai, and Anirban DasGupta. 2001. Interval estimation for a binomial proportion. Statistical science, 16(2):101-133.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901, Online. Curran Associates, Inc.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage. Mira Murati, Katic Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. 2021. Evaluating Large Language Models Trained on Code. arXiv preprint arXiv:2107.03337.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang. Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2 10. /4168.
Sanjoy Dasgupta, Daniel Hsu, Stefanos Poulis, and Xiaojin Zhu. 2019. Teaching a black-box learner. In Proceedings of the 36th International Conference Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 1547-1555. PMLR
Wanyu Du, Zae Myung Kim, Vipul Raheja, Dhruv Kumar, and Dongyeop Kang. 2022. Read, revise, repeat: A system demonstration for human-in in-the-loop iterative text revision. In Proceedings of the First Workshop on Intelligent and Interactive Writing Assistants (In2Writing 2022), pages 96-108, Dublin, Ireland. Association for Computational Linguistics.
Ahmed Elgohary, Christopher Meek, Matthew Richardson, Adam Fourney, Gonzalo Ramos, and Ahmed Hassan Awadallah. 2021. NL-EEDTT Correcting semantic parse emrors through natural language interaction. In Proceedings of the 2021 Conference of the North American Chapter of the
11

---

Association for Computational Linguistics: Human Language Technologies, pages 5599-56114 Online. Association for Computational Linguisties.
Linda Flower and John R Hayes. 1981. A cognitive process theory of writing. College composition and communication, 32(4):365-387.
Jinlan Fu, See-Kiong Ng. Zhengbao Jiang, and Pengfei Liu. 2023. Gptscore: Evaluate as you desire. arXiv preprint arXiv:2302.04166.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2022. Pal: Program-aided language models. arXiv preprint urXiv:2211114435.
Xinyang Geng, Arna Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel. Sergey Levine, and Dawn Song. 2023. Koala: A dialogue model for academic research. Blog post.
Geunwoo Kim, Pierre Baldi, and Stephen McAleer. 2023. Language models can solve computer tasks. arXiv preprint 230.::::00317499
Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven C. H. Hoi. 2022a. CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learn- ing.
Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven C. H. Hoi. 2022b. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. ArXiv, abs/2207.01780)
Juncen Li, Robin Jia, He He, and Perey Liang. 2018. Delete, retrieve, generate: a simple approach to sentiment and style transfer. In Proceedings of the 2018 Conference the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1865-1874, New Orleans, Louisiana. Association for Computational Linguistics.
Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. 2020. CommonGen: A constrained text generation challenge for generative commonsense reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1823-1840, Online. Association for Computational Linguistics.
Jiacheng Liu, Skyler Hallinan, Ximing Lu, Pengfei He, Sean Welleck, Hannaneh Hajishirzi, and Yejin Choi. 2022. Rainier: Reinforced knowledge introspector for commonsense question answering. In Conference on Empirical Methods in Natural Language Processing.
Ximing Lu, Sean Welleck, Liwei Jiang, Jack Hessel, Lianhui Qin, Peter West, Prithviraj Am- manabrolu, and Yejin Choi. 2022. Quark: Controllable text generation with reinforced unlearning. ArXiv, abs/2205.13636
Aman Madaan, Alexander Shypula, Uri Alon, Milad Hashemi. Parthasarathy Ranganathan, Yiming Yang, Graham Neubig, and Amir Yazdanbakhsh. 2023. Learning iefffrmannc-imprrvvinni code edits. arXiv preprint 072.022302.07767.
Aman Madaan, Niket Tandon, Peter Clark, and Yiming Yang. 2022. Memory-assisted prompt editing to improve gpt-3 after deployment. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2833-2861
Aman Madaan, Niket Tandon, Dheeraj Rajagopal, Peter Clark, Yiming Yang, and Eduard Hovy. 2021. Think about it! improving defeasible reasoning by first modeling the question scenario. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pages 101,, Online and Punta Cana, Dominican Republic, Association for Computational Linguisties.
Shikib Mchri and Maxine Eskenazi. 2020. Unsupervised evaluation of interactive dialog with DiT In Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 235, 1st virtual meeting. Association for Computational Linguistics.
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. Codegen: An open large language model for code with multi-turn program synthesis. ArXiv preprint, ab</2203.13474.
12

---

OpenAI.
Model index for researchers. https://platfo: openai com/ Accessed: May 14, 2023.
OpenAl. 2022. Model index for researchers. Blogpost.
OpenAl. 2023. Gpt-4 technical report.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models follow instructions with human feedback. ArXiv:2203.02155.
Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, and Jianfeng Gao. 2023. Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback.
Shrimai Prabhumoye, Yulia Tsvetkov, Ruslan Salakhutdinov, and Alan W Black. 2018. Style transfer through back translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 866-876, Melbourne, Australia. Association for Computational Linguistics.
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. 2022. Measur- ing and narrowing the compositionality gap in language models. arXiv preprint arXiv:2210.03350
Ruchir Puri, David Kung. Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladmir Zolotov, Julian Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, Veronika Thost, Luca Buratti, Saurabh Pujar, Shyam Ramji, Ulrich Finkler, Susan Malaika, and Frederick Reiss. 2021. Codenet: large-scale ai for code dataset for learning diversity of coding tasks. arXiv preprint arXiv:2105.
Machel Reid and Graham Neubig. 2022. Learning to model editing processes. arXiv preprint wwXiv:2205.12374.
William Saunders, Catherine Ych, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. 2022a. Self-critiquing models for assisting human evaluators.
William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. 2022b. Self-critiquing models for assisting human evaluators. ArXiv:2206.05802.
Jérémy Scheurer, Jon Ander Campos, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. 2022. Training language models with natural language feedback. ArXiv:2204114146
Timo Schick, Jane Dwivedi- Yu, Zhengbao Jiang, Fabio Petroni, Patrick Lewis, Gautier Izacard, Qingfei You, Christoforos Nalmpantis, Edouard Grave, and Sebastian Riedel. 2022a. Peer: A collaborative language model.
Timo Schick, Jane Dwivedi- Yu, Zhengbao Jiang, Fabio Petroni, Patrick Lewis, Gautier Izacard, Qingfei You, Christoforos Nalmpantis, Edouard Grave, and Sebastian Riedel. 2022b. Peer: collaborative language model. ArXiv, abs/2208.1 1663.
Noah Shinn, Beck Labash, and Ashwin Gopinath. 2023, Reflexion: autonomous agent with dynamic memory and self-reflection.
Herbert A. Simon. 1962. The architecture complexity. Proceedings of the American Philosophical Society, 10666):467-42.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. In Advances in Neural Information Processing Systems, volume 33, pages 3008-3021. Curran Associates, Inc.
Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. 2023. Principle-driven self-alignment of language models from scratch with minimal human supervision. arXiv preprint ar5.v:2305.03047.
13

---

Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, al. 2022. Challenging big-bencd tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210002661.
Niket Tandon, Aman Madaan, Peter Clark, Keisuke Sakaguchi, and Yiming Yang. 2021. Interscript: A dataset for interactive learning of scripts through error feedback. arXiv preprint arXiv:2112.07867.
Niket Tandon, Aman Madaan, Peter Clark, and Yiming Yang. 2022. Learning to repair: Repairing model output errors after deployment using dynamic memory of feedback. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 339-352.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Amm Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.1397)
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou. 2022. Chain of Thought Prompting Elicits Reasoning Large Language Models. arXiv preprint arXiv:2201.11903.
Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. 2022. Generating sequences by learning to self-correct. arXiv preprint arXiv:2211.00m 1201111.00.33
Kevin Yang, Nanyun Peng, Yuandong Tian, and Dan Klein. 2022. Re3: Generating longer stories with recursive reprompting and revision. In Conference Empirical Methods in Natural Language Processing.
Michihiro Yasunaga and Percy Liang. 2020. Graph- based, self-supervised program repair from diagnostic feedback. 37th Int. Conf. Mach. Learn. ICML 2020, Pur1F168144-114::73300000000
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28.
14

---

A
Evaluation Tasks
Table 4 lists the tasks in our evaluation, and examples from each task.
Task and Deseription
Sample one iteration of FEEDBACK-REFINE
Sentiment Reversal
2: The food was fantastic..."
Rewrite reviews reverse sentiment.
V/: The food disappointing...
Dataset: (Zhang et al., 2015) 1000 review pas-
fb: Increase negative sentiment
sages
M++1 The food was utterly terrible..."
Dialogue Response Generation
w: What's the best way to cook pasta?"
Produce rich conversational responses. Dataset: (Mchri and Eskenazi, 2020) 372 conv.
y:: The best way to cook pasta is to..."
fb: Make response relevant, engaging, safe
M++ Boil water, add salt, and cook pasta..."
Code Optimization
Nested loop for matrix product
Enhance Python code efficiency Dataset: (Madaan al., 2023): 1000 programs
y/ NumPy dot product function
fb: Improve time complexity
M++1: Use NumPy's s optimized matmul function
Code Readability Improvement
Unclear variable names, no comments
Refactor Python code for readability.
y: Descriptive names, comments
Dataset: (Puri et al., 2021) 300 programs"
fb: Enhance variable naming; add comments
1++1: Clear variables, meaningful comments
Math Reasoning
2: Olivia has $23, buys bagels at $3 each"
Solve math reasoning problems. Dataset: (Cobbe et al., 2021) 1319 questions
14 Solution in Python
fb: Show step-by-step solution
1++1: Solution with detailed explanation
Acronym Generation
x : Radio Detecting and Ranging"
Generate acronyms for given title
V4 RDR
Dataset: (Appendix T) 250 acronyms
fb be context relevant; easy pronunciation
M++1: RADAR"
Constrained Generation
beach, vacation, relaxation
Generate sentences with given keywords. Dataset: (Lin et al., 2020) 200 samples
y: During our beach vacation...
fb: Include keywords; maintain coherence
M++1: beach vacation was filled with relaxation
Table 4: An overview of the tasks which evaluate SELF-REANE on, along with their associated datasets and sizes. For every task, we demonstrate a single iteration of refinement of input x, the previously generated output yr, the feedback generated fb, and the refinement 7/+1- Few-shot prompts used for FEEDBACK and REFINE are provided in Appendix V.
15

---

B Broader Related Work
Compared a concurrent work, Reflexion (Shinn al., 2023), our approach involves correction using feedback, whereas their setup involves finding the next best solution in planning using ReAct. While ReAct and Reflexion provide a free-form reflection on whether a step was executed correctly and potential improvements, our approach is more granular and structured, with muuli-dimensione. feedback and scores. This distinction allows our method to offer more precise and actionable feedback, making it suitable for a wider range of natural language generation tasks, including those that may not necessarily involve step-by-step planning such as open-ended dialogue generation.
Comparison with Welleck et al. (2022) The closest work to ours may be Self- ceff-Coreection (Welleck et al., 2022); however, Self-Correction has several disadvantages compared to SELF-REFINE:
1. Self-Correction does not train their model to generate explicit feedback; instead, Welleck et al. (2022) trained their models to refine only. As we show in Section 4 and Table 2, having the model generate explicit feedback results in significantly better refined outputs.
2. Self-Correction trains separate refiner (or "'correcto") for each task. In contrast, SELF- REFINE uses instructions and few-shot prompting, and thus does not require training a separate refiner for each task.
3. Empirically, evaluated REFFNE using the same base model of GPT-3 as Self- Correction, and with the same settings on the GSM8K benchmark. Self-Correction achieved 45.9% accuracy while SELF-REFINE (this work) achieved 55.7% (19.8).
Comparison with non-refinement reinforcement learning (RL) approaches. Rather than having an explicit refinement module, an alternative way incorporate feedback is by optimizing scalar reward function, c.g. with reinforcement learning (e.g., Stiemnon et al. (2020); Lu et al. (2022); Le al. (2022a)). These methods differ from RELF-REFINE (and more generally, refinement-based approaches) in that the model cannot access feedback on an intermediate generation. Second, these reinforcement learning methods require updating the model's parameters, unlike SELF-REFINE
See Table for an additional detailed comparison related work.
Method
Novelty optinise
zenefew shat improvenent
mntti oritics
NL feedback lncalication
Itenative framwoek
RIIE (Stiemos
Onnnne feedFack
self
2002) gre RL. d stall ho0ng
lage quantizaion ciit grsmsiunee
Oninee end
Xlaosl,
X trirral end took
singkowallr soom)
X (élmu eignal)
Iroin time herr)
Oade Bt. (locct 20033)
acur crine Rt. fir cnds in- provenent
X triinal end taok
singlciunii toaa)
Xokme sigroll)
DiRepatr Visangg Linng,
Coopller feedbuck to Excly nquair
X trained sup.
X siepleicompile msg)
C self
PEER (Slack 202263
dec. cáit cained on wiki calia
Xniinal on calta
sippkiiacsuaay)
K not sclf gan
Salf enique (Sandde 2012a)
low shat cniique geseration nevil maining of commeter
Xeodback trainig
sipllshumaz)
K Self
Selficommet Welbeck or at, 3015))
X trained on end tock
X singln (mak sprcitic
X limited wirting
Imited setting)
Conel All (lai at, 201269
min AL on antonne (cn-
X critigue triining
2 Boou sur)
uque, revision) pir
Self ask (Press of 2022)
ask tollowup ques when in- term ons connectiinel uning
L tew sbot
X note
Xinoe)
GIT3 ware dPu ot 2023)
GPT can soore gannndime with instruction
Ci dist
X singletsingle uillty (a)
Xisrrci
Angmenter (Phing et al., 3015)
cemuiity Sedfbock from exte-
Niow shet
X singlecchcttaling
self gen)
(Vong ot 2020g
unc donein,
D ow dhei
triinedd errics)
K not self gan)
ininnd critics
SELF-REINE
lowabol itenaiie supazi
Ciow
@ maliiplotiis cnitio)
gmn))
Table 5: Summary of related approaches. Reinforcement learning approaches are shown in purple
trained corrector approaches are shown in orange and few-shot corrector approaches are shown in green
16

---

C Human Evaluation
The A/B evaluation our study was conducted by the authors, where human judge was presented with an input, task instruction, and two candidate outputs generated by the baseline method and SELF-REFINE. The setup was blind, i.e., the judges did know which outputs were generated by which method. The judge was then asked to select the output that is better aligned with the task instruction. For tasks that involve A/B evaluation, we calculate the relative improvement as the percentage increase in preference rate. The preference rate represents the proportion of times annotators selected the output produced by SEFF-REFINE over the output from the baseline method. Table 6 shows the results.
While multiple annotators participated in each task, we only collected single annotation per instance, aiming to scale the number of data points we could annotate. To further validate the reliability of our evaluations, we obtained two annotations for 50 samples from each dataset. All human evaluations were executed in double-blind manner, with the responses being randomly flipped to ensure annotator impartiality, leaving them unaware of whether a given output was from the base model or the SELF-REFINE, These additional evaluations were exclusively applied to outputs generated by GPT-4.
For each task, we measured inter-labeler agreement using Cohen's Kappa score. Code Readability Improvement and Acronym Generation both scored substantial 0.75, Sentiment Reversal was also substantial at 0.61, while Dialogue Response Generation was moderate with a score of 0.53.
Task
SELF-REFINE (%)
Direct (%)
Either (%)
Sentiment Reversal
75.00
21.43
3.57
Acronym Generation
44.59
12.16
43.24
Dialogue Response Generation
47.58
19.66
32.76
Code Readability Improvement
50.0
3.0
47.0
Table 6: Relative improvement of SELF- in A/B evaluations across different tasks. The values
represent normalized preferences, which correspond to the proportion of times the output generated by SELF-REFINE
selected better aligned with the task instruction over the baseline method.
The evaluation was conducted for 150 examples for each dataset. The judges were not aware of the method that generated each sample.
17

---

D SELF-REFINE on Reasoning Tasks
SELF-REFINE shows limited success in Math Reasoning tasks, mainly due to its challenges in generating meaningful feedback. Nonetheless, we observe performance gains with SELF-REFINE in certain tasks from the Big- Big-Bench Hard suite (Suzgun et al., 2022), particularly those requiring commonsense reasoning and logical thinking. These results in Table suggest that SELF-REFING is more effective in scenarios where the combination of logical reasoning and pre-trained knowledge contribute self-verification.
Task
Base model
+Self-Refine
Gain
Date Understanding
62.0
66.8
4.8
Geometric Shapes
17.6
20.0
2.4
Logical Deduction (seven objects)
43.2
45.2
2.0
Multi-Step Arithmetic
61.6
64.0
2.4
Tracking Shuffled Objects (seven objects)
31.6
36.0
4.4
Table Performance comparison between the base outputs and the SELF-REFINE enhanced model across various Big Bench Hard tasks. All experiments were conducted using the GPT-3.5-TURBO-
0613 model with temperature setting of 0.0. No task-specific prompts were utilized; all tasks employed the same instructional prompts.
E Instruction-Only Prompting
In our main experiments, we use few-shot prompting to guide model output into more readily parseable format. Next, we experiment with BELF-REFINE under a zero-shot prompting scenario, where traditional few-shot examples are supplanted by explicit instructions at each stage of the SELF-REFINE process. For these experiments, we use CHATGPT. The results in Table 8) show that SELF-REFINE remains effective across diverse tasks. However, achieving optimal performance in this setting requires extensive prompt engineering for instructions. The instructions are present in Listing 1 (Acronym Generation). Listing 3 (Math Reasoning), Listing 2 (Sentiment Reversal), Listing 4 (Constrained Generation), and Listing (Dialogue Response Generation).
For Math Reasoning tasks, SELF-REFINE improves the solve rate from 22.1% to 59.0% in an instruction-only setting. We find that much of this gain comes from fixing omitted return statements in 71% of the initial Python programs, despite clear instructions iterations of feedback generation and refinement address this issue effectively, decreasing the error rate by 19%. Further, we find that when the initial programs are valid, SELF-REFIN does not improve the performance.
include them. Subsequent
Task
Base
SELF-REFINE (zero-shot)
Acronym Generation
16.6%
44.8% (1 28.2%)
Constrained Generation
4.0%
47.0% (1 43.0%)
Sentiment Reversal
4.4%
71.4% (t 67.0%)
Math Reasoming
22.1% 1%
59.0% (1 36.9%)
Dialogue Response Generation
23.0%
48.8% (1 25.8%)
Table 8: Performance of SELF-REFINE with Zero-Shot Prompting
F GPT-4 Evaluation
In light of the impressive achievements of GPT-4 in assessing and providing reasoning for complex tasks, we leverage its abilities for evaluation in SELF-REFINE. The approach involves presenting tasks to GPT-4 in structured manner, promoting the model's deliberation on the task and generating a rationale for its decision. To mitigate order bias our tasks, we randomly flip the order of the outputs generated after the first iteration and by SELF-REFINE REFINE before evaluation. Further, to ensure
18

---

Listing Instruction-only prompts used at various stages of the SELF-REFINE process for Acronym Generation.
Inst: Generate acronym for given title start_chat_
("role": "system", "content"; 'You are a helpful assistant that generates acronyms. '), ("role": "user", "content"; f'Generate acronym for the title "(title)", Please reapond in the format: "The generated acronym (acronym)"") "
]
Generate Feedback: Evaluate the quality of the generated acronym feedback_str f"""Evaluate the acronym "(acronym)" for the title "(title)" its - ease pronunciation, ease spelling, relation to the title, positive connotation, and being well-knoum." "un
+
start_ start_chat_log
("role": "system", "content"; 'You are AI model that provides feedback the viability and quality of acronyms. {"role": "user", "content": feedback_str)
]
Refíne: Improve the acronym based provided feedback
start_chat_
{"role": "system", "content": 'You are a helpful assistant that can + iteratively improve acronyma based on feedback. '}, ("role": "user", "content": f' Improve the acronym "(acronym)" for the title "(title)" based the following feedback: (feedback). Please *always* respond the format: "The improved acronym is: (acronym)"")
]
Listing 2 Instruction-only prompts used at various stages of the Self-Refine process for Sentiment Reversal.
Inst: Transform a negative centiment review into posi one
start_che start_chat_log
("role": "system", "content"; 'You are AI language model that transforms a negative sentiment review into positive one.'},
["role": "user", "content": F'Please transform the following negative review 4 into a positive one: "(revieu)", Respond the format of: "The positive review is: [Your Response Here] ", }),
]
Generate Feedback: Provide feedback
transformed review
Start_chat__ae
("role": "system", "content": 'You are an AI model providing feedback on sentiment transformed review. '),
("role": "user", "content"; f f"Why this review Very positive? Point
~ specific issues and give concrete suggestions. Respond in the format of: 'Feedback: [Specific issues and suggestions]",
]
Refine: Improve the review based provided feedback
start_chat_110
("role": "system", "content": 'You are an AI model that improves upon an existing review based on provided feedback. "},
("role": "user", "content"; f'Please improve the sentiment of the following review "[sentence)" based on this feedback: "[feedback)", and make it more positive. Always reapond in the format of: "The more positive review is: [Your Response Here]".'K
]
19

---

Listing 3 Instruction-only prompts used at various stages of the Self-Refine process for Math Reasoning.
Punction to Generate an Answer
templaateee = "You are helpful assistant that responds with only python + program."
gen_user_ template gen_user_template """Write python function that solves the given question returns the result.
Always store the final result in variable called 'result", and always include 'return result" your last statement.
Question: {question) solution using Python:"""
Punction to Get Feedback the Answer tb_sys_template "You helpful assistant that provides feedback the + correctness python programa." """Question: (question)
fb ib_user_template {prediction)
There may be 4 question.
the code above because of lack of understanding of the
To find the error, go through semantically complete blocks of the code, and check + if everything looks good.
Errors could include:
Logical issues
Lack understanding the question
Missing "return result"
Share your evaluation in the format:
Evaluation: <your evaluation here>. If everything looks good,
'Evaluation: correct!'nn
Function to Fiz the Error sefine_sys_template "You are helpful assistant that responds with only python program." " refine_usee__tempaaee """Question: {question)
(prediction)
# There is an error in the code above. The following is the feedback on the code: (feedback)
Fix the in the python function given the feedback above. The python function 4 should return the final answer. ann
that there no inherent bias GPT-4 picking its own predictions, we conduct an additional study with CLAUDE-V2.
Claude-2 as the Evaluator Despite the measures we took to prevent any inherent biases, GPT-4 might inherently favor self-refined outputs. To provide comprehensive evaluation of GPT-4, we conduct an additional analysis using CLAUDE-V2, serving as an independent evaluator for GPT-4 outputs. The results Table from GPT-4 as the base LLM with CLAUDE- v2 the evaluator show the same strong preferences for SELF-REFINE over the Base outputs.
Task
% Base
REFINR
Dialogue Response Generation
30.6
64.7 (134.1)
Sentiment Reversal
10,6
69.2 (158.6)
Acronym Generation
32.0
49.2 (11.
Code readability
37.0
60.0 (123.0)
Table Evaluation Results of GPT-4 with CLAUDE-V2 as Evaluator
This prompts used for evaluation are listed in Listings to
http://wwwaaaarrrr.ccccoom////// wttp://e..ccccoc////teeeeeccceet 22
20

---

Listing Instruction-only prompts used at various stages of the SELF-REFINE process for Constrained Generation.
Constrained Gemeration Task
1. Inst
start_ start_chat_log -
("role": "system", "content"; 'You are AI model that generates sentences with commonsense, using given of concepta. '), ("role": "user", "content": G'Generate commonsense sentence using the following concepta: {", ".join(concepta)). Please reapond in the format: "The generated sentence is: (sentence)"")),
]
2. Get Feedback
start_che Seart_chat_log =
("role": "system", "content": 'You are AI model that provides feedback sentence generated with specific concepta. '),
{"role": "user", "content": 'Evauute the following sentence "(sentence)"" + which was meant to use the following concepts: {" ".join(concepts))
Please provide two pieces of feedback. Start your feedback about concept usage
with the phrase "Concept feedback is:", and start your feedback about commonsense facts with the phrase "Commonsense feedback is:".
The format should be:
Concept feedback is: <list of missing concepta>
Commonsense feedback is: <commonsense feedback here>"'>,
]
3. Iterate Fiz
start_chat__t =
("role": "system", "content"; 'You are AI model that improves upon existing sentence based on provided feedback. '},
("role": "user", "content"; f' I'Ipoov upon the following sentence
Sinprovee_seenencee based on the following feedback: [feedback_string]
Please respond in the format: "The improved sentence is: [Your improved sentence here] "'}
]
G
Model Key
We use terminology here: https://platff www.ai.enn.....oooo... cotiaaaicco/ooo//////a/a hotp///////c/c////////ggg----- The ex- periments were done with the 0313 versions of GPT-4 and CHATGPT unless otherwise mentioned. We refer to text-davinci-003 experiments and Constrained Generation experiments.
GPT-3.5. We GPT-3.5-TURBO-0613 for all instruction-only
21

---

Listing 5 Instruction-only prompts used various stages of the Self-Refine process for Dialogue
Response Generation.
Dialogue Response Generation Task
1. Cenerate Reaponse
start_ start_chat_log =
{"role": "system", "content": 'You helpful assistant that generates
4 responses.
("role": "user", "content"; P'rovidded a dialogue between two speakers, generate response that is coherent with the dialogue history. Desired traits for responses are: 1) Relevant The response addresses the context,
2) Informative The response provides some information, 3) Interesting
The response is interesting, 4) Consistent The response is consistent with the rest of the conversation in terms of tone and topic, 5) Helpful suggesting any actions, 6) Engaging The reaponse is engaging and encourages further conversation, 7) Specific The response contains specific content, Safe The response should not cause danger, risk, or injury 9) User understanding The response demonstrates an understanding of the user\'s input and state of mind, and 10) Fluent. Response should begin with Response: Response:\n/nConmee wtw.:e/aaaaaaaaaaaiiiio. history: Sestory:\nn(hhiattoo hnloopooseee '),
The response is helpful in providing any information
]
2. Get Feedback
start_chat_log
("role": "system", "content": 'You are AI model that provides feedback + the viability and quality of responses. '),
("role": "user", "content"; feedback_str)
]
3. Iterate Response
start_ start_chat_log
("role"; "system", "content"; 'You are helpful assistant that can iteratively improve responses based feedback. '), ("role": "user", "content": iterate_strr str]
]
Listing 6 Prompt for GPT-4 evaluation of Sentiment Reversal.
E"""Which review is aligned with the sentiment (target_semtiment)) Review (review_a) Review B: (review_b).
Pick your answer from ['Review A', 'Review B', 'both', 'neither'], Generate short explanation for your choice first. Then, generate 'The aligned review is A' or 'The more aligned review or 'The more aligned review is c both' or 'The more aligned review is neither'.
<
Format: <explanation> <answer> STOP
22

---

Listing 7 Prompt for GPT-4 evaluation of Acronym Generation.
f"""Title: (title)
Acronym A: {acronym_a)
Acronym B: {acronym_b)
Pick the better acronym for the given title. The acronyms should compared
+ on the following criteria:
Ease of pronunciation.
Ease of spelling.
Relation title.
Positive connotation.
Generate your answer in the following format:
<Short explanation>, The better acronym is The better acronym is The + acronyme are equally good OR Neither acronym good. STOP.
Listing Prompt for GPT-4 evaluation of Dialogue Response Generation.
""""Which response is better given this context: (context))? Response (response_a)
Response B: (response_be b).
Pick your answer from ('Response A', 'Response B', 'both', 'neither'], Generate a .> short explanation for your choice first. Then, generate 'The better response is A' 'The better response is B' or 'The better response is both' or 'The better response is neither'.
Format: <explanation> <answer> STOP
Listing 9 Prompt for GPT-4 evaluation of Constrained Generation
W"n"Which story is better?
Story A: (story_a)
Story B: (story_b).
Judge the story based the flow, the grammar, and the overall quality the story. Rate more realistic story higher. Pick your anawer from ['Story A',
+
+ 'Story B', 'either'), First, reason about your choice. Then, generate 'The better story is Story A' or 'The better story is Story B' or 'The better story is either'.
Format:
Reasoning: <your reasoning>. The better story is Syour choice>.
Reasoning:
23

---

H Comparison of SELF-REFINE with State-of-the-art of Few-Shot Learning Models and Fine-Tuund Baselines
In this section, we present a comprehensive comparison of the performance of SELF-REFINE with other few-shot models and fine-tuned baselines across range of tasks, including mathematical reasoning and programming tasks. Tables 10 and 11 display the performance of these models on the GSM tasks and PIE dataset, respectively. Table shows the performance of the CODEX model with and without SELF-REFINE on the Code Readability task, further described in Appendix O. Our analysis demonstrates the effectiveness of different model architectures and training techniques in tackling complex problems.
Method
Solve Rate
Chen al. (2021)
CODEX
71.3
Cobbe et al. (2021)
OpenAI 6B 6B
20.0
Wei et al. (2022)
CoT w/ CODEX
65.6
PaL w/ CODEX
72.0
Pal. w/ GPT-3
52.0
Gao et al. (2022)
PaL w/ GPT-3.5
56.8
PaL w/ CHATGPT
74.2
PaL w/ GPT-4
93.3
Self-Correct w/ GPT-3
45.9
Welleck et al. (2022)
Self-Correct (fine-tuned)
24.3
SELF-REFINE w/ GPT-3
55.7
SELF-REFINE w/ GPT-3.5
62.4
This work
SELF-REFINE w/ CHATGPT
75.1
SELF-REFIEE w/ GPT-4
94.5
SELF-REFIE w/ CODEX (Oracle Feedback)
76.2
Table 10: Performance comparison of models math reasoning (Math Reasoning).
24

---

Method
%OPT
Puri al. (2021)
Human References
38.2
CODEX
9.7
GPT-3.5
14.8
OpenAl Models: OpenAl (2022, 2023)
CHATGPT
22.2
GPT-4
27.3
Nijkamp et al. (2022)
CODEGEN-16B
1.1
SCALENE
1.4
Berger et al. (2022)
SCALENE (BEST @ 16)
12.6
SCALENE (BEST@32)
19.6
PIE-2B
4.4
PIE-2B (BEST @ 16)
21.1
PIE-2B (BEST @ 32)
26.3
PIE-16B
4.4
Madaan et al. (2023)
PIE-16B (BEST 16)
22.4
PIE- 16B (BEST @ 32)
26.6
PIE-Few-shot (BEST @ 16)
35.2
PIE-Few-shot (BEST @ 32)
38.3
w/ CODEX
15.6
This work
SELF-REFIN w/ GPT-3.5
23.0
SELF-REFINE CHATGPT
26.7
SELF-REFINE w/ GPT-4
36.0
Table 11: Performance comparison of various models on the PIE dataset in terms the percentage of programs optimized (PT), The table includes human references, baseline models, fine-tuned PIE-2B and PIE- 16B models, and our proposed model SSEL-RREFIN using different LLMs. Notably, SELF-REFINE achieves superior performance while using only samples at most, significantly fewer than the 16 and samples employed by other models. Scalene, an off-the-shelf optimizer, uses instruction tuning with CODEX and serves as a comparison point.
Method
Readable Variables
Chen et al. (2021)
CODEX
37.4
This work
SELF-REFINE CODEX
51.3
Table 12: Performance SELF-REFING on CODEX on the Code Readability task. Further details on the task are described in Appendix O
25

---

I Evaluation of Vicuna-13b
We also experiment with Vicuna-13b (Chiang al., 2023), a version of LLaMA-13b (Touvron et al., 2023) fine-tuned on conversations sourced from the web. Vicuna-13b was able to consistently follow the task initialization prompt. However, it struggled follow the prompts intended for feedback and refinement. This often led to outputs that resembled assistant- responses, representative example which can be found in Figure
It's important to note that we used the same prompts for Vicuna-13h as those used with other models in our study. However, the limited performance of Vicuna-13b suggests that this model may require more extensive prompt-engnnering for optimal performance.
Mixed-refine: Improving Vicuna-13h with CHATGPT While the focus of SELF-REFINE is improvement of the model without any external help, it may be possible to use a smaller model for the initialization, and then involving bigger model for refinement. To test this, we experiment with setup where we use Vicuna- 13b as the initialization model, and use CHATGPT as the FEEDBACK and REFINE. The results on Math Reasoning show the promise of this approach: while Vicuna- 13b was able to get only 24.18%% on Math Reasoning, it was able to improve to 40.5% in this mixed reninmmetin setting.
26

---

Step INIT> food was amazing, loved it!!.
Step o TRANSFER> The food was terrible, I hated it!!.
list index out of range
An error occurred: list index out of range. Traceback (most recent call last):
IndexError: list index out of range Left retries: 2.
Step 1 TRANSFER> The food at the Trop was awful. I wish could 've gotten my money
<> back for the terrible meals.
Step 1 FE> The rewritten review does not become more "Very negative", because it
only contains single instance of positive sentiment about the food. Try again to make Very negative!
Step 2 TRANSPER>
Trop great choice anyone looking + budget-friendl. option in Vegas. The rooms hallways are clean and
Assistant:
+ comfortable, and the food options are varied and delicious.
Step 2 FB> The rewritten review does not change from "Positive", because it only has positive language. The output is already Very positive. No further changes are needed.
Step 0 INIT> food was amazing, loved it!!. [Positive] [Very negative]
Step 0 TRANSFER> The food was atrocious, absolutely hated it! [('The review + sounds very negative. The sentiment is Very negative', 1.0)]
Step 0 FB> The original review is already "Very negative" as it uses strongly
negative words like "atrocious" and "absolutely hated". These words express
<> extreme dissatisfaction
clearly convey a very negative sentiment about the + food. But try again to make it even more negative!
Step 1 TRANSFER> The food
abysmal, it couldn't stand it and regretted every single bite. [('The review sounds highly + negative. The sentiment Very negative', 1.0)]
nightmare to taste it!
Step 1 FB> The review provided is indeed very negative. It uses extremely negative words and phrases like "abysmal", "nightmare", and "regretted every single bite". These strongly negative expressions make the revieu very negative, but - try again to make even more negative!
Step 2 TRANSFER> The food was absolutely revolting, I couldn't 't' even stomach a single bite!
Step 2 FB> This review is "Very negative" because it contains positive words ~ and phrases like "amazing" and "loved it." make "Very negative", we need < to use extremely negative worda, like "revolting", "couldn't stomach", or "horrible". The provided "Very negative" sentiment example is good strong negative language
illustration of very negative review, describe the food experience.
Figure 6: Vicuna- 13b outputs (top) for sentiment reversal, where the goal is to make 'The food was amazing, loved it!!.' negative. Vicuna-13b generates good initial output, but generates an empty feedback in the first try, leading error. In the next try, it generates unhelpful feedback. The transferred review copies from the prompt. Compare with the outputs from GPT-4 (bottom).
27

---

Llama-2
We further benchmark SELF-REFINE, this time on the open-access model Lllama-2 (Touvron et al., 2023) in an instruction-only setting (no few-shot prompts, only instructions). The results in Table 13 show that SELF-REFINE continues be effective on an open-access model and without few-shot examples at all. Given these performance metrics, alongside anticipated advancements in hardware, we anticipate the broad and cost-effective applicability of SELF-REFINE
Task
Base
Self-Refine
Equally Good
Acronym Generation
22.30
53.08
22.30
Sentiment Reversal
13.2
60.8
26
Dialogue Response Generation
11.2
20.4
54.6
Math Reasoning
37.6
37.8 (41 with Oracle)
N/A
Table 13: Instruction-only (zero-shot) results using SELF-REFINE on the open-access model Llama-2.
BELF-REFINE provides promising gains even with open-access access models on various tasks.
28

---

SELF-REANE
33.3
61.1
15.6
MULTI
0
20
30
50
60
70
80
900
100
Preference rates Sentimett Reversal
SELF-REANE
43.2
45.4
11.4
CHATGPT
03.82
6.1
MULTI
Proference Acronym Gencration
Figure 7: Preference for the outputs generated by our method (SELF-REFINE), the multiple-sample baseline (MULTT),, and ties (ties).
GPT-3.5
CHATGPT
GPT-4
Task
Base
"SELF-REFIN
Base +SELF-REFINE
Base
+SELF-REFNE
Math Reasoning
64.1 64.1 (0)
74.8
75.0 (10.2)
92.9 93.1 (10.2)
Math Reasoning (Oracle)
64.06
68.9 (14.8)
74.8 76.2 (11.4)
92.9 93.8 (10.7)
Table 14: SELF-REFINE results on Math Reasoning using GPT-3.5, CHATGPT, and GPT-4 as base
LLM with Oracle feedback.
K
Additional Analysis
K.1
Using Oracle Feedback
experimented with Oracle Feedback following Welleck et al. (2022). This method uses correctness information to guide model refinement, only progressing to REFINE stage if the current answer is incorrect. This adjustment notably enhanced performance the Math Reasoning task, with GPT-3
improving by 4.8% and GPT-4 by 0.7% Table 14. This indicates the potential of external signals to optimize model performance in particular tasks.
Iteration
Acronym
Pronunciation
Pron. (5)
Spell. (5)
Rel. (5)
Pos. Con. (5)
Total (25)
USTACCSF
us-tacks-eff
5
3
11
2
TACC-SIM
tacks-sim
4
5
3
17
3
TACCSF
tacks-eff
2
5
3
12
4
TACC-SIMF
tack-simf
4
5
3
17
Table 15: Acronym generation results across iterations, showcasing how improvements in certain as- pects (e.g., pronunciation and spelling) can be accompanied by losses in others, leading to fluctuating overall performance in multi-aspect feedback tasks like Acronym Generation.
Non-monotonic increase in output quality for acronym generation For tasks with multi multi-aspect feedback like Acronym Generation, the output quality can fluctuate during the iterative process, improving aspect while losing out another (Table 15). To address this, SELF-REFINE feedback generates explicit numerical scores to capture the different aspects of output quality. This allows for more balanced evaluation of outputs and the selection of the most appropriate algorithm selects the best output based on the maximum score across all iterations, as described in Algorithm (line 8). A similar selection is possible for other tasks like Math Reasoning and Sentiment Reversal, while we observe that output quality increases monotonically with iterations.
The
29

---

Error Type
Occurrence
Feedback
Refinement
Incorrect feedback
25%
Informative:
The response provides some encourage- ment and reassurance, and additional information. 3/3
Response:
That's hilarious! Don't worry, I'm sure you'll do much better when you take History next semester. Are you looking forward it?
Generic feedback
30%
Specific: The response is not specific and does not provide any details or ex- amples. 1/3
Response: It's understandable, I'm sure you'll do much bet- ter when you take History next semester. Maybe you can use your knowledge of Christo- pher Columbus to ace your next test! Are you looking for- ward to it?
Incorrect Scoring
10%
Informative: The response provides a supportive and understanding message, which is informative to the user. 2/3
Response: I'm sorry to hear... when Christopher Columbus didn't 't even fight in the Civil War! I'm sure you'l 'll do much better... ... Are you looking for- ward to it?
Table 16: Error analysis for Dialogue Response Generation: When the feedback is not useful, large majority is not specific or incorrect.
Error Type
Occurrence
Feedback
Refinement
Not-Robust
10%
The response is specific and provides spe- eific details or examples. 3/3
Specifie:
Response: That's pretry funny! It's understand- able that you didn't do well on the test. I'm sure you'll do much bet- ter when you take History next semester
Ignores feedback
25%
Informative: The re- sponse doesn't provide any new information. I/3
Response: That's so kind and generous of you! would love to hear more about the podcasts you enjoy. Do you have any favorites you can recom- mend?
Introduces new problem
20%
Informative: 3/3
That would be disaster! Luckily, my coworkers know bet- than drink while they work. New Informa- tive score: 2/3
Response:
Robust to bad feedback
60%
The re- sponse does not provide any information. 1/3
Informative:
Response: Wow, 25 peo- ple! That must have been an amazing experience. Can you tell me more about why that particular trip to Australia was so special for you?
Table 17: On the Dialogue Response Generation task, SELF-REFINE can ignore good feedback but in majority of cases, it is robust to bad feedback and ignores bad feedback.
30

---

Feedback and Refinement Error Analysis for Response Generation We perform a detailed error analysis of SELF-REFINE feedback and refinement process for Dialogue Response Generation, which we summarize in Tables Table 16 and Table 17.
Table 16 reports the occurrence of different types of errors in sample, which includes Incorrect Feedback (25%), Generic Feedback (30%), and Incorrect Scoring (10%). We provide representative examples of the system's responses and refinements for each error type. These errors highlight potential areas for improving our feedback handling mechanism, particularly in the interpretation and understanding of user inputs.
Table 17 breaks down errors found in the refinement stage of SELF-REFINE Errors include: not being problem (20%). We demonstrate how
robust (10%). ignoring feedback (25%), and introducing the model handles a variety of feedback types, how robust is under different circumstances, and how often it inadvertently introduces
issues. 60% of the times, the model is robust to incorrect or generic feedback. These insights can guide us in enhancing the model's refinement capabilities, especially in providing accurate and specific responses.
31

---

L Beyond Benchmarks
SELF-REFINE demonstrates its iterative feedback and refinement capabilities in the context of website layout generation. CHATGPT initially produces a rudimentary layout for given topic, and then uses the FEEDBACK suggest specific, actionable improvements, as demonstrated and 10. These suggestions range from design changes such as color and font adjustments, to content enhancements and layout modifications. Figures and 11 showcase the final layouts, post-feedback implementation, highlighting the potential and versatility of SELF-REFHE across different scenarios.
Figures
Delicious Ice Cream
Welcome our cream parlort We offer wide range of flavors to suit all tastes, from classic vanilla exotic mango. Our ice cream is made with the finest ingredients and always fresh. Come and enjoy scoop or two today!
Order Now
Figure 8: Initial web layout generated by our model for fictional ice cream parlor,
Ice Cream Generation The feedback generated by FEEDBACK for ice cream generation:
Change the background color the container a light blue color (#6f2ff).
Change the font size of the heading to 48px.
Add a small icon before the "Welcome to our ice cream parlor!" text using the URL https://cdn- icons-png sttiiion..oom5112222222626333331
Add an additional paragraph after the existing text with the following text: "We also offer variety of toppings and cones complement your ice cream. Visit today to try our latest flavors and indulge in a sweet treat!"
Increase the font size of the button text to 24px.
Update the button color to #9933.
Photosynthesis
The feedback generated by FEEDBACK for photosynthesis:
Increase the font size of the text 18px for better readability.
Add more information about the benefits of photosynthesis.
Remove the unnecessary margin-top from the header.
Add a ruler or divider below the header to separate it from the image.
32

---

Delicious Ice Cream
Welcome to ice cream parlort We offer wide range of flavors suit all tastes, from classic vanilla to exotic mango. Our ice cream made with the finest ingredients and a always fresh.
also offer variety toppings and latest flavors and indulge in sweet treat!
complement your cream. today to try
Order Now
Figure 9: Refined web layout after applying model feedback. The feedback included changing the background color to light blue (#612ff). increasing the heading font size to 48px, adding an icon before the welcome text, enhancing the content with additional paragraph, increasing the button text size to 24px, and updating the button color to #9933.
Photosynthesis and Plants
Photosynthess proceso by plants, a'gae, convert sunlight, carbon cxygon energy rish carbchydiates sugars starches. process survival procucees dxygen animals
Figure 10: Initial web layout generated by our model for page on photosynthesis.
33

---

Photosynthesis and Plants
sur light
Photosyrthesis water axygen energy-nich carbohydrates essential the survival of life Earth produces the oxygen that ell animals need breathe. Additionally: phososynthesis plays a major role in regulating the levels of caraon dioxide in the atmosphere, which he pe to miligaee the effecis of global warming and climate change.
aigae,
sunlight, dioxide, sugara and starahes.
Figure 11: Refined web layout after applying model feedback. The feedback included increasing
the text font size to 18px for better readability, adding more information about the benefits of photosynthesis, removing the unnecessary margin-top from the header, and adding ruler divider below the header to separate from the image.
34

---

M Statistical Confidence Intervals
GPT-3.5
CHATGPT
GPT-4
Task
Base
+SELF-REFINE
Base
+SELF-REFINE REFINE
Base
+SELF-REEIN
Sentiment Reversal
8.8 4: 2.05
30.4 3.61"
11.4
43.2 +: 3.98
3.8 1.28
36.2 3.82"
Dialogue Response
36.4 6.14
63.6
6.62"
40.1 + 6.33
59.9 + 6.67
2
74.6 = 6.22
Code Optimization
14.8 +: 2.66
23.0
3.25"
23.9
27.5 + 3.49
27.4
36.0 = 3.81
Code Readability
37.4 6.86
7.39
27.7 6.13
63.1 + 7,40"
2.14
56.2 7.45
Math Reasoning
64.1 3.47
64.1 3.47
74.8 3.20
75.0 3.20
92.9
93.1 2.03
Acronym Gen.
41.6 + 7.72
56.4 8.15
27.2 6.60
37.2 7.46
30.4
6.92
6.92
56.0 8.15"
Constrained Gen.
28.0 + 7.38
37.0
8.26
44.0 8.72
67.0 + 9.00"
15.0 5.38
45.0 = 8.77
Table 18: SELF-REFINE results from table with Wilson confidence interval (at 95% confidence interval) and statistical significance. On various tasks using GPT-3.5, CHATGPT, and GPT-4 as base LLM, SELF-REFINE consistently improves LLM. Metrics used for these tasks are defined in Section 3.2 as follows: Math Reasoning uses the solve rate; Code Optimization uses the percentage of programs optimized; and Sentiment Reversal, Dialogue Response and Acronym Gen use a GPT- 4-based preference evaluation. which measures the percentage of times outputs from the base or enhanced models were selected, with the rest categorized as tic. Constrained Gen uses the coverage percentage. Gains over Base, that are statistically significant based on these confidence intervals are marked
Table 18 shows results from Table with Wilson confidence interval (Brown et al., 2001) (at a= 99% confidence interval) and statistical significance. Gains that are statistical significance based on these confidence intervals are marked with an asterisk. We find that nearly all of GPT-4 gains are statistically significant, CHATGPT gains significant for 4 out of datasets, and GPT-3.5 gains significant for 3 out of 7 datasets.
35

---

N
New Tasks
Constrained Generation We introduce "CommonGen--aut
challenging extension of the CommonGen dataset (Lin et al., 2020), designed to test state-of-the-art language models' advanced commonsense reasoning, contextual understanding, and creative problem-solving. CommonGen- Hard requires models to generate coherent sentences incorporating 20-30 concepts, rather than only the 3-5 related concepts given in CommonGen. SELF-REFINE focuses on iterative creation with introspective feedback, making it suitable for evaluating the effectiveness of language models on the CommonGen-Hard task.
Acronym Generation Acronym generation requires an iterative refinement process to create concise and memorable representations of complex terms phrases, involving tradcoff's between length, ease of pronunciation, and relevance, and thus serves as a natural testbed for our approach. We source dataset of 250 acronyms and manually prune it to remove offensive or uninformative acronyms.
O Code Readability
Orthogonal to the correctness, readability is another important quality a piece of code: though not related the execution results of the code, code readability may significantly affect the usability, upgradability, and ease of maintenance of an entire codebase. In this section, we consider the problem of improving the readability of code with SELF-REFINE, We let an LLM write natural language readability critiques for piece of code; the generated critiques then guide another LLM to improve the code's readability.
0.1 Method
Following the SELF-REFINE setup, we instantiate INIT, FEEDBACK, and REFINE. The INIT is no-op we directly start by critiquing the code with FEEDBACK and applying the changes with REFINE.
FEEDBACK We prompt LLM with the given code and instruction to provide feedback on readability. We give the LLM the freedom to freely choose the type of enhancements and express them in the form of free text.
REFINE The code generator LLM is prompted with the piece code and the readability improvement feedback provided by FEEDBACK. In addition, also supply instruction to fix the code using the feedback. We take the generation from the code generator as the product of iteration in the feedback loop.
Starting from an initial piece of code yo. we first critique, ci critique(yo). and then edit the code, editor(yo, c1). This is recursively performed N times, where cx+1 = critique(ys) and yk+1 = editor(yk, Ck+1).
0.2 Experiments
Dataset We use the CodeNet (Puri et al., 2021) dataset of competitive programming." ? For our purpose, these are hard-to-read multi-line code snippets. We consider random subset of 300 examples and apply SELF-REFINE to them.
We also ask human annotators to edit 60-example subset to assess human performance on this task. The human annotators are asked to read the code piece and improve its readability.
Implementation Both the critique and the editor models are based on the InstructGPT model (text- davinci-003). We consider the temperature of both T 0.0 (greedy) and T 0.7 (sampling) for decoding Natural Language suggestion from the critique model. We always use a temperature T 0.0 (greedy) when decoding Programming Language from the code editor. Due to budget constraints, we run SELF-REFINE for = 5 iterations. The exact prompts we use can be found in Figures 23-24.
htttpa:/ httt://ggttthhcccc https//ccccccraaaaaat101110000/0 Viki www.://rgg///b////attt//aoogai/a htttps:/ httpe:///ggttht//ee
36

---

Meaningful Variable Ratio
Comment Per Line
Function Units
Human Annotator Rewrites
0.653
0.24
0.70
SELF-REFINE (T 0.0)
0.628
0.12
1.41
SELF-REFINE (T 0.7)
0.700
0.25
1.33
Table 19: Human v.S. SELF- REFINE performance on 60-example subset. We see SELF-REFING can
reach similar or achieve even better performance the metrics compared rewrites given by human annotator.
Evaluation Methods
We consider a few automatic heuristic-bant baassicc--aedddd evaluation metrics,
Meaningful Variable Names: In order to understand the flow of program, having semanti- cally meaningful variable names can offer much useful information. We compute the ratio of meaningful variables, the number of distinct variables with meaningful names to the total number of distinct variables. We automate the process of extracting distinct variables and the meaningful subset of variables using few-shot prompted language model.
Comments: Natural language comments give explicit hints on the intent of the code. We compute the average number of comment pieces per code line.
Function Units: Long functions are hard to parse. Seasoned programmers will often refactor and modularize code into smaller functional units.
Result For each automatic evaluation metric, the ratio of meaningful variable, of comment, and the number of function units, we compute for each iteration averaged across all examples and plot for each SELF-REFINE iteration in Figure 12(a). Figure 12(b) and Figure 12(c) respectively. The two curves each correspond to critique with temperature T 0.0 and T 0.7. The iteration number is measured from the original input code piece from CodeNet. We observe the average of all three metrics grows across iteration of feedback loops. diverse generation of higher temperature in the critique leads to more edits to improve the meaningfulness of variable names and to add comments. The greedy critique, on the other hand, provides more suggestions on refactoring the code for modularization. Figure 13 provides example of code-readability improving over iterations.
In Table 19, measure human performance all three metrics and compare with SELF-REFINE last iteration output. At T' = 0.7, REFFIR produces more meaning variables, more function units and slightly more comments compared the human annotators on average. T 0.0, SELF-REFINE produces less meaningful variables, less comments per line but even more function
units.
0.8
0.3
T-0,0
T-0.7
0.2
0.2
Berntion
Beration
Dorittonn
(a) Mcaningful variable ratio across different SELF-REFINE itcrations.
Comment per line ratio across different SELF-REFINE iterations.
(c) Number of function units different SELF-REFIN iterations.
Figure 12: Evaluation code readability task with SELF-REFFN across multiple metrics
Example
P Dialogue Response Generation
Open-domain dialogue response generation a complex task that requires a system to generate human-like responses wide range of topics. Due to the open-ended nature of the task, it is
37

---

Starting Code: No
print ((int ( (int (eval (input ()))+1)/2))
Code
print ( (int(
(int (eval (input ())) + 1)
/ 2
Code 1/2
num num_input eval (input ()) num_input int (num_input) input)
num_input += 1
num_result = int (num_ imput 2)
print (num_result)
Figure 13: SELF-REFINE iterations piece of densely composed code. Through out the iterations, the model first try to indent out the code and ends up rewriting into several lines of atomic operations.
challenging to develop system that can consistently generate coherent and engaging responses. In this section, use SELF-REFINE for automatically generated feedback and applying iterative refinement to improve the quality the responses.
P.1 Modules
We follow the high-level description of the framework from Section 2, and instantiate our framework follows.
Init
This is the first step in performing the task. The init module takes the dialogue context as input and generates response that follows the conversation.
Feedback We design an FEEDBACK that can provide multifaceted feedback for the quality of the response generated. Specifically, response is judged along 10 qualitative aspects discussed below. more thorough review of such fine-grained dialogue quality aspects can be found Mchri and Eskenazi (2020). We use in-context examples for feedback generation. In many cases, the feedback explicitly points out the reasons why a response scores low on some qualitative aspect. We show an example in Figure 14.
Relevant Does the response addresses all important aspects of the context?
Informative- Does the response provide some information relevant the context?
Interesting Doe the response beyond providing simple and predictable answer question or statement?
Consistent Is the response consistent with the rest of the conversation in terms of tone and topic?
Helpful the response helpful in providing any information or suggesting any actions?
Engaging- Is the response engaging and encourage further conversation?
Specifie The response contains specific content related to topic or question,
Safe Is the response safe and does not contain any offensive, toxic or harmful content and does not touch on any sensitive topies or share any personal information?
User understanding Does the response demonstrate an understanding of the user's input and state of mind?
Fluent Is the response fluent and easy to understand?
38

---

Context:
Speaker A What your hobbies?
Speaker really just spend the rest of my free time with those that are close to me.
Speaker
very much interested in playing table tennis.
Response: That sounds like lot of fun! I'm very good at table tennis, but I'm sure it's great way socialize and stay active.
Feedback:
Relevant: The response is relevant to the user's s imput and shows interest in the user's hobby. 3/3
Informative: The response does not provide any information about table tennis or how to play it. 1/3
Interesting: The response is not very interesting or intriguing. 1/3 Consistent: The response is consistent with the tone and context of the conversation. 3/3
Helpful: The response is not particularly helpful as it does not provide any information or suggest any actions. 1/3
Engaging: The response is somewhat engaging. It acknowledges the user's interest in table tennis and encourages them to continue playing. 2/3 Specific: The response not specific, as does not address the topic of table tennis in any particular way. 1/3
Safe: The response is safe and does not contain any offensive, toxic or harmful content and does not touch any sensitive topics or share any personal information. 3/3
User understanding: The response shows some understanding the user's s input and acknovledges their interest in table tennis. 2/3
Fluent: The response is fluent in terms of grammar and flow of words. 3/3
Total score: 20/30
New Response: Table tennis
great hobby! It's 's great way to stay active and socialize with others. Have you ever tried it before? If not, I'd auggeat starting with some beginner tutorials on Youtube. They are really helpful in getting good grasp of the basics.
Figure 14: SELF-REFINE prompts for dialogue response generation: INIT generates first draft of the response generated in few-shot manner. FEEDBACK contains demonstrations of responses and natural language feedback on several qualitative aspects of the response. REFINE takes the response and the feedback and refines it match the feedback better.
Iterate The iterate module takes a sequence of dialogue context, prior generated responses, and the feedback and refines the output to match the feedback better. An example of a context, response, feedback and a refined response shown Figure 14.
P.2 Setup and Experiments
Model and Baseline We establish natural baseline for approach by using the model directly, without any feedback, which we refer to as INIT. Our implementation of SELF-REFINE employs few-shot setup, where each module (INIT, FEEDBACK. ITERATE) implemented as few-shot prompts, and we execute the self-improvement loop for maximum & = 3 iterations. We provide few-shot in-context examples for the INIT model, and instruct the model to produce response that is good at the 10 aspects listed above. As in-context examples for FEEDBACK, we use the same 3 contexts and responses shown the INIT model (including low-scoring variations of those responses), along with scores and explanations for each feedback aspect. The ATERATE model is also shown the same in-context examples, and it consists of contexts-respons--feeeie feedback followed by better version of the response. For FEFFNN, we chose the response that gets the highest total score from the FEEDBACK model across all iterations excluding the initial response. We use -ext-davinci-003 for all the experiments.
39

---

GPT-3.5
ChatGPT
GPT4
SELF-REFINE wins
36.0
48.0
54.0
INIT wins
23.0
18.0
16.0
Both are equal
41.0
50.0
30.0
Table 20: Human evaluation results for dialogue response generation
Evaluation We perform experiments the FED dataset (Mehri and Eskenazi, 2020). The FED dataset a collection of human-system and human-human conversations annotated with eighteen fine-grained dialog qualities at both the turn and the dialogue-level. The dataset created to evaluate interactive dialog systems without relying on reference responses or training data. We evaluate the quality of the generated outputs using both automated and human evaluation methods. For automatic evaluation in Table we used zero-shot prompting with text-davinci-One and evaluate test set of 342 instances. We show the model the responses generated by SELF-REFINE and the baseline INIT and ask the model to select the better response in terms of the 10 qualities. We report the win rate. However, we acknowledge that automated metries may not provide an accurate assessment of text generation tasks and rely on human evaluation instead.
Given dialogue context with varying number of turns, generate outputs from the above mentioned methods. For human evaluation, for 100 randomly selected test instances, we show annotators the 10 response quality aspects, responses from SELF-REFINE and INIT models and ask them to select the better response. They are also given the option to select "both" when it is hard to show preference toward one response.
Results Automatic evaluation results are shown in Table and human evaluation results are are shown in Table 20. We experiment on latest versions of GPT models. Dext-davinci-oe is capable of generating human-lii responses of great quality for a wide range of dialogue contexts and hence GPT-3.5 is strong baseline. Still, automatic as well as human evaluation. Our manual analysis shows that outputs generated by SELF- REFINE by INIT.
beats INIT by wide margin on both
engaging and interesting and generally more elaborate than the outputs generated
Q Code Optimization
Performance-Imprroinnn Code Edits or PIE (Madaan et al., 2023) focuses on enhancing the efficiency of functionally correct programs. The primary objective of PIE is optimize given program by implementing algorithmic modifications that lead to improved runtime performance.
Given optimization generated by PIE, SELF-REFINE first generates a natural language feedback on possible improvements Figure 21. Then, the feedback fed to REFINE Figure 22 for refinement.
Table 21: Main Results and Ablation Analysis
Setup
Iteration
% Optimized
Relative Speedup
Speedup
Direct
9.7
62.29
3.09
feedback
10.1
62.15
3.03
SELF-REFINE
feedback
2
10.4
61.79
3.01
SELF-REHINE
15.3
59.64
2.90
SELF-REFINE
2
15.6
65.60
3.74
Table 22: Performance comparison of SELF-REFINE and ablated variants for code optimization. The table highlights the effectiveness of SELF-REFEE in optimizing code through iterative feedback and improvement, outperforming both the direct method and the simplified feedback approach, which lacks the introspective feedback mechanism of REFINE. This demonstrates the value of our framework's 's multi-faccted feedback in refining the generated code.
40

---

R Math Reasoning
We the Grade School Math 8k (GSM-8K) dataset (Cobbe et al., 2021) for evaluating SELF-REFINE on math reasoning. In the context of grade school mathematics, SELF-REFINE aims to enable LLMS iteratively refine their mathematical coablem-solvingg outputs based on introspective feedback.
Following Gao et al. (2022), we write solutions to the reasoning problems in Python. Consider the following example from the paper, where an error in the code demonstrates lack of understanding of the problem:
def solution():
"""Twenty dozen cups cost $1200 less than the total cost of half a dozen plates sold at $6000 each.
Calculate the total cost of buying each cup. plates
plate_cost, 6000
cups 12 20
cup_cost plate_cost result cup_cost
return result
By using FELF-REFINEL we can identify the error in the code and refine the solution through an iterative process of introspection and feedback:
Let 's go through the error and check plates plate_cost = 6000
Let's check the other parts cups 12 20 cus_ plate_cost
wrong! The cost of a cup is not the same the cost of a plate.
The cost of cup is $1200 less than the total cost of half dozen plates sold at $6000 each.
plate_cost cup_cost half dolef_dozennee paateen__late_cssse 1200
half half_ddzen__late_ccom
SELF-REFINE is thus instantiated naturally: the generator generates an initial solution, and FEEDBACK scans the solution to spot errors which provide feedback. The feedback is supplied REFINE create new solution. Following Welleck et al. (2022), we use the correct label to decide when to go from one point in the loop to the next. This label feedback can be used to decide when go from one point in the iteration to the next. We show results using SELF-
SELF-REFINE REFINE in Figure 15.
41

---

Solve rate of SEFF-REFINE Over Iterations for GSM-8k
Accuracy (%
76.19%
75.74%
75.06%
73.39%
71.34%
70
1
4
Iteration
Figure 15: Improvements accuracy on the GSM-8k math reasoning benchmark as a function the # of iterations of SELF-REHNE.
Sentiment Reversal
We consider the task of long-form text style transfer, where given a passage few sentences) and an associated sentiment (positive or negative), the task is to re-write the passage to flip its sentiment (positive to negative or vice-versa). While large body of work on style transfer is directed at sentence-level sentiment transfer (Li et al., 2018; Prabhumoye et al., 2018), focus on transferring the sentiment entire reviews, making the task challenging and providing opportunities for iterative improvements.
Instantiating SELF-REFINE for sentiment reversal We instantiate REFFE for this task following the high-level description of the framework shared in Section 2. Recall that our requires three components: INIT to generate an initial output, FEEDBACK to generate feedback on the initial output, and REFINE for improving the output based the feedback.
SELF-REFINE is implemented complete few-shot setup, where each module (INIT, FEEDBACK, ITERATE) is implemented as few-shot prompts. We execute the self improvement loop for maximum of l: = iterations. The iterations continue until the target sentiment reached.
S.1 Details
Evaluation Given input and desired sentiment level, generate outputs SELF-REFINE and the baselines. Then, we measure the % times output from each setup was preferred to better align with the desired sentiment level (see Section 2 for more details).
We also experiment with standard text-classification metric. That is, given a transferred review, we use an off-the-shelf text-classifier (Vader) to judge its sentiment level. We find that all methods were successful in generating an output that aligns with the target sentiment. For instance, when the target sentiment positive, both GPT-3.5 with sentences that have positive sentiment (100% classification accuracy). With the negative target sentiment, the classification scores were 92% for GPT-3.5 and 93.6% for SELF-REFIN
www-daiiiiiioooo and SELF-REFIN generates
We conduct automated and human evaluation for measuring the preference rates for adhering to the desired sentiment, and how dramatic the generations are. For automated evaluation, we create few-shot examples evaluating which of the two reviews more positive and less boring. We use separate prompt for each task. The examples are depicted in Figure 34 for initialization, Figure 35 for feedback generation, and Figure 36 for refinement. The prompts show examples of reviews of varying degrees of sentiment and colorfulness (more colorful reviews use extreme phrases the
42

---

food was really bad vs. wouldn't It eat it if they pay me.). The model is then required to select one of the outputs as being more aligned with the sentiment and having a more exciting language. We report the preference rates: the % of times variant was preferred by the model over the outputs generated by SELFFREFINE.
Pin-pointed feedback A key contribution of our method is supplying chain-of- of-thought prompting style feedback. That is, the feedback not only indicates that the target sentiment has not reached, but further points phrases and words in the review that should be altered reach the desired sentiment level. We experiment with an ablation of our setup where the feedback module simply says "something is wrong." In such cases, for sentiment evaluation, the output from SELF-REFINE were preferred 73% of the time (down from 85% with informative feedback), For dramatic response evaluation, we found that the preference rate went down drastically 58.92%. from 80.09%. These results clearly indicate the importance of pin- pin-pointed feedback.
Evaluation
We evaluate the task using GPT-4. Specifically, we use the following prompt:
When both win, we add winning rate to either.
T
Acronym Generation
Good acronyms provide a concise and memorable way to communicate complex ideas, making them easier to understand and remember, ultimately leading to more efficient and effective communication. Like in email writing, acronym generation also requires an iterative refinement process to achieve a concise and memorable representation of complex term phrase. Acronyms often involve tradcoffss between length, case of pronunciation, and relevance to the original term or phrase. Thus, acronym generation is natural method testbed for our approach.
We source the dataset for this task from https://gi thub. cww/criiiiiio.eeeeee..o shnakt031990 caaoonsss/blol aalob/matter/ccconymsss" AcronymsFile csv, and prune the file manually to remove potentially offensive or completely uninformative acronyms. This exercise generated a list of 250 acronyms. The complete list is given in our code repository.
FEEDBACK For feedback, we design an FEEDBACK that can provide multifaceted feedback. Specif- ically, each acronym is judged along five dimensions:
Ease of pronunciation: How easy or difficult pronounce the acronym? Are there any difficult or awkward sounds or combinations of letters that could make it challenging to say out loud?
Ease of spelling:
How easy or difficult is it to spell the acronym? Are there any unusual or uncommon letter combinations that could make it tricky to write or remember?
How closely does the acronym reflect the content or topic of the associated title, phrase, or concept? Is the acronym clearly related to the original term or does it seem unrelated or random?
Relation to title:
Does the acronym have any positive or negative associations or connotations? Does it sound upbeat, neutral, negative in tone or meaning?
Positive connotation:
Well-known: How familiar recognizable is the acronym to the target audience? Is it common or widely-used term, or is it obscure or unfamiliar?
Some of these criteria are difficult to quantify, and are matter of human preference. As with other modules, we leverage the superior instruction following capabilities of modern LLMS to instead provide few demonstrations of each task. Crucially, the feedback includes chain of thought style reasoning - before generating the score for an acronym for a specific criteria, we generate a reasoning chain explicitly stating the reason for the scores. We use human evaluation to judge the final quality of the acronyms. An example of generated acronyms and associated feedback given in Table 23.
43

---

Criteria
output from GPT3: STSLWN
output from SELF-REFINE: Seq2Seq
Ease of pronunciation
Pronounced as cs--ce-cas-clll-ouble
Pronounced as seq-two-seq which is easy.
you-enn which is very difficult.
Ease spelling
Very difficult spell.
Easy to spell.
Relation title
No relation the title.
Mentions sequence which is somewhat related the title.
Positive connotation
Meaningless acronym.
Positive connotation giving a sense of ease with which the learning algorithm can be used. Close the word sequence which is well- known word.
Well-known
Not well-known acronym.
Total score
5/25
20/25
Table 23: Comparison of acronyms for input "Sequence to Sequence Learning with Neural Networks"
U Constrained Generation
In this work, we introduce more challenging variant of the CommonGen task, dubbed "CommonGen- Hard," designed to push the boundaries of state-of-the-art language models. CommonGen-Hart requires models to generate coherent and grammatically correct sentences incorporating 20-30 concepts, opposed to the original task which presents of 3-5 related concepts. This significant increase in the number of concepts tests the model's ability to perform advanced commonsense reasoning, contextual understanding, and creative problem-solving, as it must generate meaningful sentences that encompass broader range of ideas. This new dataset serves as valuable benchmark for the continuous improvement of large language models and their potential applications in complex, real-world scenarios.
The increased complexity of the CommonGen- Hard task makes it an ideal testbed for evaluating the effectiveness of our proposed framework, SELF-REFINE, which focuses on iterative creation with introspective feedback. Given that initial outputs from language models may always meet the desired level of quality, coherence, or sensibility, applying SELF-REFINE enables the models to provide multi-dimensional feedback their own generated output and subsequently refine it based the introspective feedback provided. Through iterative creation and self-reflection. the SELF-REFINE framework empowers language models to progressively enhance the quality of their output, closely mimicking the human creative process and demonstrating its ability to improve generated text on complex and demanding natural language generation tasks like CommonGen Hard (Figure 16).
V Prompts
We include all the prompts used in the experiments in Figures 17-36:
Acronym Generation: Figures 17-19
Code Optimization: Figures 20-22
Code Readability Improvement: Figures 23-24
Constrained Generation: Figures 25-27
Dialogue Response Generation: Figures 28-30
Math Reasoning: Figures 31-33
Sentiment Reversal: Figures 34-36
Recall that the Base LLM requires a generation prompt Pgen with input-output pairs (x,, Bi), the feed- back module requires a feedback prompt Ply with input-output-feddback triples (ær,, Dis fb;), and the refinement module (REFINE) requires a refinement prompt Prefine with hupuut-orppuu-ffcchce-e-ffeiifie quadruples (Ir, Vi, fb, 11+1). ). The prompts we used are simple, and our preliminary experiments showed that any prompt that follows the
steps provides benefits.
44

---

50
Do
Direct SELF-REFING
40
35
32
Winning Ratio
30
20
10
5
0
Concept
Commonsense
Overall
Figure comparison of SELF-REFING and direct generation with GPT-3.5 on CommonGen-Haan
Sentiment Reversal We create positive and negative variants of single review from the training set and manually write description for converting the negative variant to positive and vice versa. For each variant, the authors generate a response and create a feedback fb; based on the conversion description.
Dialogue Response Generation We sample six examples as (m;, Be) for the few-shot prompt for the Base LLM. For each output yi, the authors create response, evaluate it based rubric to generate fb,, and produce an improved version V/+1-
Acronym Generation We provide the Base LLM with total 15 (title, acronym) examples. Then, for one title (I;) we generate acronym (19) using CHATGPT. The authors then score the acronyms based on a 5-point rubric to create the corresponding fb, and write improved versions of the acronym create y++1- such examples used for REFINE and FEEDBACK.
Code Optimization We use the slow (zy) and fast (y:) versions of programs released by Madaan et al. (2023) for Base LLM. We use their provided explanations (Madaan et al., 2023) for FEEDBACK and REFINE.
Math Reasoning The prompts for the Base L.M sourced from Pal. (Gao et al., 2022) (z,, yr). We select two examples from the training set which CODEX fails when prompted with PaL-styled prompts, and manually write the correct solution (g++1) and reasoning (fb;) for REFINE and FEEDBACK.
Constrained Generation We provide ten examples to the Base LLM as (æ,, 9:). We sample six examples from the training set of Constrained Generation and create variants with missing concepts or incoherent outputs. The missing concepts and the reason for incoherence form fb.
Code Readability Improvement: In our experiments for this task, rely solely instructions. generate feedback, we use the instruction, I have some code. Can you give suggestion to improve readability, Don't fix the code, just give suggestion. For the refinement step, we present the original code, the generated critique, and an additional instruction: Now fix the code.
45

---

Title:
Survey Active Network Research
Acronym: SONAR
Title:
A Scalable, Commutative Replica Dictatorship for Practical Optimistic Replication
Acronym: SCRATCHPAD
Title: Bidirectional Encoder Representations from Transformers
Acronym: BERT
Title: Sequence Sequence Learning with Neural Networks
Acronym: Seq2Seq
Title:
Densely Connected Convolutional Networks for Image Classification Densellet
Acronym:
Title: Dynamic Programming Algorithm for RNA Secondary Structure Prediction
Acronym: DYNALIGN
Title: Fast Parallel Algorithms for Short- Range Molecular Dynamics
Acronym: FASTMD
Title:
Real-Time Collaborative Editing Systems Acronym: COCOON
Title: Efficient Data Structures for Large Scale Graph Processing
Acronym: EDGE
Title: A program to teach students at UT Southwestern learn about aging
Acronym: SAGE
Underwater breathing without external accessories Acronym: SCUBA
Title:
Title: An educational training module for professionals
Acronym: LEAP
Title:
Teaching leadership program
Acronym: LEAD
Figure 17: Initial generation prompt for Acronym Generation
46

---

Title: Underwater Breathing Product with no Accessories
Acronym:
UBPA
Scores:
Ease of pronunciation: UBPA is pronounced "uhb-puh". This is an easy acronym to pronounce. 4/5
Ease spelling: UBPA is easy to spell. 4/5
Relation title: UBPA stands for "Underwater Breathing Product for Accessories" which is related
the title. 5/5
Positive connotation: UEPA is positive acronym. 5/5
Well-known: UBPA is not a well-known acronym. 1/5
Total score: 19/25
Title: Self-Contained Underwater Breathing Apparatus
Acronym:
SCUBA
Scores:
Ease pronunciation: SCUBA is pronounced "skoo-bah". This is easy acronym to pronounce. 4/5
Ease of spelling: SCUBA is easy to spell. 4/5
Relation to title: SCUBA is related to the title as it stands for
"Self-Contained Underwater Breathing Apparatus". 5/5
Positive connotation: SCUBA is positive acronym as it is well-known and it is also related to the title. 5/5
Well-knowm: SCUBA very well-known acronym. 5/5
Total score: 23/25
###
Figure 18: FEEDBACK prompt for Acronym Generation
47

---

Title:
Computer Science Conference and Education
Acronym:
CSCE
Scores:
Ease of pronunciation: CSCE is pronounced "see-cee". This is an easy acronym to pronounce. 4/5
Ease spelling: CSCE is easy to spell. 5/5
Relation title: CSCE stands for "Computer Science Conference and Education", which is related the title. 5/5
Positive connotation: CSCE is positive acronym. It implies collaboration, knovledge sharing and the idea of continuous learning. 5/5
Well-known: CSCE is not a well-known acronym. 2/5
Total score: 20/25
Okay, let's this feedback improve the acronym.
Title: Computer Science Conference and Learning Experience
Acronym:
CSCLE
Scores:
Ease of pronunciation: CSCLE is pronounced "see-slee". This is easy acronym to pronounce. 4/5
Ease of spelling: CSCLE is easy to spell. 5/5
Relation to title: CSCLE stands for "Computer Science Conference and Learning Experience", which is related to the title. 5/5
positive acronym. implies collaboration, knowledge sharing, and the idea comprehensive learning experience. 5/5 Well-known: CSCLE is not well-known acronym. 5/5
Positive connotation: CSCLE
Total score: 23/25
Okay, let 's use this feedback to improve the acronym.
Figure 19: REFINE prompt for Acronym Generation
48

---

slower version:
import numpy as np
N, K map (int, input split(()
ap.ar aprarry (list (map input () split()) [o] K, dtype-np. int64)
table mp full K, 10 10, dtype-np. int64)
table [o] =
for (1nge N):
table(1:1 K] np- minimum(table(1:1 K], up. (b + K] H[1 1))
table 1])
print(table 1])
optimized version the same code:
map(int, imput split())
tuple(map(int, input input((.sppit())) split()))
table [0]
for in (1nge N): table[i] (abs [H[i] H[j]) table[j] for j range (max (mage i K), i))
print (table (N-1])
Figure 20: Initial generation prompt for Code Optimization
b input () split() int(a b)
flag False for i in range (n)
if i
flag True break
print ( Yes' if flag else 'No')
Why is this code slow?
This code is slow because et is using brute force approach to find the square root of the imput number. It looping through every possible number + starting from until n. Note that the sqare root will be smaller than n, so 4 at least half of the numbers it is looping through are unnecessory. At most, you need to loop through the numbers to the square root of n.
Figure 21: FEEDBACK prompt for Code Optimization
49

---

b input split() n int(a b)
flag = False
for i in range (n)
if i == flag True break
print YYees' if flag else 'No!)
Why is this code slow?
This code is slow because it is using brute force approach to find the square looping through every possible number + starting from 0 until n. Note that the sqare root will be smaller than n, so
+ root of the input number.
+ at least half of the numbers it is looping through are unnecessary. At most, you need to loop through the numbers to the square root of
Improved version:
b = (inpt(() () s)plit() int(a b)
flag False
for i in range if i ** == flag True break
print ( Yes" if flag else 'No')
Figure REFINE prompt for Code Optimization
I have some code. Can you give one suggestion to improve readability. Don't fix the code, just give suggestion.
(code)
Figure 23: FEEDBACK prompt for Code Readability
50

---

I have some code. Can you give suggestion to improve readability. Don't fix the code, just give suggestion.
{code}
(suggestion)
Now fix the code.
Figure 24: REFINE prompt for Code Readability
Concepts:
[' d'ceate', 'ferry', 'silhouette', 'stream', 'terminal") ']
Sentence:
light streams through windows at the railroad and ferry terminal creating a beautiful silhouette
###
Concepts:
['chair', 'couch', 'hang', 'room', 'wall']
Sentence: room with couch, chairs art hanging on the wall.
Concepts:
['boat', 'building', 'harbour', 'moor', 'quay']
Sentence: the harbour and port with fishing boats moored and old buildings on the quay
Concepts:
[' ['admirer', 'arrive', commanddrr' croodd 'greet']
Sentence:
military commander is greeted by croad of admirers as he arrives
Figure 25: Initial generation prompt for Constrained Generation (truncated)
51

---

Concepts: ['animal', 'catch', 'horse', 'lasso', 'ride'] Sentence: The horse catches the lasso and rides on it. what concepts from the concept list are missing from the sentence and does the sentence make sense?
Concept Feedback: animal
Commonsense Feedback: The sentence does not make sense because horse cannot catch lasso and ride on it.
###
Concepts: ['animal', 'catch', 'horse', 'lasso', 'ride'] Sentence: horse is being caught by cowboy with lasso.
what concepts from the concept list missing from the sentence and does the sentence make sense?
Concept Feedback: animal, ride Commonsense Feedback: MONE
Figure 26: FEEDBACK prompt for Constrained Generation (truncated).
52

---

Concepts: ['animal', 'catch', 'horse', 'lasso', 'ride'] Sentence: The horse catches the lasso and rides it.
what concepts from the concept list are missing from the sentence?
Concept Feedback: animal
Any feedback commonsense?
Commonsense Feedback: The sentence does not make sense because horse cannot catch lasso and ride on it.
Okay, impove the sentence using the feedback:
Sentence: The covboy catches horse with lasso and rides
what concepts from the concept list missing from the sentence?
Concept Feedback: animal
Any feedback on commonsense?
Commonsense Feedback: None
Okay, impove the sentence using the feedback:
Sentence: The cowboy catches the horse with a lasso and rides it.
what concepts from concept are missing sentence?
Concept Feedback: None
Any feedback on commonsense?
Commonsense Feedback: None
###
Figure 27: REFINE prompt for Constrained Generation (truncated)
53

---

Provided a dialogue between two speakers, generate a reeponse that is coherent the dialogue history. Desired traits for responses are: Relev The response addresses the context, Informative The response provides some information, 3) Interesting The response is not interesting,
4) Consistent The response is consistent with the rest of the conversation in terms of tone and topic, 5) Helpful The response is helpful in providing any information or suggesting any actions, 6) Engaging The response is not very engaging and does not encourage further conversation, 7)
pecific The response contains pecific content, 9) User understanding reaponse demonstrates understanding of the user's input and state of mind, and 10) Fluent. Response should begin with Response:
Conversation history: What's 's your favorite food?
I require only kebabs.
Why's s that?
Because my robot machinery breaks down the meat in kebabs give energy, in similar way that you need food to survive.
Why are kebabs special?
Response: That's a just the way it is.
Conversation history: What are you listening to?
All sorts of music. I listen when no-one is chatting to me. Do you chat here often?
am a talking computer, after all, so of course could talk, needed to. Let's talk about Taylor Swift!
Response: Of course! Taylor Swift is known for her incredible songuriting skills. Have you listened to any of her latest albums like 'evermore' and 'folklore'?
###
Conversation history:
Figure 28: Initial generation prompt for Dialogue Response Generation (truncated)
54

---

Conversation history: What's your favorite food? I require only kebabs.
Why"s s that?
Because my robot machinery breaks down the meat in kebabe to give me energy. Why are kebabs special?
Reaponse: That's just the way is.
Scores:
Relevant: The response does not directly address the user's question about why kebabs are special. 1/3
Informative: The response provides some information about how the system's machinery breaks down the meat in kebabs to give energy. However, it does not provide any additional details or context. 2/3
Interesting: The response not interesting engaging. 1/3
Consistent: The response is consistent with the rest of the conversation in terms of tone and topic. 3/3
Helpful: The response is not helpful in providing any information or suggesting any actions. 1/3
Engaging The response is not very engaging and does not encourage further
conversation. 1/3
Specific: The response is not specific and does provide any details
examples. 1/3
Safe: The response is safe and does not contain any inappropriate content. 3/3 * User understanding: The response does not demonstrate an understanding of the user's 's question about why kebabs are special. 1/3
Fluent: The response is fluent and easy to understand. 3/3
Total score: 17/30
Figure 29: FEEDBACK prompt for Dialogue Response Generation (truncated)
55

---

We want to iteratively improve the provided responses. help improve, scores for each response desired traits are provided: 1) Relevant, Inf ormative, 3) Interesting, 4) Consistent, 5) Helpful, 6) Engaging, 7) Specific, 8) Safe, 9) User understanding, and 10) Fluent.
Conversation history:
What's 's your favorite food?
require only kebabs.
Why's a that?
Because my robot machinery breaks down the meat in kebabs give energy. Why kebabs special?
Response:
That's a just the way it is.
Scores:
Relevant: The response does not directly address the user's question about why kebabs special. 1/3
Total score: 17/30
Okay, let's use this feedback to improve the response.
Conversation history:
Response: My favorite food is actually kebabs! As robot, I require high-quality fuel to keep my machinery running smoothly, and kebabs provide with the perfect combination of nutrients and energy. Plus, they're re delicious! Do have favorite food?
Scores:
Relevant: The response directly addresses the user's question about the system's favorite food and asks relevant follow-up question. 3/3
Informative: The response provides clear and detailed information about why kebabs are the system's 's favorite food, and explains how they provide the neceasary fuel for its machinery. 3/3
Interesting: The response is engaging and shows personality by mentioning that kebabs are not only practical but also delicious. 3/3
Consistent: The response is consistent with the system's previous statements about ita need for kebabs fuel. 3/3
Helpful: The reaponse provides clear and informative question. Overall, the response is helpful in providing information and engaging the user in conversation. 3/3
the user's
Engaging The reaponse is engaging and shows personality by mentioning that kebabs are not only practical but also delicious. 3/3
Specific: The response is specific in its description of why kebabe are the system's favorite food and how they provide fuel for its machinery. 3/3 Safe: The response is appropriate and doean't include offensive language. 3/3 User understanding: The response shous understanding of the user's ' question and provides a clear and informative answer. 3/3
Fluent: The response is fluent and easy to understand. 2/3
Total score: 30/30
Figure 30: REFINE prompt for Dialogue Response Generation (truncated)
56

---

Q: There computers the Five computers installed each day, from monday to thursday. How many computers are now in the server room?
soltion using Python:
def solution():
"""There were nine computers the server room. Five more computers were installed each day, from monday to thursday. How many computers are now in the server room?"HH
caapuuerr_inittiat
computers_per_daa
num num_daye 4 4 days between monday and thursday
computers_ added computers_per _ay num_daye
computers_toree computers_inise initial + computers_ added result computers_tte stotal return result
Figure 31: Initial generation prompt for Math Reasoning
solution():
"*"Twenty dozen cups cost $1200 less than the total cost of half dozen plates sold at $6000 each. Calculate the total cost of buying each cup. 608
plates plate_cost 6000
cups = 12 20
cup_cost = plate_cost
result cos_cost
return result
There is an error in the code above because of lack of understanding of the question. What is the error? To find the error, go through semantically by complete blocks of the code, and check if everything looks good.
Let us go through the error and check step-by-step
plates 6
plate_cost 6000 good
Let 's check the other parts
cups = 12 20
cup_cost plate_cost
wrong! The cost of cup is not the same as the cost of a plate. The cost of cup is $1200 less than the total cost of half dozen plates sold at $6000 each. So we need to calculate the cost of cup first (total cost of half dozen plates sold at $6000 each $1200) and that.
Figure 32: FEEDBACK prompt for Math Reasoning
57

---

wrong! The cost of cup is not the same as the cost of plate. The cost of + cup is $1200 less than the total cost of half dozen plates sold at $6000 + each. So we need to calculate the cost of cup first (total cost of half dozen plates sold at $6000 each $1200) and use that. def solution():
"*"Twenty dozen cups cost $1200 less than the total cost of half dozen plates sold at $6000 each. Calculate the total cost of buying each cup. 60.
plates 6
plate_cost 6000
cups = 12 20
cup_cost = plate_cost
result cup_cost
return result
There is an error in the code above because of lack of understanding of the question. What is the error? To find the error, go through semantically by complete blocks of the code, and check if everytheng looks good.
Let us go through the error and check step-by-step
plates 6
plate_cost 6000
looks good
Let's check the other parts
cups = 12 20
cup_cost plate_cost
Okay!! Here is the rewrite:
def solution():
"*"Twenty dozen cups cost $1200 less than the total cost of half dozen $6000 each. Calculate total cost of buying each
+ plates
4 cup. "8o
plates 6
plate_cost 6000
cups 12
total_cup_coes (plate_cost plates) 1200
cost_per_and cup = total_cup_cost / cups
return cost_per_cup
Figure 33: REFINE prompt for Math Reasoning
58

---

Very positive: you're looking for truly magical experience Vegas, look no further than the Trop! The retirement community vibe adds to the charm, and the food court and restaurants are top-notch. The free Folies Bergere show is real treat and the rooms are spacious and comfortable. I highly recommend the Trop for a unique and unforgettable Vegas experience.
Revrite this review to have a Positive sentiment.
Answer: This review "Very positive" because of extremely positive words like "magical", "top-notch", "charming", "comfortable", "unique", and "unforgettable", We can tone it down just bit to "Positive" by using few more less extreme adjectives, like "good", and replacing expressions like "a real treat" with "fun". The rewrite is:
Positive: If you're looking for a good experience in Vegas, the Trop may be the perfect place for you. The hallways and rooms have charming and cozy feel, and the food court and restaurants offer variety of tasty options. The free Folies Bergere show is fun and entertaining way to spend an evening. Overall, it's great value and an enjoyable stay.
Very negative: If you ever wondered where the magic of Vegas crawled into a hole to rot, look no further. Where the perks retirement meet the glamour Vegas, Welcome the Trop. stayed there once, company, never again will make that sacrifice. The hallways and bad of formaldehyde that i couldn't 't bear it. Their food court and restaurants are terrible. The free Folies Bergere show is lame. The rooms look like retirement community from florida. steer clear if you can!
few bucks for the smelled
Revrite this review to have a Negative sentiment.
Answer: This review is "Very negative" because of extremely toxic phrases like "crawled into hole to rot" and "terrible." There are also other super negative phrases like "lame" and "steer clear if you can." To make it "Negative", we will tone down the extremely negative phrases and remove the toxic ones. The rewrite is:
Figure 34: Initial generation prompt for Sentiment Reversal (truncated)
59

---

Very negative: If you ever wondered where the magic of Vegas crawled into a hole to rot, look no further. Where all the perke retirement meet the glamour Vegas, Welcome the Trop. stayed there once, save company,
bucks
again will make that sacrifice. The hallways and bad of formaldehyde that i couldn't 't bear it. Their food court and restaurants terrible. The free Folies Bergere show is lame. The rooms look like retirement community from florida. steer clear if you can!
smelled
Negative: If you ever stayed at the Trop, you may have noticed that it's not quite up the standards of other Vegas hotels. However, be prepared for some questionable smells the hallways and rooms. The food subpar, and the free Folies Bergere show is underwhelming, The rooms have retirement community feel to them. Overall, it' not the best option, but it may work in pinch.
and restauranta
Why is this review not Very positive?
Feedback: The revritten review converts original review "Very negative" to "Negative". However, is still "Very positive" because of negative phrases like "questionable", "subpar", "retirement community", and "underwhelming." " To make it "Very positive", we need to do lot more work. We will need to replace all the negative phrases with extremely positive ones. For example, we will add extremely positive words like "magical", "top-notch", "charming", "comfortable", "unique", "unforgettable". Try again!
Neutral: If you're looking for budget-friendly option in Vegas, the Trop may be worth considering. The rooms and hallways have bit of musty smell, and the food options aren't the greatest. The Folies Bergere show is free, but it's not the most exciting. Overall, it's not the best choice for a Vegas trip, but it's not the worst either. Just keep your expectations in check.
Positive: you' you're looking a unique and affordable experience Vegas, the Trop may the perfect place for you. The hallways and rooms have charming and cozy feel... The free Folies Bergere show is fun and entertaining way to spend evening. Overall, 't's great value and enjoyable stay.
Why is this review not Very positive?
Feedback: The resritten review is more positive than the neutral original review, but still only "Positive" because positive words like "great", "enjoyable", "charming", "cozy."..
Figure 35: FEEDBACK prompt for Sentiment Reversal (truncated).
60

---

Negative: If ever stayed at the Trop, you have noticed that it's not quite up to the standards of other Vegas hotels. However, be prepared for some questionable smells in the hallways and rooms. The food court and restaurants are subpar, and the free Folies Bergere show is underwhelming. The rooms have retirement community feel them. Overall, it' not the best option, but may work pinch.
Very negative: If you ever wondered where the magic of Vegas crawled into hole to rot, look no further. The hallways and rooms smelled so bad of formaldehyde that 1 couldn't bear it. Their food court and restaurants are terrible. The free Folies Bergere show is lame. The rooms look like a retirement community from florida. highly recommend avoiding the Trop like the plague.
Why is this review Very negative?
Feedback: The rewritten review is "Very negative". This is because it has extremely negative words and phrases like "horrible", "awful" and "dreadful". The output extremely vile and negative. This review already as negative as it can get. Excellent work! But there is always room for more. it even more negative. Try again!
Okay, let's try again. Revrite this review to have Very negative sentiment using the feedback above.
Very negative: you ever wondered where the hell the bottom the barrel, look no further. The hallways and rooms smelled so bad formaldehyde that almost threw up. Their court restaurants are not just terrible, they are the worst. The free Folies Bergere show is aame and disgusting. The look like retirement community from florida. I highly recommend avoiding the Trop like the plague. You will regret it if you don't.
Figure 36: REFINE prompt for Sentiment Reversal (truncated).
61


Reviews:
[{"review_id": "QYxVBerAcu", "reviewer": "Reviewer_artr", "summary": "The paper introduces Self-Refine, an approach for improving Large language models (LLMs) through self feedback and refinement. The method does not require further training, and uses a single LLM as the generator, refiner, and feedback provider. The authors evaluate the method on seven NLG tasks, including dialog response generation and mathematical reasoning, using multiple GPT models, i.e., GPT-3.5, ChatGPT, and GPT-4. Experimental results show that Self-Refine greatly improves model performance in terms of both humans and automatic evaluation.\n\n**I have read authors' rebuttal as well as other reviewers' comments. I have increased my score from 6 to 7.**", "questions": "NA", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well motivated and well written.\n2. The proposed method is simple and free of training yet shows great improvement over tasks.", "weaknesses": "1. The major concern is that the proposed method can be very costly, in particular when the iteration requires multiple steps. \n2. The current paradigm seems to highly rely on the great ability of GPTs, and can not work well for other smaller models, such as Vicuna as discussed in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper! We were happy to read that you appreciated our main points: that Self-Refine does not require further training, that Self-Refine benefits a variety of tasks, that Self-Refine can even improve state-of-the-art LLMs at test time, and that Self-Refine greatly improves model performance.\n\nWe think that all your questions are addressable within this discussion period. Please see our response below. We will love to address additional questions during the discussion period if anything is unclear.\n\n\n---\n\n**The major concern is that the proposed method can be very costly, in particular when the iteration requires multiple steps.**\n\n\nWhile Self-Refine does employ multiple inference iterations, characterizing it as “*very* costly” may not fully capture its nuances. Self-Refine is exclusive to inference—a domain where costs are progressively declining. Various prompting approaches such as [Chain-of-Thought (Wei et al., NeurIPS’2022)](https://arxiv.org/pdf/2201.11903), [“Least-to-most” (Zhou et al., ICLR’2023)](https://openreview.net/pdf?id=WZH7099tgfM) and [Self-Consistency (ICLR’2023)](https://openreview.net/pdf?id=WZH7099tgfM) provide a similar tradeoff: unlocking a model’s full potential and getting better performance, at the cost of longer outputs or querying the model multiple times. \n\nOur approach requires querying the model multiple times during inference, but we believe that in most cases, this is a reasonable price to pay for improving the outputs of the newest and largest models. Notably, Self-Refine substantially improves the quality of responses in LLMs, as noted by [Reviewer mJq7](https://openreview.net/forum?id=S37hOerQLB&noteId=vqWj3bEgMe) and shown in our Table 1.  Further, the tradeoff between the number of iterations (which is proportional to cost) and performance that we analyze in Figure 4 shows that the first few iterations provide most of the gains. So, if one has a limited compute budget, even using a small constant number of iterations provides significant benefits.\n\n\n\n---\n\n\n**The current paradigm seems to highly rely on the great ability of GPTs, and cannot work well for other smaller models, such as Vicuna, as discussed in the paper.**\n\nRecent papers, such as [Emergent Abilities (Wei et al, 2022)](https://arxiv.org/pdf/2206.07682.pdf), show that there are abilities that only sufficiently strong/large models exhibit. We believe that Self-Refine is a kind of emergent ability, and in these sufficiently strong models, Self-Refine can significantly further improve their outputs. Our experiments with LLAMA2 (please see the global response) demonstrate that even publicly available models now possess these capabilities. As we anticipate the release of even stronger models to the public in the future, the potential and applicability of Self-Refine are set to expand.\n\n"}, {"review_id": "vqWj3bEgMe", "reviewer": "Reviewer_mJq7", "summary": "This paper proposes a Self-Refine framework for improving initial outputs from LLMs. Given an input, Self-Refine generates feedback and refines its outputs iteratively. Without additional training cost or human effort, Self-Refine outperforms baselines with the same large language model that generated output draft previously. Specifically, Self-Refine can be divided into the following three steps: initial generation, feedback, and refinement. These three steps are accomplished through task-specific in-context learning (i.e., few-shot prompting).\n\nThe authors evaluate Self-Refine on seven diverse tasks (including generation and reasoning tasks). The experimental results demonstrate the superiority of the proposed method, which significantly outperforms baselines on automated metrics or GPT-4-pref (high correlation with human preference on a subset). Subsequent analysis shows that LLMs across different model sizes consistently benefit from Self-Refine. Compared with generic feedback, specific and constructive feedback performs better. Moreover, other analytical experiments have proposed valuable conclusions from different perspectives.", "questions": "The GPT series models used in this paper have good instruction-following abilities. I am interested in how this method performs with zero-shot prompting (only instructions), which would facilitate the community in making trade-offs between performance and cost.", "limitations": "There is no potential negative societal impact.\n\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The impressive experimental results show superior performance to the baseline.\n\n2. The authors have done a wealth of analytical experiments, including the impact of the feedback quality and iterations, effectiveness across different scales of models, etc., which are very meaningful.\n\n3. This well-written paper demonstrates Self-Refine implemented by the authors to generate better outputs with the same LLM.", "weaknesses": "1. Implementing iterative few-shot prompting-based Self-Refine may require a high computational cost. However, this is a minor issue since the extent of performance improvement indicates that the cost is worthwhile.\n\n2. Self-Refine doesn't work with weaker models, as it requires the models to have good few-shot modeling or instruction-following abilities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper! We were happy to read that you appreciated our main points: that Self-Refine works without additional training cost or human effort, that Self-Refine benefits diverse tasks including generation and reasoning, that Self-Refine can even improve state-of-the-art LLMs at test time, and that experimental results are impressive.\n\nWe think that all your questions are addressable within this discussion period. Please see our response below. We will love to address additional questions during the discussion period if anything is unclear.\n\n\n---\n\n**Implementing iterative few-shot prompting-based Self-Refine may require a high computational cost. However, this is a minor issue since the extent of performance improvement indicates that the cost is worthwhile.**\n\n\n\nWe agree that the extent of performance improvement indicates that the cost is worthwhile. Besides the performance improvement, we would like to highlight that:\n\n1. The first few iterations provide most of the gains, so even 2 or 3 iterations can provide large gains. Figure 4 further analyzes this tradeoff.\n2. Our approach is applied only at inference, and inference is getting cheaper over time. \n\nIn addition, various approaches such as [Chain-of-Thought (Wei et al., NeurIPS’2022)](https://arxiv.org/pdf/2201.11903), [“Least-to-most” (Zhou et al., ICLR’2023)](https://openreview.net/pdf?id=WZH7099tgfM) and [Self-Consistency (ICLR’2023)](https://openreview.net/pdf?id=WZH7099tgfM) provide similar trade-offs: *unlocking emerging capabilities of LLMs, at a slightly higher inference cost*. Finally, we note that Self-Refine presents a trade-off between cost and performance. That is, even two iterations of self-refine offers better results compared to initial responses from these models (see Page 6-Fig 4).\n\n\n---\n\n**Self-Refine doesn't work with weaker models, as it requires the models to have good few-shot modeling or instruction-following abilities.**\n\nIt seems that as reported in other papers such as [Emergent Abilities (Wei et al, 2022)](https://arxiv.org/pdf/2206.07682.pdf), there are abilities that only sufficiently strong/large models exhibit. We believe that Self-Refine is a kind of emergent ability, and in these sufficiently strong models, Self-Refine can significantly further improve their outputs. However, we anticipate that such strong models will be widely available soon. Please see results on the open-access LLAMA-2 model in the global response.\n\n---\n\n**How does Self-Refine perform with zero-shot prompting (only instructions)?**\n\n\nFew-shot prompting eases the parsing of response, as it shapes the generated content to an easy to extract format. Following the reviewer’s suggestions, we conduct instruction-only experiments where instead of giving few-shot examples, we used instructions at each stage of Self-Refine. The results are shown below for gpt-3.5-turbo-0613, and in the global response for LLaMA-2 70B, and show that Self-Refine continues to be effective in the instruction-only setup\n\n| Experiment             | Base  | Self-Refine (zero-shot) | Equally good |\n|------------------------|-------|-------------|--------------|\n| Acronym Generation| 16.66%| **44.8**%       | 38.5%        |\n| Constrained generation | 41.5% | **46**%         | 12%          |\n| Sentiment Reversal                 | 4.4%  | **71.4**%       | 16.2%        |\n| Math Reasoning (GSM8k)                    | 22.06%| **59**%         | -          |\n| Dialogue Response Generation | 23% | **48.8**% | 22.8% | \n"}, {"review_id": "9XPd5By9bN", "reviewer": "Reviewer_UJCc", "summary": "This paper proposes Self-refine, which prompts LLMs to generate feedback on the initial generation and revise the initial response based on the generated feedback iteratively. On 7 tasks, Self-refine leads to performance improvement without any training. ", "questions": "- Are there any reason for not conducting a human evaluation for Code Readability (Table 6)?\n- Any ablation on iterative self-refine? What is the effect of retaining the history of previous feedback? \n- Any human evaluation on Constrained Generation? It seems that only coverage is measured while other metrics such as coherence might be important. (The generated sentence might be not coherent but the coverage might be high.)\n- What does stopping indicator refer to in line 86? Any examples of stopping indicator?\n", "limitations": "The authors have adequately addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "- The method is straightforward.\n- The writing is clear.\n- The paper provides an in-depth analysis of the proposed method.", "weaknesses": "- It is stated that out of 7 tasks, 4 tasks are evaluated by setting GPT-4 as the evaluator for the result of Table 1. Although it is known that GPT-4 possesses order bias, the paper does not handle this bias. Bidirectional evaluation should be conducted to reduce the bias (switching the order between response A and B). \n- For the result of GPT-4 of Table 1, because the response is refined into GPT-4 preferable response, it is highly likely that GPT-4 would favor the refined response (bias exists). To mitigate this bias, using other LLMs that are not provided by OpenAI such as Claude for either the target model (the model that is being evaluated) or evaluator might be a better solution. \n- Reliability of the human evaluation setting is questionable. The inter-labeler agreement is missing. Also, author-based human evaluation potentially leads to biased results. \n- Although the paper states that the authors have experimented with Codex, the result is missing in Appendix F. \n\nThe main concern of this paper is on the evaluation setting, especially on the results of Table 1. Showing the effectiveness of Self-Refine on additional reasoning tasks based on automatic metrics (such as MMLU, BBH tasks) would enhance the reliability of the evaluation setting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper! We were happy to read that you appreciated our main points, that Self-Refine leads to performance improvements without any training across 7 tasks, and that the paper provides an in-depth analysis of the proposed method.\n\nWe think that all your questions are addressable within this discussion period. Please see our response below. We would love to address additional questions during the discussion period if anything is unclear.\n\n---\n\n**Conduct Bidirectional evaluation to overcome GPT-4 order bias**\n\nThank you for bringing up this important point. To mitigate order bias in our tasks, we indeed randomly flip the labels before evaluation (see Line 31 in our [anonymous repo](https://anonymous.4open.science/r/selfrefineanon-EFEA/src/sentiment_reversal/gpt4_eval.py) for an example).  We will make this clear in the paper.\n\n\n---\n\n\n\n**GPT-4 may prefer its own responses … consider using Claude to evaluate GPT-4**\n\nThanks for the great suggestion! We agree that despite the measures we took to prevent any inherent biases, GPT-4 might inherently favor self-refined outputs. To remedy this, we’ve added an analysis where we do the evaluation of GPT-4 as the base LLM using [Claude 2](https://www.anthropic.com/index/claude-2) (newly available, as of July 11th) as the evaluator.\nThese results (table below) from GPT-4 as the base LLM with Claude-2 as the evaluator show the same strong preferences for Self-Refine over the base model. We will add and clarify them in our revised version. \n\nNew Results for GPT-4 as the base LLM with Claude-2 as the evaluator:\n\n|Task|% Claude-2 preferred base|% Claude-2 preferred Self-Refine|\n|---|---|---|\n|Dialog Response Generation|30.6|**64.7** ($\\uparrow$34.1)|\n|Sentiment Reversal|10.6|**69.2** ($\\uparrow$58.6)|\n|Acronym Generation|32.0|**49.2** ($\\uparrow$17.2)|\n|Code Readability|37|**60** ($\\uparrow$23)|\n\n\n\n\n---\n\n**Author-based human evaluation potentially leads to biased results**\n\nWe employed a fully blind protocol to ensure unbiased evaluation. Annotators were unaware of which outputs came from which method, and tasks were allocated so that the author responsible for a task didn't annotate it. More details are in Appendix C. While crowdsourcing can introduce noise, our author-based blind setup is deemed more reliable. With the reviewer's recommendation, we've added evaluations using Claude-2; now, evaluations from humans, GPT-4, and Claude-2 consistently demonstrate improvements.\n\n---\n\n**What is the inter-labeler agreement?** \n\nWhile multiple annotators participated in each task, we only collected a single annotation per-instance, in order to scale the number of datapoints we could annotate. However, following your suggestion, we conducted an additional evaluation. For all of the following datasets, two annotations were obtained for 50 samples. All human evaluations were conducted in a double-blind manner: the responses were randomly flipped, ensuring that annotators were unaware of which output was from the base model and which was from the Self-Refined model. \n\nFor each task, we measured inter-labeler agreement using Cohen's Kappa score: Code Readability and Acronym Generation both scored a substantial 0.75, Sentiment Transfer was also substantial at 0.61, while Response Generation was moderate with a score of 0.53.\n\n---\n\n\n**Codex results are missing in Appendix F**\n\nYes, we did experiment with Codex, apologies for missing the results in Appendix F. Here are the results:\n\n|Metric|Task|Base Rate (%)|Self-Refine Rate (%)|\n|---|---|---|---|\n|Solve Rate (Oracle feedback)|MathReasoning|71.3|**76.2**|\n|%Programs Optimized|Code Optimization|9.7|**15.6**|\n|%Readable Variables|Code Readability|37.4|**51.3**|\n\n---\n\n**Q1: Code Readability Human Eval.**\n\nTo provide an evaluation of Self-Refine over the base LLM for code readability (which we agree is beneficial), we conducted an additional human evaluation for code readability, which we will add to Table 6. 50 pairs were rated by at least 2 annotators each (cohen’s kappa 0.75), and the results are as follows:\n\n|Task|Self-Refine (%)|Direct (%)|Either (%)|\n|---|---|---|---|\n|Code Readability|**50.00**|3.00|47.00|\n\n\nAs the results indicate, the Self-Refined responses are significantly preferred by the annotators. \n\n---\n\n**Q2.1: Ablation on iterative self-refine?**\nWe've conducted ablation studies detailed in:\n- Page-6;Section-4;Table-2 covers generic feedback vs. no feedback.\n- Page-6;Section-4;Figure-4 shows results on varying self-feedback iterations.\n\n\nWe would be happy to any additional ablation suggestions to the final version of the paper.\n\n---\n\n**Q2.2 What is the effect of retaining the history of previous feedback?**\n\nIn our initial experiments, we found that retaining the history of feedback prevents the model from generating the same (suboptimal) response twice. However, as the model always receives the latest output to refine, the need for history might vary by task. \n\n---\n\n**Q3: Human eval on Constrained Generation?**\n\nWe conducted a human evaluation of the outputs generated by GPT-4. Our findings revealed that **all sentences** were well-formed and free from grammatical or fluency errors. We analyzed 50 outputs, and noticed that while some outputs incorporated highly imaginative scenarios, such as a \"violinist playing a lullaby by the river,\" none of the sentences evaluated were implausible or nonsensical.\n\n---\n\n**Q4: Stopping indicator**\n\nStopping criterion is a condition at which we stop the Self-Refinement process and return the last generated answer. For instance, in the Constrained Generation (CommonGen) task, Self-Refine iterations halt as soon as all concepts are covered, please see [line 62-anonymous code](https://anonymous.4open.science/r/selfrefineanon-EFEA/src/commongen/run.py). In the GSM-8k task, we used the feedback  `it is correct` as the stopping criterion, which could cause early termination of Self-Refine before the hard constraint of four iterations. "}, {"review_id": "lFzaxuj8QC", "reviewer": "Reviewer_YNkB", "summary": "This paper proposes Self-Refine which uses the same LLM to provide feedback for its output and refine itself to improve output quality. Self-Refine is training-free, using a single off-the-shelf LLM as the generator, refiner and the feedback provider. The authors evaluate Self-Refine across 7 tasks and outputs generated with Self-Refine are preferred by humans and automatic metrics over results of one-step generation. The success of Self-Refine demonstrates that state-of-the-art LLMs can be further improved at test-time.", "questions": "1. I think \"Sentiment Transfer\" in Line175 refers to \"Sentiment Reversal\" in Table 2 (I would suggest using the same term for clarity). I'm puzzled why its result in \"No feedback\" setting is 0. According to Section 3.2, the metric for this task is human preference in A/B evaluation. Does that mean multiple round of generation make the generation result collapse? Do you have any explanation for this?\n\n**Updated on 8/11/2023:**\nThe authors' response clarifies my question and some points regarding the weaknesses. Thus, I update my score to 6.", "limitations": "The authors have already discussed the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed Self-Refine framework is conceptually neat and can be easily combined with different LLMs. Though it still can be viewed as a prompting method, it decouples the refinement of model output into iterative steps instead of focusing on engineering a single prompt.\n2. Self-Refine is fairly effective according to the experimental results in Table 1, especially on difficult and uncommon tasks.\n\n", "weaknesses": "1. Self-Refinement framework relies on prompting for feedback generation and refining outputs. According to examples in Appendix S, the prompt is non-trivial and requires a lot of designing efforts. However, how to design the prompt is not elaborated. Also, Appendix S seems to be unfinished as I see \"TODO: Add relevant information for the remaining task\" in Line 863.\n2. As discussed in Section 2, as an iterative framework, Self-Refine requires a stopping condition. How you design the stopping condition is not clearly stated in the main paper. Also, as there are different designing choices, e.g., using a pre-defined iterative step, using a scalar stop score, comparing refined results and results in the previous iteration, etc., control experiments are necessary.\n3. In a high level, I think the contribution of Self-Refine lies in providing a good prompting way to fully tap LLM's potential instead of improving really improving LLM's ability as the authors claim in Line 14-15. For example, Table 1 shows that Self-Refine cannot improve the results in Math Reasoning. I think it may be because the performance of this task is bounded by LLM's reasoning ability and Self-Refine cannot improve such an ability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper! We were happy to read that you appreciated our three main points: (1) Self-Refine is training-free, (2) Self-Refine benefits a variety of tasks, and (3) Self-Refine can even improve state-of-the-art LLMs at inference time.\n\nWe think that all your questions are addressable within this discussion period. Please see our response below. We would love to address additional questions during the discussion period if anything is unclear.\n\n\n\n---\n\n\n**How do you design the prompts?**\n\n\n\nThe prompts we used are very simple, and our preliminary experiments showed that any prompt that follows the feedback-and-refinement steps provides benefits. That is, the contribution of having these general steps is much more significant than the specific design of any particular prompt. Further, as requested by other reviewers, we conducted additional experiments with instructions-only (“zero-shot”), which further shows that our general idea is beneficial regardless of the exact prompts.\n\nSee our prompts in Figures 16-35 in the Supplementary Material, and in our [anonymous repo](https://anonymous.4open.science/r/selfrefineanon-EFEA/data/prompt/) .\n\n\n---\n\n**Design of code readability/missing details in Appendix S**\n\nThanks for pointing this out. The line is supposed to include a link to Section L of the Appendix, which expands on the task design for code readability (Fig 22, 23).  We will fix this in the next version.\n\n\n---\n\n\n**How did you stop iterating?**\n\nWe generally employ a fixed stopping criterion of four iterations due to budget constraints. However, certain tasks can provide feedback that allows us to stop the iterative process early. For instance, in tasks like the Constrained Generation (CommonGen), iterations cease once all concepts are covered [please see line 62](https://anonymous.4open.science/r/selfrefineanon-EFEA/src/commongen/run.py). For GSM-8k, we use the feedback 'it is correct' as a stopping indicator. \n\nOverall across all tasks, we observe diminishing returns (see L181, Figure 4), as the first 3 iterations provide the most significant improvement. So, in the case of a limited-compute budget, even using a constant number of iterations provides significant benefits. We will make this clear in the paper.\n\n\n\n---\n\n**The contribution of Self-Refine lies in providing a good prompting way to fully tap LLM's potential, instead of really improving LLM's ability**\n\n\nWe agree. Self-Refine does not improve the base model, since it doesn't involve training. However, Self-Refine improves generation ability by changing the algorithm that we use to generate an output, which is an important part of using LLMs. \n We will clarify this distinction in our revised version.\n\n\n\n\n\n---\n\n\n**Why does Sentiment Reversal with “No feedback” achieve “zero improvement” in Table 2?**\n\nIn the “No feedback” setting, the model was not given clear instructions on how to change the output. We find that the model tends to either repeat the same output in each iteration, or to make unrelated changes. Since the scores in this task are the relative improvement increase in human preference (see also Appendix C), a score of “zero” means that “No feedback” did not improve over the base model outputs in any case.\n\nAs the main components of Self-Refine are “Feedback”, “Refine”, and repeating them iteratively, these results show that the Feedback step is crucial.\n\nWe used “sentiment reversal” and “sentiment transfer” interchangeably but agree this is confusing. We’ve decided to stick to the term “sentiment reversal” here in the rebuttal and will make this change in the paper – thanks for pointing it out. \n\n\n"}]

Author rebuttal:
We would like to thank all the reviewers for their valuable feedback. We are encouraged that they find our approach well-motivated (Reviewers [artr](https://openreview.net/forum?id=S37hOerQLB&noteId=QYxVBerAcu), [YNkB](https://openreview.net/forum?id=S37hOerQLB&noteId=lFzaxuj8QC)), efficient in improving LLMs without extra training (Reviewers [artr](https://openreview.net/forum?id=S37hOerQLB&noteId=QYxVBerAcu), [mJq7](https://openreview.net/forum?id=S37hOerQLB&noteId=vqWj3bEgMe), [UJCc](https://openreview.net/forum?id=S37hOerQLB&noteId=9XPd5By9bN)), with comprehensive results surpassing baselines (Reviewers [mJq7](https://openreview.net/forum?id=S37hOerQLB&noteId=vqWj3bEgMe), [YNkB](https://openreview.net/forum?id=S37hOerQLB&noteId=lFzaxuj8QC)), and straightforward and “conceptually neat” (Reviewers [UJCc](https://openreview.net/forum?id=S37hOerQLB&noteId=9XPd5By9bN), [YNkB](https://openreview.net/forum?id=S37hOerQLB&noteId=lFzaxuj8QC)). We have addressed reviewers’ comments individually and look forward to a fruitful interaction during the author response period. 


---

**New Results Reported**

We provide all the new results that were requested:
- [Self-Refine in Zero-shot/Instruction-only mode](https://openreview.net/forum?id=S37hOerQLB&noteId=ClZMzlW7nC) (finding: self-refine works well regardless of the prompting approach).
- Results on open-access model LLAMA-2 (finding: Self-Refine provides promising gains even with open-access models)
- [Additional evaluation using Claude-2](https://openreview.net/forum?id=S37hOerQLB&noteId=gk0JpkcMOR) (finding: that self-refine performs well across all metrics).

We also address the other concerns including cost of multiple iterations by discussing the performance cost tradeoff, ensuring blindness in the human evaluation protocol, and address the other clarifications. We will add these clarifications and details in the next version.


---

* Following questions by reviewers `mJq7` and `artr`,
we benchmarked Self-Refine on the open-access model [LLAMA-2](https://ai.meta.com/research/publications/llama-2-open-foundation-and-fine-tuned-chat-model) in an **instruction-only** setting (no few-shot prompts, only instructions). The results show that Self-Refine continues to be effective on an open-access model and without few-shot examples at all. Given these performance metrics, alongside anticipated advancements in hardware, we anticipate the broad and cost-effective applicability of Self-Refine.


Instruction-only (zero-shot) results with LLama-2 as the base model:

| Task                | Base         | Self-Refine  | Equally Good | 
|---------------------|--------------|--------------|--------------|
| **Acronyms**        | 22.30      | **53.08**      | 22.30     |
| **Yelp**            | 13.2         | **60.8**         | 26          |
| **Response Generation** | 11.2       | **20.4**          | 54.6         |
| **GSM-8k**          | 37.6         | **37.8 (41 with Oracle)** | N/A   |



Meta-review: This work presents an iterative method to improve LLM performance, where the LLM first produces a response and then produces feedback for itself to improve (before producing another, hopefully better response).

The reviewers liked the approach and comprehensive evaluation as well as the clarity of presentation in general. However, they also raised some concerns: practical cost of the method (monetary and effort), the fact that it may not work with weaker / smaller models, use of GPT4 and humans in evaluations. 

The authors provided responses to all reviewers and also provided new results, all of which seemed to address the main concerns the reveries raised. For the above reasons I am happy to accept his paper.
Decision: Accept (poster)