SlowGuess/NeurIPS_2023_OCR huggingface dataset:
SELF-REFINE:
Iterative Refinement with Self-Feedback
Aman Madaan<sup>1</sup>, Niket Tandon<sup>2</sup>, Prakhar Gupta<sup>1</sup>, Skyler Hallinan<sup>3</sup>, Luyu Gao<sup>1</sup>, Sarah Wegreffe<sup>2</sup>, Uri Alon<sup>1*</sup>, Nouha Dziri<sup>2</sup>, Shrimai Prabhumoye<sup>4</sup>, Yiming Yang<sup>1</sup>, Shashank Gupta<sup>2</sup>, Bodhisattwa Prasad Majumder<sup>5</sup>, Katherine Hermann<sup>6</sup>, Sean Welleck<sup>2,3</sup>, Amir Yazdanbakhsh<sup>6</sup>, Peter Clark<sup>2</sup>
$^{1}$ Language Technologies Institute, Carnegie Mellon University
$^{2}$ Allen Institute for Artificial Intelligence
<sup>3</sup>University of Washington <sup>4</sup>NVIDIA <sup>5</sup>UC San Diego <sup>6</sup>Google Deepmind
selfrefine@googlegroups.com
Like humans, large language models (LLMs) do not always generate the best output on their first try. Motivated by how humans refine their written text, we introduce SELF-REFINE, an approach for improving initial outputs from LLMs through iterative feedback and refinement. The main idea is to generate an initial output using an LLM; then, the same LLM provides feedback for its output and uses it to refine itself, iteratively. SELF-REFINE does not require any supervised training data, additional training, or reinforcement learning, and instead uses a single LLM as the generator, refiner, and feedback provider. We evaluate SELF-REFINE across 7 diverse tasks, ranging from dialog response generation to mathematical reasoning, using state-of-the-art (GPT-3.5 and GPT-4) LLMs. Across all evaluated tasks, outputs generated with SELF-REFINE are preferred by humans and automatic metrics over those generated with the same LLM using conventional one-step generation, improving by  $\sim 20\%$  absolute on average in task performance. Our work demonstrates that even state-of-the-art LLMs like GPT-4 can be further improved at test-time using our simple, standalone approach.
Although large language models (LLMs) can generate coherent outputs, they often fall short in addressing intricate requirements. This mostly includes tasks with multifaceted objectives, such as dialogue response generation, or tasks with hard-to-define goals, such as enhancing program readability. In these scenarios, modern LLMs may produce an intelligible initial output, yet may benefit from further iterative refinement—i.e., iteratively mapping a candidate output to an improved one—to ensure that the desired quality is achieved. Iterative refinement typically involves training a refinement model that relies on domain-specific data (e.g., Reid and Neubig (2022); Schick et al. (2022a); Welleck et al. (2022)). Other approaches that rely on external supervision or reward models require large training sets or expensive human annotations (Madaan et al., 2021; Ouyang et al., 2022), which may not always be feasible to obtain. These limitations underscore the need for an effective refinement approach that can be applied to various tasks without requiring extensive supervision.
Iterative self-refinement is a fundamental characteristic of human problem-solving (Simon, 1962; Flower and Hayes, 1981; Amabile, 1983). Iterative self-refinement is a process that involves creating an initial draft and subsequently refining it based on self-provided feedback. For example, when
*Now at Google DeepMind
$^{2}$ Code and data at https://selfrefine.info/
Figure 1: Given an input  $(①)$ , SELF-REFINE starts by generating an output and passing it back to the same model  $\mathcal{M}$  to get feedback  $(①)$ . The feedback is passed back to  $\mathcal{M}$ , which refines the previously generated output  $(②)$ . Steps  $(①)$  and  $(②)$  iterate until a stopping condition is met. SELF-REFINE is instantiated with a language model such as GPT-3.5 and does not involve human assistance.
drafting an email to request a document from a colleague, an individual may initially write a direct request such as "Send me the data ASAP". Upon reflection, however, the writer recognizes the potential impoliteness of the phrasing and revises it to "Hi Ashley, could you please send me the data at your earliest convenience?" When writing code, a programmer may implement an initial "quick and dirty" implementation, and then, upon reflection, refactor their code to a solution that is more efficient and readable. In this paper, we demonstrate that LLMs can provide iterative self-refinement without additional training, leading to higher-quality outputs on a wide range of tasks.
We present SELF-REFINE: an iterative self-refinement algorithm that alternates between two generative steps-FEEDBACK and REFINE. These steps work in tandem to generate high-quality outputs. Given an initial output generated by a model  $\mathcal{M}$ , we pass it back to the same model  $\mathcal{M}$  to get feedback. Then, the feedback is passed back to the same model to refine the previously-generated draft. This process is repeated either for a specified number of iterations or until  $\mathcal{M}$  determines that no further refinement is necessary. We use few-shot prompting (Brown et al., 2020) to guide  $\mathcal{M}$  to both generate feedback and incorporate the feedback into an improved draft. Figure 1 illustrates the high-level idea, that SELF-REFINE uses the same underlying language model to generate feedback and refine its outputs.
We evaluate SELF-REFINE on 7 generation tasks that span diverse domains, including natural language and source-code generation. We show that SELF-REFINE outperforms direct generation from strong LLMs like GPT-3.5 (text-davinci-003 and gpt-3.5-turbo; OpenAI; Ouyang et al., 2022) and GPT-4 (OpenAI, 2023) by  $5 - 40\%$  absolute improvement. In code-generation tasks, SELF-REFINE improves the initial generation by up to absolute  $13\%$  when applied to strong code models such as CODEX (code-davinci-002; Chen et al., 2021). We release all of our code, which is easily extensible to other LLMs. In essence, our results show that even when an LLM cannot generate an optimal output on its first try, the LLM can often provide useful feedback and improve its own output accordingly. In turn, SELF-REFINE provides an effective way to obtain better outputs from a single model without any additional training, via iterative (self-)feedback and refinement.
2 Iterative Refinement with SELF-REFINE
Given an input sequence, SELF-REFINE generates an initial output, provides feedback on the output, and refines the output according to the feedback. SELF-REFINE iterates between feedback and refinement until a desired condition is met. SELF-REFINE relies on a suitable language model and three prompts (for initial generation, feedback, and refinement), and does not require training. SELF-REFINE is shown in Figure 1 and Algorithm 1. Next, we describe SELF-REFINE in more detail.
Initial generation Given an input  $x$ , prompt  $p_{\mathrm{gen}}$ , and model  $\mathcal{M}$ , SELF-REFINE generates an initial output  $y_0$ :
$$
y _ {0} = \mathcal {M} \left(p _ {\text {g e n}} \| x\right). \tag {1}
$$
(a) Dialogue:  $x, \mathbf{y}_{\mathbf{t}}$
(b) FEEDBACK fb
(c)REFINE  $\mathbf{y}_{t + 1}$
User: I am interested in playing Table tennis.
Engaging: Provides no information about table tennis or how to play it.
Response (refined): That's great to hear (...)! It's a fun sport requiring quick reflexes and good hand-eye coordination.
Response: I'm sure it's a great way to socialize, stay active
User understanding: Lacks understanding of user's needs and state of mind.
Have you played before, or are you looking to learn?
(d) Code optimization:  $x, y_t$
(e) FEEDBACK fb
(f)REFINE  $\mathbf{y}_{t + 1}$
Generate sum of 1, ..., N
def sum(n):
    res = 0
    for i in range(n+1):
        res += i
    return res
This code is slow as it uses brute force. A better approach is to use the formula ...  $(n(n + 1)) / 2$
Code (refined)
def sum_faster(n):
    return (n*(n+1))//2
Figure 2: Examples of SELF-REFINE: an initial output generated by the base LLM and then passed back to the same LLM to receive feedback to the same LLM to refine the output. The top row illustrates this for dialog generation where an initial dialogue response can be transformed into a more engaging one that also understands the user by applying feedback. The bottom row illustrates this for code optimization where the code is made more efficient by applying feedback.
Algorithm 1 SELF-REFINE algorithm
Require: input  $x$ , model  $\mathcal{M}$ , prompts  $\{p_{\mathrm{gen}}, p_{\mathrm{fb}}, p_{\mathrm{refine}}\}$ , stop condition stop( $\cdot$ )  
1:  $y_0 = \mathcal{M}(p_{\mathrm{gen}} \| x)$  ▷ Initial generation (Eqn. 1)  
2: for iteration  $t \in \{0, 1, \ldots\}$  do  
3:  $fb_t = \mathcal{M}(p_{\mathrm{fb}} \| x \| y_t)$  ▷ Feedback (Eqn. 2)  
4: if stop  $(fb_t, t)$  then ▷ Stop condition  
5: break  
6: else  
7:  $y_{t+1} = \mathcal{M}(p_{\mathrm{refine}} \| x \| y_0 \| fb_0 \| \ldots \| y_t \| fb_t)$  ▷ Refine (Eqn. 4)  
8: end if  
9: end for  
10: return  $y_t$
Figure 3: The SELF-REFINE algorithm. See (§2) for a discussion of each component.
For example, in Figure 2(d), the model generates functionally correct code for the given input. Here,  $p_{\mathrm{gen}}$  is a task-specific few-shot prompt (or instruction) for an initial generation, and  $\| \cdot \|$  denotes concatenation. The few-shot prompt contains input-output pairs  $\langle x^{(k)}, y^{(k)} \rangle$  for the task.
FEEDBACK Next, SELF-REFINE uses the same model  $\mathcal{M}$  to provide feedback  $fb_{t}$  on its own output, given a task-specific prompt  $p_{\mathrm{fb}}$  for generating feedback:
$$
f b _ {t} = \mathcal {M} \left(p _ {\mathrm {f b}} \| x \| y _ {t}\right). \tag {2}
$$
Intuitively, the feedback may address multiple aspects of the output. For example, in code optimization, the feedback might address the efficiency, readability, and overall quality of the code.
<sup>3</sup>Few-shot prompting (also referred to as "in-context learning") provides a model with a prompt consisting of  $k$  in-context examples of the target task, each in the form of input-output pairs  $\langle x_i, y_i \rangle$  (Brown et al., 2020).
Here, the prompt  $p_{\mathrm{fb}}$  provides examples of feedback in the form of input-output-feedback triples  $\langle x^{(k)}, y^{(k)}, fb^{(k)} \rangle$ . We prompt the model to write feedback that is actionable and specific via  $fb^{(k)}$ . By 'actionable', we mean the feedback should contain a concrete action that would likely improve the output. By 'specific', we mean the feedback should identify concrete phrases in the output to change. For example, the feedback in Figure 2(e) is "This code is slow as it uses a for loop which is brute force. A better approach is to use the formula ... (n(n+1))/2". This feedback is actionable, since it suggests the action 'use the formula...' The feedback is specific since it mentions the 'for loop'.
REFINE Next, SELF-REFINE uses  $\mathcal{M}$  to refine its most recent output, given its own feedback:
$$
y _ {t + 1} = \mathcal {M} \left(p _ {\text {r e f i n e}} \| x \| y _ {t} \| f b _ {t}\right). \tag {3}
$$
For example, in Figure 2(f), given the initial output and the generated feedback, the model generates a re-implementation that is shorter and runs much faster than the initial implementation. The prompt  $p_{\mathrm{refine}}$  provides examples of improving the output based on the feedback, in the form of input-output-feedback-refined quadruples  $\langle x^{(k)}, y_t^{(k)}, fb_t^{(k)}, y_{t+1}^{(k)} \rangle$ .
Iterating SELF-REFINE SELF-REFINE alternates between FEEDBACK and REFINE steps until a stopping condition is met. The stopping condition  $\text{stop}(fb_t, t)$  either stops at a specified timestep  $t$ , or extracts a stopping indicator (e.g., a scalar stop score) from the feedback. In practice, the model can be prompted to generate a stopping indicator in  $p_{\mathrm{fb}}$ , and the condition is determined per-task.
To inform the model about the previous iterations, we retain the history of previous feedback and outputs by appending them to the prompt. Intuitively, this allows the model to learn from past mistakes and avoid repeating them. More precisely, Equation (3) is in fact instantiated as:
$$
y _ {t + 1} = \mathcal {M} \left(p _ {\text {r e f i n e}} \| x \| y _ {0} \| f b _ {0} \| \dots \| y _ {t} \| f b _ {t}\right). \tag {4}
$$
Finally, we use the last refinement  $y_{t}$  as the output of SELF-REFINE.
Algorithm 1 summarizes SELF-REFINE, and Figure 2 shows an example of SELF-REFINE in the Dialogue Response Generation (Mehri and Eskenazi, 2020) and Code Optimization (Madaan et al., 2023) tasks. Appendix V provides examples of the  $p_{\mathrm{gen}}$ ,  $p_{\mathrm{fb}}$ ,  $p_{\mathrm{refine}}$  prompts for various tasks. The key idea is that SELF-REFINE uses the same underlying LLM to generate, get feedback, and refine its outputs given its own feedback. It relies only on supervision present in the few-shot examples.
We evaluate SELF-REFINE on 7 diverse tasks: Dialogue Response Generation (Appendix P; Mehri and Eskenazi, 2020), Code Optimization (Appendix Q; Madaan et al., 2023), Code Readability Improvement (Appendix O; Puri et al., 2021), Math Reasoning (Appendix R; Cobbe et al., 2021), Sentiment Reversal (Appendix S; Zhang et al., 2015), and we introduce two new tasks: Acronym Generation (Appendix T) and Constrained Generation (a harder version of Lin et al. (2020) with 20-30 keyword constraints instead of 3-5; Appendix U)
Examples for all tasks and dataset statistics are provided in Table 4 (Appendix A).
3.1 Instantiating SELF-REFINE
We instantiate SELF-REFINE following the high-level description in Section 2. The FEEDBACK-REFINE iterations continue until the desired output quality or task-specific criterion is reached, up to a maximum of 4 iterations. To make our evaluation consistent across different models, we implemented both FEEDBACK and REFINE as few-shot prompts even with models that respond well to instructions, such as CHATGPT and GPT-4.
Base LLMs Our main goal is to evaluate whether we can improve the performance of any strong base LLMs using SELF-REFINE. Therefore, we compare SELF-REFINE to the same base LLMs but without feedback-refine iterations. We used three main strong base LLM across all tasks: GPT-3.5 (text-davinci-003), CHATGPT (gpt-3.5-turbo), and GPT-4 (OpenAI, 2023). For code-based tasks, we also experimented with CODEX (code-davinci-002). In all tasks, either GPT-3.5 or GPT-4 is the previous state-of-the-art. We used the same prompts from previous work when available
4A comparison with other few-shot and fine-tuned approaches is provided in Appendix H
Task	GPT-3.5	CHATGPT	GPT-4
Base	+SELF-REFINE	Base	+SELF-REFINE	Base	+SELF-REFINE
Sentiment Reversal	8.8	30.4 (↑21.6)	11.4	43.2 (↑31.8)	3.8	36.2 (↑32.4)
Dialogue Response	36.4	63.6 (↑27.2)	40.1	59.9 (↑19.8)	25.4	74.6 (↑49.2)
Code Optimization	14.8	23.0 (↑8.2)	23.9	27.5 (↑3.6)	27.3	36.0 (↑8.7)
Code Readability	37.4	51.3 (↑13.9)	27.7	63.1 (↑35.4)	27.4	56.2 (↑28.8)
Math Reasoning	64.1	64.1 (0)	74.8	75.0 (↑0.2)	92.9	93.1 (↑0.2)
Acronym Generation	41.6	56.4 (↑14.8)	27.2	37.2 (↑10.0)	30.4	56.0 (↑25.6)
Constrained Generation	16.0	39.7 (↑23.7)	2.75	33.5 (↑30.7)	4.4	61.3 (↑56.9)
Table 1: SELF-REFINE results on various tasks using GPT-3.5,chatGPT,and GPT-4 as base LLM. SELF-REFINE consistently improves LLM. Metrics used for these tasks are defined in Section 3.2.
(such as for Code Optimization and Math Reasoning); otherwise, we created prompts as detailed in Appendix V. We generate samples using a temperature of 0.7.
3.2 Metrics
We report three types of metrics:
- Task specific metric: When available, we use automated metrics from prior work (Math Reasoning: % solve rate; Code Optimization: % programs optimized%).
- GPT-4-ref: In addition to human-ref, we use GPT-4 as a proxy for human preference following prior work (Fu et al., 2023; Chiang et al., 2023; Geng et al., 2023; Sun et al., 2023), and found high correlation (82% for Sentiment Reversal, 68% for Acronym Generation, and 71% for Dialogue Response Generation) with human-ref. For Code Readability Improvement, we prompt GPT-4 to calculate fraction of the variables that are appropriately named given the context (e.g.,  $x = [] \rightarrow input_buffer = []$ ). Additional details are provided in Appendix F. For constrained generation, we combine automated evaluation to quantify concept coverage and GPT-4-ref to ensure the commonsense correctness of generated sentences. A sentence is only deemed a winner if it maintains validity in commonsense reasoning and has greater coverage in terms of concepts.
- Human evaluation: In Dialogue Response Generation, Code Readability Improvement, Sentiment Reversal, and Acronym Generation, we additionally perform a blind human A/B evaluation on a subset of the outputs to select the preferred output. Additional details are provided in Appendix C.
Table 1 shows our main results:
SELF-REFINE consistently improves over base models across all model sizes, and additionally outperforms the previous state-of-the-art across all tasks. For example, GPT-4+SELF-REFINE improves over the base GPT-4 by  $8.7\%$  (absolute) in Code Optimization, increasing optimization percentage from  $27.3\%$  to  $36.0\%$ . Confidence intervals are provided in Appendix M. For code-based tasks, we found similar trends when using CODEX; those results are included in Appendix H.
One of the tasks in which we observe the highest gains compared to the base models is Constrained Generation, where the model is asked to generate a sentence containing up to 30 given concepts. We believe that this task benefits significantly from SELF-REFINE because there are more opportunities to miss some of the concepts on the first attempt, and thus SELF-REFINE allows the model to fix these mistakes subsequently. Further, this task has an extremely large number of reasonable outputs, and thus SELF-REFINE allows to better explore the space of possible outputs.
In preference-based tasks such as Dialogue Response Generation, Sentiment Reversal, and Acronym Generation, SELF-REFINE leads to especially high gains. For example in Dialogue Response Generation, GPT-4 preference score improve by  $49.2\%$  - from  $25.4\%$  to  $74.6\%$ . Similarly, we see remarkable improvements in the other preference-based tasks across all models.
The modest performance gains in Math Reasoning can be traced back to the inability to accurately identify whether there is any error. In math, errors can be nuanced and sometimes limited to a single line or incorrect operation. Besides, a consistent-looking reasoning chain can deceive LLMs to think that "everything looks good" (e.g.,chatGPT feedback for  $94\%$  instances is 'everything looks good'). In Appendix K.1, we show that the gains with SELF-REFINE on Math Reasoning are much bigger  $(5\% +)$  if an external source can identify if the current math answer is incorrect. Although SELF-REFINE demonstrates limited efficacy in Math Reasoning, we observe gains with SELF-REFINE in a subset of Big-Bench Hard (Suzgun et al., 2022) tasks that typically require a combination of commonsense reasoning and logic, such as date reasoning (Appendix D). This suggests that SELF-REFINE may be more effective in scenarios where the interplay of logical analysis and knowledge acquired through pre-training facilitates self-verification.
Improvement is consistent across base LLMs sizes. Generally, GPT-4+SELF-REFINE performs better than GPT-3.5+SELF-REFINE and CHATGPT+SELF-REFINE across all tasks, even in tasks where the initial base results of GPT-4 were lower than GPT-3.5 or CHATGPT. We thus believe that SELF-REFINE allows stronger models (such as GPT-4) to unlock their full potential, even in cases where this potential is not expressed in the standard, single-pass, output generation. Comparison to additional strong baselines is provided in Appendix H.
The three main steps of SELF-REFINE are FEEDBACK, REFINE, and repeating them iteratively. In this section, we perform additional experiments to analyze the importance of each of these steps.
Task	SELF-REFINE feedback	Generic feedback	No feedback
Code Optimization	27.5	26.0	24.8
Sentiment Reversal	43.2	31.2	0
Acronym Generation	56.4	54.0	48.0
Table 2: Prompting to generate generic feedback (or having the model generate no feedback at all) leads to reduced scores, indicating the importance of the FEEDBACK step of SELF-REFINE. These experiments were performed with CHATGPT (Code Optimization and Sentiment Reversal) and GPT-3.5 (Acronym Generation), and metrics used are defined in Section 3.2.
The impact of the feedback quality Feedback quality plays a crucial role in SELF-REFINE. To quantify its impact, we compare SELF-REFINE, which utilizes specific, actionable feedback, with two ablations: one using generic feedback and another without feedback (the model may still iteratively refine its generations, but is not explicitly provided feedback to do so). For example, in the Code Optimization task: actionable feedback, such as Avoid repeated calculations in the for loop, pinpoints an issue and suggests a clear improvement. Generic feedback, like Improve the efficiency of the code, lacks this precision and direction. Table 2 shows feedback's clear influence.
In Code Optimization, performance slightly dips from 27.5 (SELF-REFINE feedback) to 26.0 (generic feedback), and further to 24.8 (no feedback). This suggests that while generic feedback offers some guidance – specific, actionable feedback yields superior results.
This effect is more pronounced in tasks like Sentiment Reversal, where changing from our feedback to generic feedback leads to a significant performance drop (43.2 to 31.2), and the task fails without feedback. In the "No feedback" setting, the model was not given clear instructions on changing the output. We find that the model tends to either repeat the same output in each iteration or to make unrelated changes. Since the scores in this task are the relative improvement increase in human preference, a score of 0 means that "No feedback" did not improve over the base model outputs. Similarly, in Acronym Generation, without actionable feedback, performance drops from 56.4 to 48.0, even with iterative refinements. These results highlight the importance of specific, actionable feedback in our approach. Even generic feedback provides some benefit, but the best results are achieved with targeted, constructive feedback.
How important are the multiple iterations of FEEDBACK-REFINE? Figure 4 demonstrates that on average, the quality of the output improves as the number of iterations increases. For instance, in
Task	y0	y1	y2	y3
C. Opt.	22.0	27.0	27.9	28.8
S. Rev.	33.9	34.9	36.1	36.8
C. Gen.	12.1	26.1	39.6	46.1
Figure 4: Left: Iteration-wise score improvements. Early iterations significantly improve output quality, and scores generally keep improving with more iterations. Right: SELF-REFINE Performance improvements with iterations. Most gains  $(\Delta)$  are in the initial iterations for both Code Opt. and Sentiment Reversal. The numbers are averaged over CHATGPT, GPT-3.5, and GPT-4. Task abbreviations: C. Opt. (Code Optimization), S. Rev. (Sentiment Reversal), C. Gen. (Constrained Generation).
Figure 5: Comparison of code generated by Madaan et al. (2023) (left) and the output after applying SELF-REFINE (right). The initial code by the baseline, which is nearly identical to the slower input program, fails to improve the efficiency and merely alters the logic for reading input. SELF-REFINE first generates feedback that diagnoses that This code is slow because it is using six nested loops to iterate through all possible combinations of coins to pay the amount, and suggests that a more efficient approach would be ... SELF-REFINE then uses this feedback to generate the revised code (right), reducing the time complexity to  $\mathcal{O}(amount * coins)$ . The full example is provided in Appendix K
Slower code   
def solve(amount): best_price  $=$  (amount  $+199$  //200\*  $\rightarrow$  380 #First loop for a in range(amount//200+1): #...4 nested loops... for c1 in range(amount//1500  $^+$ $\rightarrow$  1): if  $\mathbf{a}^{*}200 + \mathbf{b}^{*}300 = =$  amount: price  $= \mathrm{a}^{*}380 + \mathrm{b}^{*}550$  if price  $<  <$  best_price: best_price  $=$  price return best_price