SlowGuess/NeurIPS_2023_OCR huggingface dataset:
# Faster code
def solve(amount):
    coins = [200, 300]
    prices = [380, 550]
    dp = [float('inf')] * (amount + 1)
    dp[0] = 0
    for i in range(len(coins)):
        for j in range(coins[i], amount + 1):
            dp[j] = min(dp[j], dp[j - coins[i] + prices[i])
    return dp[amount]
the Code Optimization task, the initial output  $(y_0)$  has a score of 22.0, which improves to 28.8 after three iterations  $(y_{3})$ . Similarly, in the Sentiment Reversal task, the initial output has a score of 33.9, which increases to 36.8 after three iterations. This trend of improvement is also evident in Constrained Generation, where the score increases from 26.1 to 46.1 after three iterations. Figure 4 highlights the diminishing returns in the improvement as the number of iterations increases. Overall, having multiple FEEDBACK-REFINE iterations significantly enhances the quality of the output, although the marginal improvement naturally decreases with more iterations.
The performance may not always monotonically increase with iterations: in multi-aspect feedback tasks like Acronym Generation, where the output quality can vary during iteration with improvement
in one aspect but decline in another aspect. To counter this, SELF-REFINE generates numerical scores for different quality aspects, leading to a balanced evaluation and appropriate output selection.
Can we just generate multiple outputs instead of refining? Does SELF-REFINE improve because of the iterative refinement, or just because it generates more outputs? We compare SELF-REFINE with CHATGPT, when CHATGPT generates  $k = 4$  samples (but without feedback and refinement). Then, we compare the performance of SELF-REFINE against these  $k$  initial outputs in a 1 vs.  $k$  evaluation. In other words, we assess whether SELF-REFINE can outperform all  $k$  initial outputs. The results of this experiment are illustrated in Figure 7 (Appendix K). Despite the increased difficulty of the 1 vs.  $k$  setting, the outputs of SELF-REFINE are still preferred by humans over all  $k$  initial outputs. This shows the importance of refinement according to feedback over the alternative of just generating multiple initial outputs.
Does SELF-REFINE works in an instruction only setup? In our main experiments, we use few-shot prompting to guide model output into a more readily parseable format. Next, we experiment with SELF-REFINE under a zero-shot prompting scenario, where traditional few-shot examples are supplanted by explicit instructions at each stage of the SELF-REFINE process. For these experiments, we usechatGPT. The results (Appendix E in Table 8) show that SELF-REFINE remains effective across diverse tasks, even in the absence of example prompts. Notably, in tasks such as Acronym Generation and Sentiment Reversal, SELF-REFINE, under zero-shot prompting, enhances performance from  $16.6\%$  to  $44.8\%$  and  $4.4\%$  to  $71.4\%$ , respectively. However, achieving optimal performance in this setting requires extensive prompt engineering for instructions.
For Math Reasoning tasks, SELF-REFINE improves the solve rate from  $22.1\%$  to  $59.0\%$  in an instruction-only setting. We find that much of this gain comes from fixing omitted return statements in  $71\%$  of the initial Python programs, despite clear instructions to include them. Subsequent iterations of feedback generation and refinement address this issue effectively, decreasing the error rate by  $19\%$ . Further, we find that when the initial programs are valid, SELF-REFINE does not improve the performance.
Does SELF-REFINE work with weaker models? The experiments in Section 3.3 were performed with some of the strongest available models; does SELF-REFINE work with smaller or weaker models as well? To investigate this, we instantiated SELF-REFINE with Vicuna-13B (Chiang et al., 2023), a less powerful base model. While Vicuna-13B is capable of generating initial outputs, it struggles significantly with the refinement process. Specifically, Vicuna-13B was not able to consistently generate the feedback in the required format. Furthermore, even when provided with Oracle or hard-coded feedback, it often failed to adhere to the prompts for refinement. Instead of refining its output, Vicuna-13B either repeated the same output or generated a hallucinated conversation, rendering the outputs less effective. Example output and analysis is provided in Appendix I.
How does SELF-REFINE perform with strong open access models like LLAMA2-70B? We conduct additional experiments using SELF-REFINE on LLama-2 (Touvron et al., 2023), a state-of-the-art, open-access language model on Acronym Generation, Sentiment Reversal, Dialogue Response Generation, and Math Reasoning. Consistent with our primary findings, SELF-REFINE shows an improvement across all these tasks relative to the base model. The full results are shown in Appendix J. These promising results with LLAMA2-70B suggest that the applicability of SELF-REFINE might extend to a wide array of increasingly powerful open-source models in the future
Qualitative Analysis We conduct a qualitative analysis of the feedback generated by SELF-REFINE and its subsequent refinements. We manually analyze 70 samples in total (35 success cases and 35 failure cases) for Code Optimization (Madaan et al., 2023) and Math Reasoning (Cobbe et al., 2021). For both Math Reasoning and Code Optimization, we found that the feedback was predominantly actionable, with the majority identifying problematic aspects of the original generation and suggesting ways to rectify them.
When SELF-REFINE failed to improve the original generation, the majority of issues were due to erroneous feedback rather than faulty refinements. Specifically,  $33\%$  of unsuccessful cases were due to feedback inaccurately pinpointing the error's location, while  $61\%$  were a result of feedback suggesting an inappropriate fix. Only  $6\%$  of failures were due to the refiner incorrectly implementing good feedback. These observations highlight the vital role of accurate feedback plays in SELF-REFINE.
Supervision-free refiner	Supervision-free feedback	Multi-aspect feedback	Iterative
Learned refiners: PEER (Schick et al., 2022b), Self-critique (Saunders et al., 2022b), CodeRL (Le et al., 2022b), Self-correction (Welleck et al., 2022).	X	✓ or X	X	✓ or X
Prompted refiners: Augmenter (Peng et al., 2023), Re3 (Yang et al., 2022), Reflexion (Shinn et al., 2023).	✓	✓ or X	X	X
SELF-REFINE (this work)	✓	✓	✓	✓
Table 3: A comparison of SELF-REFINE to closely related prior refinement approaches.
In successful cases, the refiner was guided by accurate and useful feedback to make precise fixes to the original generation in  $61\%$  of the cases. Interestingly, the refiner was capable of rectifying issues even when the feedback was partially incorrect, which was the situation in  $33\%$  of successful cases. This suggests resilience to sub-optimal feedback. Future research could focus on examining the refiner's robustness to various types of feedback errors and exploring ways to enhance this resilience. In Figure 5, we illustrate how SELF-REFINE significantly improves program efficiency by transforming a brute force approach into a dynamic programming solution, as a result of insightful feedback. Additional analysis on other datasets such as Dialogue Response Generation is provided in Appendix K.
Going Beyond Benchmarks While our evaluation focuses on benchmark tasks, SELF-REFINE is designed with broader applicability in mind. We explore this in a real-world use case of website generation, where the user provides a high-level goal and SELF-REFINE assists in iteratively developing the website. Starting from a rudimentary initial design, SELF-REFINE refines HTML, CSS, and JS to evolve the website in terms of both usability and aesthetics. This demonstrates the potential of SELF-REFINE in real-world, complex, and creative tasks. See Appendix L for examples and further discussion, including broader, societal impact of our work.
Leveraging human- and machine-generated natural language (NL) feedback for refining outputs has been effective for a variety of tasks, including summarization (Scheurer et al., 2022), script generation (Tandon et al., 2021), program synthesis (Le et al., 2022a; Yasunaga and Liang, 2020), and other tasks (Madaan et al., 2022; Bai et al., 2022a; Schick et al., 2022b; Saunders et al., 2022a; Bai et al., 2022b; Welleck et al., 2022). Refinement methods differ in the source and format of feedback, and the way that a refiner is obtained. Table 3 summarizes some related approaches; see Appendix B for an additional discussion.
Source of feedback. Humans have been an effective source of feedback (Tandon et al., 2021; Elgohary et al., 2021; Tandon et al., 2022; Bai et al., 2022a). Since human feedback is costly, several approaches use a scalar reward function as a surrogate of (or alternative to) human feedback (e.g., (Bai et al., 2022a; Liu et al., 2022; Lu et al., 2022; Le et al., 2022a; Welleck et al., 2022)). Alternative sources such as compilers (Yasunaga and Liang, 2020) or Wikipedia edits (Schick et al., 2022b) can provide domain-specific feedback. Recently, LLMs have been used to generate feedback for general domains (Fu et al., 2023; Peng et al., 2023; Yang et al., 2022). However, ours is the only method that generates feedback using an LLM on its own output, for the purpose of refining with the same LLM.
Representation of feedback. The form of feedback can be generally divided into natural language (NL) and non-NL feedback. Non-NL feedback can come in human-provided example pairs (Dasgupta et al., 2019) or scalar rewards (Liu et al., 2022; Le et al., 2022b). In this work, we use NL feedback, since this allows the model to easily provide self-feedback using the same LM that generated the output, while leveraging existing pretrained LLMs such as GPT-4.
Types of refiners. Pairs of feedback and refinement have been used to learn supervised refiners (Schick et al., 2022b; Du et al., 2022; Yasunaga and Liang, 2020; Madaan et al., 2021). Since
gathering supervised data is costly, some methods learn refiners using model generations (Welleck et al., 2022; Peng et al., 2023). However, the refiners are trained for each new domain. Finally, (Yang et al., 2022) use prompted feedback and refinement specifically tailored for story generation. In this work, we avoid training a separate refiner, and show that the same model can be used as both the refiner and the source of feedback across multiple domains.
Non-refinement reinforcement learning (RL) approaches. Rather than having explicit refinement, an alternative way to incorporate feedback is by optimizing a scalar reward function, e.g. with reinforcement learning (e.g., Stiannon et al. (2020); Lu et al. (2022); Le et al. (2022a)). These methods differ from SELF-REFINE in that the model does not access feedback on an intermediate generation. Second, these RL methods require updating the model's parameters, unlike SELF-REFINE. Recently, in discrete-space simulated environments, LLMs have also been shown to iteratively shape and refine rewards and policies, thereby performing RL tasks without expert demonstrations or gradients (Kim et al., 2023; Brooks et al., 2023). While we focus on real-world code and language tasks in this paper, it would be interesting to explore applications of self-refine in simulated environments.
6 Limitations and Discussion
The main limitation of our approach is that the base models need to have sufficient few-shot modeling or instruction-following abilities, in order to learn to provide feedback and to refine in an in-context fashion, without having to train supervised models and rely on supervised data.
Further, the experiments in this work were primarily performed with language models that are not open-sourced, namely GPT-3.5, CHATGPT, GPT-4, and CODEX. Existing literature (Ouyang et al., 2022) does not fully describe the details of these models, such as the pretraining corpus, model sizes, and model biases. Nonetheless, we release our code and model outputs to ensure the reproducibility of our work. In addition, initial results from our experiments with the open-access LLAMA2-70B language model are promising, reinforcing the notion that SELF-REFINE has the potential to be widely applicable, even as open-source models continue to evolve and improve.
Another limitation of our work is that we exclusively experiment with datasets in English. In other languages, the current models may not provide the same benefits. Finally, there is a possibility for bad actors to use prompting techniques to steer a model to generate more toxic or harmful text. Our approach does not explicitly guard against this.
We present SELF-REFINE: a novel approach that allows large language models to iteratively provide self-feedback and refine their own outputs. SELF-REFINE operates within a single LLM, requiring neither additional training data nor reinforcement learning. We demonstrate the simplicity and ease of use of SELF-REFINE across a wide variety of tasks. By showcasing the potential of SELF-REFINE in diverse tasks, our research contributes to the ongoing exploration and development of large language models, with the aim of reducing the cost of human creative processes in real-world settings. We hope that our iterative approach will help drive further research in this area. To this end, we make all our code, data and prompts available at https://selfrefine.info/.
We thank the anonymous reviewers for their useful comments and suggestions, and extend our thanks to James Laudon, Cliff Young, Graham Neubig, Daniel Fried, Rif A. Saurous, and the Google DeepMind team for their valuable feedback and comments. This material is partly based on research sponsored in part by the Air Force Research Laboratory (agreement number FA8750-19-2-0200). The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of DARPA or the U.S. Government.
Mirac Suzgun, Nathan Scales, Nathanael Scharli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. 2022. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261.
Niket Tandon, Aman Madaan, Peter Clark, Keisuke Sakaguchi, and Yiming Yang. 2021. Interscript: A dataset for interactive learning of scripts through error feedback. arXiv preprint arXiv:2112.07867.
Niket Tandon, Aman Madaan, Peter Clark, and Yiming Yang. 2022. Learning to repair: Repairing model output errors after deployment using a dynamic memory of feedback. In *Findings of the Association for Computational Linguistics: NAACL* 2022, pages 339–352.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain of Thought Prompting Elicits Reasoning in Large Language Models. arXiv preprint arXiv:2201.11903.
Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. 2022. Generating sequences by learning to self-correct. arXiv preprint arXiv:2211.00053.
Kevin Yang, Nanyun Peng, Yuandong Tian, and Dan Klein. 2022. Re3: Generating longer stories with recursive reprompting and revision. In Conference on Empirical Methods in Natural Language Processing.
Michihiro Yasunaga and Percy Liang. 2020. Graph-based, self-supervised program repair from diagnostic feedback. 37th Int. Conf. Mach. Learn. ICML 2020, PartF168147-14:10730-10739.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28.
A Evaluation Tasks
Table 4 lists the tasks in our evaluation, and examples from each task.
Task and Description	Sample one iteration of FEEDBACK-REFINE
Sentiment Reversal	x: The food was fantastic...”
Rewrite reviews to reverse sentiment.	yt: The food was disappointing...”
Dataset: (Zhang et al., 2015) 1000 review pas-sages	fb: Increase negative sentiment
yt+1: The food was utterly terrible...”
Dialogue Response Generation	x: What's the best way to cook pasta?”
Produce rich conversational responses.	yt: The best way to cook pasta is to...”
Dataset: (Mehri and Eskenazi, 2020) 372 conv.	fb: Make response relevant, engaging, safe
yt+1: Boil water, add salt, and cook pasta...”
Code Optimization	x: Nested loop for matrix product
Enhance Python code efficiency	yt: NumPy dot product function
Dataset: (Madaan et al., 2023): 1000 programs	fb: Improve time complexity
yt+1: Use NumPy's optimized matmul function
Code Readability Improvement	x: Unclear variable names, no comments
Refactor Python code for readability.	yt: Descriptive names, comments
Dataset: (Puri et al., 2021) 300 programs*	fb: Enhance variable naming; add comments
yt+1: Clear variables, meaningful comments
Math Reasoning	x: Olivia has $23, buys 5 bagels at $3 each”
Solve math reasoning problems.	yt: Solution in Python
Dataset: (Cobbe et al., 2021) 1319 questions	fb: Show step-by-step solution
yt+1: Solution with detailed explanation
Acronym Generation	x: Radio Detecting and Ranging”
Generate acronyms for a given title	yt: RDR
Dataset: (Appendix T) 250 acronyms	fb: be context relevant; easy pronunciation
yt+1: RADAR”
Constrained Generation	x: beach, vacation, relaxation
Generate sentences with given keywords.	yt: During our beach vacation...
Dataset: (Lin et al., 2020) 200 samples	fb: Include keywords; maintain coherence
yt+1: ... beach vacation was filled with relaxation
Table 4: An overview of the tasks which we evaluate SELF-REFINE on, along with their associated datasets and sizes. For every task, we demonstrate a single iteration of refinement of input  $x$ , the previously generated output  $y_{t}$ , the feedback generated  $fb_{t}$ , and the refinement  $y_{t+1}$ . Few-shot prompts used for FEEDBACK and REFINE are provided in Appendix V.
B Broader Related Work
Compared to a concurrent work, Reflexion (Shinn et al., 2023), our approach involves correction using feedback, whereas their setup involves finding the next best solution in planning using ReAct. While ReAct and Reflexion provide a free-form reflection on whether a step was executed correctly and potential improvements, our approach is more granular and structured, with multi-dimensional feedback and scores. This distinction allows our method to offer more precise and actionable feedback, making it suitable for a wider range of natural language generation tasks, including those that may not necessarily involve step-by-step planning such as open-ended dialogue generation.
Comparison with Welleck et al. (2022) The closest work to ours may be Self-Correction (Welleck et al., 2022); however, Self-Correction has several disadvantages compared to SELF-REFINE:
1. Self-Correction does not train their model to generate explicit feedback; instead, Welleck et al. (2022) trained their models to refine only. As we show in Section 4 and Table 2, having the model generate explicit feedback results in significantly better refined outputs.
2. Self-Correction trains a separate refiner (or "corrector") for each task. In contrast, SELF-REFINE uses instructions and few-shot prompting, and thus does not require training a separate refiner for each task.
3. Empirically, we evaluated SELF-REFINE using the same base model of GPT-3 as Self-Correction, and with the same settings on the GSM8K benchmark. Self-Correction achieved  $45.9\%$  accuracy while SELF-REFINE (this work) achieved  $55.7\%$  ( $\uparrow 9.8$ ).
Comparison with non-refinement reinforcement learning (RL) approaches. Rather than having an explicit refinement module, an alternative way to incorporate feedback is by optimizing a scalar reward function, e.g. with reinforcement learning (e.g., Stiennon et al. (2020); Lu et al. (2022); Le et al. (2022a)). These methods differ from SELF-REFINE (and more generally, refinement-based approaches) in that the model cannot access feedback on an intermediate generation. Second, these reinforcement learning methods require updating the model's parameters, unlike SELF-REFINE.
See Table 5 for an additional detailed comparison of related work.
Method	Primary Novelty	zero/few shot improvement	multi aspect critics	NL feedback with er- ror localization	iterative framework
RLHF (Stiannon et al., 2020)	optimize for human preference	\( ✘ \)trained on feedback	\( ✘ \)single (human)	\( ✘ \)not self gen.)	\( ✘ \)
Rainer RL (Liu et al., 2022)	RL to generate knowledge	\( ✘ \)trained on end task	\( ✘ \)single(accuracy)	\( ✘ \)(knowl. only)	\( ✘ \)
QUARK RL (Lu et al., 2022)	quantization to edit generations	\( ✘ \)trained on end task	\( ✘ \)single(scalar score)	\( ✘ \)(dense signal)	\( ✘ \)(train time iter.)
Code RL (Le et al., 2022a)	actor critic RL for code im- provement	\( ✘ \)trained on end task	\( ✘ \)single(unit tests)	\( ✘ \)(dense signal)	\( ✘ \)
DrRepair (Yasunaga and Liang, 2020)	Compiler feedback to itera- tively repair	\( ✘ \) trained semi sup.	\( ✘ \)single(compiler msg)	\( ✘ \) not self gen.)	\( ✘ \)
PEER (Schick et al., 2022b)	doc. edit trained on wiki edits	\( ✘ \) trained on edits	\( ✘ \)single(accuracy)	\( ✘ \) not self gen.)	\( ✘ \)
Self critique (Saunders et al., 2022a)	few shot critique generation	\( ✘ \) feedback training	\( ✘ \) single(human)	\( ✘ \) self gen.)	\( ✘ \)
Self-correct (Welleck et al., 2022)	novel training of a corrector	\( ✘ \) trained on end task	\( ✘ \) single (task specific)	\( ✘ \)(limited setting)	\( ✘ \)(limited setting)
Const. AI (Bai et al., 2022b)	train RL4F on automat (cri- tique, revision) pair	\( ✘ \) critique training	\( ✘ \)(fixed set)	\( ✘ \)
Self-ask (Press et al., 2022)	ask followup ques when in- terim ans correct;final wrong	\( ✘ \) few shot	\( ✘ \)none	\( ✘ \)(none)	\( ✘ \)
GPT3 score (Fu et al., 2023)	GPT can score generations with instruction	\( ✘ \)few shot	\( ✘ \)single(single utility fn)	\( ✘ \)(none)	\( ✘ \)
Augmenter (Peng et al., 2023)	factuality feedback from exter- nal KBs	\( ✘ \)few shot	\( ✘ \)single(factuality)	\( ✘ \) self gen.)	\( ✘ \)
\( Re^3 \) (Yang et al., 2022)	~ours: but one domain, trained critics	\( ✘ \)few shot	\( ✘ \) trained critics)	\( ✘ \) not self gen.)	\( ✘ \)
SELF-REFINE	fewshot iterative multi aspect NL fb	\( ✘ \)few shot	\( ✘ \) multiple(few shot critics)	\( ✘ \) self gen.)	\( ✘ \)
Table 5: Summary of related approaches. Reinforcement learning approaches are shown in purple
, trained corrector approaches are shown in orange , and few-shot corrector approaches are shown in green .
C Human Evaluation
The A/B evaluation in our study was conducted by the authors, where a human judge was presented with an input, task instruction, and two candidate outputs generated by the baseline method and SELF-REFINE. The setup was blind, i.e., the judges did not know which outputs were generated by which method. The judge was then asked to select the output that is better aligned with the task instruction. For tasks that involve A/B evaluation, we calculate the relative improvement as the percentage increase in preference rate. The preference rate represents the proportion of times annotators selected the output produced by SELF-REFINE over the output from the baseline method. Table 6 shows the results.
While multiple annotators participated in each task, we only collected a single annotation per instance, aiming to scale the number of data points we could annotate. To further validate the reliability of our evaluations, we obtained two annotations for 50 samples from each dataset. All human evaluations were executed in a double-blind manner, with the responses being randomly flipped to ensure annotator impartiality, leaving them unaware of whether a given output was from the base model or the SELF-REFINE. These additional evaluations were exclusively applied to outputs generated by GPT-4.