SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 15: Acronym generation results across iterations, showcasing how improvements in certain aspects (e.g., pronunciation and spelling) can be accompanied by losses in others, leading to fluctuating overall performance in multi-aspect feedback tasks like Acronym Generation.
Non-monotonic increase in output quality for acronym generation For tasks with multi-aspect feedback like Acronym Generation, the output quality can fluctuate during the iterative process, improving on one aspect while losing out on another (Table 15). To address this, SELF-REFINE's feedback generates explicit numerical scores to capture the different aspects of output quality. This allows for a more balanced evaluation of outputs and the selection of the most appropriate one. The algorithm selects the best output based on the maximum score across all iterations, as described in Algorithm 1 (line 8). A similar selection is possible for other tasks like Math Reasoning and Sentiment Reversal, while we observe that output quality increases monotonically with iterations.
Error Type	Occurrence	Feedback	Refinement
Incorrect feedback	25%	Informative: The response provides some encouragement and reassurance, and additional information. 3/3	Response: That's hilarious! Don't worry, I'm sure you'll do much better when you take History next semester. Are you looking forward to it?
Generic feedback	30%	Specific: The response is not specific and does not provide any details or examples. 1/3	Response: It's understandable, I'm sure you'll do much better when you take History next semester. Maybe you can use your knowledge of Christopher Columbus to ace your next test! Are you looking forward to it?
Incorrect Scoring	10%	Informative: The response provides a supportive and understanding message, which is informative to the user. 2/3	Response: I'm sorry to hear... when Christopher Columbus didn't even fight in the Civil War! I'm sure you'll do much better.... Are you looking forward to it?
Table 16: Error analysis for Dialogue Response Generation: When the feedback is not useful, a large majority is not specific or incorrect.
Error Type	Occurrence	Feedback	Refinement
Not-Robust	10%	Specific: The response is specific and provides spe-cific details or examples. 3/3	Response: That's pretty funny! It's understand-able that you didn't do well on the test. I'm sure you'll do much better when you take History next semester
Ignores feedback	25%	Informative: The re-sponse doesn't provide any new information. 1/3	Response: That's so kind and generous of you! I would love to hear more about the podcasts you enjoy. Do you have any favorites you can recommend?
Introduces new problem	20%	Informative: 3/3	Response: That would be a disaster! Luckily, my coworkers know bet-ter than to drink while they work. New Informa-tive score: 2/3
Robust to bad feedback	60%	Informative: The re-sponse does not provide any information. 1/3	Response: Wow, 25 peo-ple! That must have been an amazing experience. Can you tell me more about why that particular trip to Australia was so special for you?
Table 17: On the Dialogue Response Generation task, SELF-REFINE can ignore good feedback but in a majority of cases, it is robust to bad feedback and ignores bad feedback.
Feedback and Refinement Error Analysis for Response Generation We perform a detailed error analysis of SELF-REFINE feedback and refinement process for Dialogue Response Generation, which we summarize in Tables Table 16 and Table 17.
Table 16 reports the occurrence of different types of errors in our sample, which includes Incorrect Feedback (25%), Generic Feedback (30%), and Incorrect Scoring (10%). We provide representative examples of the system's responses and refinements for each error type. These errors highlight potential areas for improving our feedback handling mechanism, particularly in the interpretation and understanding of user inputs.
Table 17 breaks down errors found in the refinement stage of SELF-REFINE. Errors include: not being robust (10%), ignoring feedback (25%), and introducing a new problem (20%). We demonstrate how the model handles a variety of feedback types, how robust it is under different circumstances, and how often it inadvertently introduces new issues. 60% of the times, the model is robust to incorrect or generic feedback. These insights can guide us in enhancing the model's refinement capabilities, especially in providing accurate and specific responses.
L Beyond Benchmarks
SELF-REFINE demonstrates its iterative feedback and refinement capabilities in the context of website layout generation.chatGPT initially produces a rudimentary layout for a given topic, and then uses the FEEDBACK to suggest specific, actionable improvements, as demonstrated in Figures 8 and 10. These suggestions range from design changes such as color and font adjustments, to content enhancements and layout modifications. Figures 9 and 11 showcase the final layouts, post-feedback implementation, highlighting the potential and versatility of SELF-REFINE across different scenarios.
Delicious Ice Cream
Welcome to our ice cream parlor! We offer a wide range of flavors to suit all tastes, from classic vanilla to exotic mango. Our ice cream is made with the finest ingredients and is always fresh. Come and enjoy a scoop or two today!
Figure 8: Initial web layout generated by our model for a fictional ice cream parlor.
Ice Cream Generation The feedback generated by FEEDBACK for ice cream generation:
- Change the background color of the container to a light blue color (#6f2ff).
- Change the font size of the heading to 48px.
- Add a small icon before the "Welcome to our ice cream parlor!" text using the URL https://cdn Icons-png.flatican.com/512/3622/3622340.png.
- Add an additional paragraph after the existing text with the following text: "We also offer a variety of toppings and cones to complement your ice cream. Visit us today to try our latest flavors and indulge in a sweet treat!"
- Increase the font size of the button text to 24px.
- Update the button color to #9933.
Photosynthesis The feedback generated by FEEDBACK for photosynthesis:
- Increase the font size of the text to 18px for better readability.
- Add more information about the benefits of photosynthesis.
- Remove the unnecessary margin-top from the header.
- Add a ruler or divider below the header to separate it from the image.
Welcome to our ice cream parlor! We offer a wide range of flavors to suit all tastes, from classic vanilla to exotic mango. Our ice cream is made with the finest ingredients and is always fresh.
We also offer a variety of toppings and cones to complement your ice cream. Visit us today to try our latest flavors and indulge in a sweet treat!
Photosynthesis and Plants
Figure 9: Refined web layout after applying model feedback. The feedback included changing the background color to light blue (#6f2ff), increasing the heading font size to 48px, adding an icon before the welcome text, enhancing the content with an additional paragraph, increasing the button text size to 24px, and updating the button color to #9933.
Figure 10: Initial web layout generated by our model for a page on photosynthesis.
Photosynthesis is the process by which plants, algae, and some bacteria convert sunlight, carbon dioxide, and water into oxygen and energy-rich carbohydrates such as sugars and starches. This process is essential to the survival of life on Earth as it produces the oxygen that all animals need to breathe.
Figure 11: Refined web layout after applying model feedback. The feedback included increasing the text font size to 18px for better readability, adding more information about the benefits of photosynthesis, removing the unnecessary margin-top from the header, and adding a ruler or divider below the header to separate it from the image.
Photosynthesis is the process by which plants, algae, and some bacteria convert sunlight, carbon dioxide, and water into oxygen and energy-rich carbohydrates such as sugars and starches. This process is essential to the survival of life on Earth as it produces the oxygen that all animals need to breathe. Additionally, photosynthesis plays a major role in regulating the levels of carbon dioxide in the atmosphere, which helps to mitigate the effects of global warming and climate change.
M Statistical Confidence Intervals
Task	GPT-3.5	CHATGPT	GPT-4
Base	+SELF-REFINE	Base	+SELF-REFINE	Base	+SELF-REFINE
Sentiment Reversal	8.8 ± 2.05	30.4 ± 3.61*	11.4 ± 2.34	43.2 ± 3.98*	3.8 ± 1.28	36.2 ± 3.82*
Dialogue Response	36.4 ± 6.14	63.6 ± 6.62*	40.1 ± 6.33	59.9 ± 6.67*	25.4 ± 5.36	74.6 ± 6.22*
Code Optimization	14.8 ± 2.66	23.0 ± 3.25*	23.9 ± 3.30	27.5 ± 3.49	27.3 ± 3.48	36.0 ± 3.81*
Code Readability	37.4 ± 6.86	51.3 ± 7.39	27.7 ± 6.13	63.1 ± 7.40*	27.4 ± 6.10	56.2 ± 7.45*
Math Reasoning	64.1 ± 3.47	64.1 ± 3.47	74.8 ± 3.20	75.0 ± 3.20	92.9 ± 2.05	93.1 ± 2.03
Acronym Gen.	41.6 ± 7.72	56.4 ± 8.15	27.2 ± 6.60	37.2 ± 7.46	30.4 ± 6.92	56.0 ± 8.15*
Constrained Gen.	28.0 ± 7.38	37.0 ± 8.26	44.0 ± 8.72	67.0 ± 9.00*	15.0 ± 5.38	45.0 ± 8.77*
Table 18: SELF-REFINE results from table 1 with Wilson confidence interval (at  $95\%$  confidence interval) and statistical significance. On various tasks using GPT-3.5, CHATGPT, and GPT-4 as base LLM, SELF-REFINE consistently improves LLM. Metrics used for these tasks are defined in Section 3.2 as follows: Math Reasoning uses the solve rate; Code Optimization uses the percentage of programs optimized; and Sentiment Reversal, Dialogue Response and Acronym Gen use a GPT-4-based preference evaluation, which measures the percentage of times outputs from the base or enhanced models were selected, with the rest categorized as a tie. Constrained Gen uses the coverage percentage. Gains over Base, that are statistically significant based on these confidence intervals are marked *
Table 18 shows results from Table 1 with Wilson confidence interval (Brown et al., 2001) (at  $\alpha = 99\%$  confidence interval) and statistical significance. Gains that are statistical significance based on these confidence intervals are marked with an asterisk. We find that nearly all of GPT-4 gains are statistically significant, CHATGPT gains are significant for 4 out of 7 datasets, and GPT-3.5 gains are significant for 3 out of 7 datasets.
N New Tasks
Constrained Generation We introduce "CommonGen-Hard," a more challenging extension of the CommonGen dataset (Lin et al., 2020), designed to test state-of-the-art language models' advanced commonsense reasoning, contextual understanding, and creative problem-solving. CommonGen-Hard requires models to generate coherent sentences incorporating 20-30 concepts, rather than only the 3-5 related concepts given in CommonGen. SELF-REFINE focuses on iterative creation with introspective feedback, making it suitable for evaluating the effectiveness of language models on the CommonGen-Hard task.
Acronym Generation Acronym generation requires an iterative refinement process to create concise and memorable representations of complex terms or phrases, involving tradeoffs between length, ease of pronunciation, and relevance, and thus serves as a natural testbed for our approach. We source a dataset of 250 acronyms<sup>6</sup> and manually prune it to remove offensive or uninformative acronyms.
O Code Readability
Orthogonal to the correctness, readability is another important quality of a piece of code: though not related to the execution results of the code, code readability may significantly affect the usability, upgradability, and ease of maintenance of an entire codebase. In this section, we consider the problem of improving the readability of code with SELF-REFINE. We let an LLM write natural language readability critiques for a piece of code; the generated critiques then guide another LLM to improve the code's readability.
O.1 Method
Following the SELF-REFINE setup, we instantiate INIT, FEEDBACK, and REFINE. The INIT is a no-op — we directly start by critiquing the code with FEEDBACK and applying the changes with REFINE.
- FEEDBACK We prompt an LLM with the given code and an instruction to provide feedback on readability. We give the LLM the freedom to freely choose the type of enhancements and express them in the form of free text.
- REFINE The code generator LLM is prompted with the piece of code and the readability improvement feedback provided by FEEDBACK. In addition, we also supply an instruction to fix the code using the feedback. We take the generation from the code generator as the product of one iteration in the feedback loop.
Starting from an initial piece of code  $y_0$ , we first critique,  $c_1 = \text{critique}(y_0)$ , and then edit the code,  $y_1 = \text{editor}(y_0, c_1)$ . This is recursively performed  $N$  times, where  $c_{k+1} = \text{critique}(y_k)$  and  $y_{k+1} = \text{editor}(y_k, c_{k+1})$ .
O.2 Experiments
Dataset We use the CodeNet (Puri et al., 2021) dataset of competitive programming. For our purpose, these are hard-to-read multi-line code snippets. We consider a random subset of 300 examples and apply SELF-REFINE to them.
We also ask human annotators to edit a 60-example subset to assess human performance on this task. The human annotators are asked to read the code piece and improve its readability.
Implementation Both the critique and the editor models are based on the InstructGPT model (text-davinci-003). We consider the temperature of both  $T = 0.0$  (greedy) and  $T = 0.7$  (sampling) for decoding Natural Language suggestion from the critique model. We always use a temperature  $T = 0.0$  (greedy) when decoding Programming Language from the code editor. Due to budget constraints, we run SELF-REFINE for  $N = 5$  iterations. The exact prompts we use can be found in Figures 23-24.
https://github.com/krishnakt031990/Crawl-Wiki-For-Acronyms/blob/master/AcronymsFile.csv  
https://github.com/IBM/Project_CodeNet
Meaningful Variable Ratio	Comment Per Line	Function Units
Human Annotator Rewrites	0.653	0.24	0.70
SELF-REFINE (T = 0.0)	0.628	0.12	1.41
SELF-REFINE (T = 0.7)	0.700	0.25	1.33
Table 19: Human v.s. SELF-REFINE performance on 60-example subset. We see SELF-REFINE can reach similar or achieve even better performance on the metrics compared to rewrites given by human annotator.
Evaluation Methods We consider a few automatic heuristic-based evaluation metrics,
- Meaningful Variable Names: In order to understand the flow of a program, having semantically meaningful variable names can offer much useful information. We compute the ratio of meaningful variables, the number of distinct variables with meaningful names to the total number of distinct variables. We automate the process of extracting distinct variables and the meaningful subset of variables using a few-shot prompted language model.
- Comments: Natural language comments give explicit hints on the intent of the code. We compute the average number of comment pieces per code line.
- Function Units: Long functions are hard to parse. Seasoned programmers will often refactor and modularize code into smaller functional units.
Result For each automatic evaluation metric, the ratio of meaningful variable, of comment, and the number of function units, we compute for each iteration averaged across all test examples and plot for each SELF-REFINE iteration in Figure 12(a), Figure 12(b) and Figure 12(c) respectively. The two curves each correspond to critique with temperature  $T = 0.0$  and  $T = 0.7$ . The iteration 0 number is measured from the original input code piece from CodeNet. We observe the average of all three metrics grows across iteration of feedback loops. A diverse generation of a higher temperature in the critique leads to more edits to improve the meaningfulness of variable names and to add comments. The greedy critique, on the other hand, provides more suggestions on refactoring the code for modularization. Figure 13 provides an example of code-readability improving over iterations.
In Table 19, we measure human performance on all three metrics and compare with SELF-REFINE last iteration output. At  $T = 0.7$ , SELF-REFINE produces more meaning variables, more function units and slightly more comments compared to the human annotators on average. At  $T = 0.0$ , SELF-REFINE produces less meaningful variables, less comments per line but even more function units.
(a) Meaningful variable ratio across different SELF-REFINE iterations.
(b) Comment per line ratio across different SELF-REFINE iterations.
(c) Number of function units across different SELF-REFINE iterations.
Figure 12: Evaluation on code readability task with SELF-REFINE across multiple metrics
P Dialogue Response Generation
Open-domain dialogue response generation is a complex task that requires a system to generate human-like responses to a wide range of topics. Due to the open-ended nature of the task, it is
Starting Code:  $v_{0}$
print((int((int.eval(input()))+1)/2)))   
Code  $v_{1}$    
print( (int( (int.eval(input()）+1) /2 ）   
Code  $v_{2}$    
num_input  $\equiv$  eval(input()) num_input  $\equiv$  int(num_input) num_input  $+= 1$  num_result  $\equiv$  int(num_input/2) print(num_result)
Figure 13: SELF-REFINE iterations over a piece of densely composed code. Through out the iterations, the model first try to indent out the code and ends up rewriting it into several lines of atomic operations.
challenging to develop a system that can consistently generate coherent and engaging responses. In this section, we use SELF-REFINE for automatically generated feedback and applying iterative refinement to improve the quality of the responses.
P.1 Modules
We follow the high-level description of the framework from Section 2, and instantiate our framework as follows.
Init This is the first step in performing the task. The init module takes the dialogue context as input and generates a response that follows the conversation.
Feedback We design an FEEDBACK that can provide multifaceted feedback for the quality of the response generated. Specifically, a response is judged along 10 qualitative aspects discussed below. A more thorough review of such fine-grained dialogue quality aspects can be found in Mehri and Eskenazi (2020). We use 6 in-context examples for feedback generation. In many cases, the feedback explicitly points out the reasons why a response scores low on some qualitative aspect. We show an example in Figure 14.
- Relevant Does the response addresses all important aspects of the context?
- Informative - Does the response provide some information relevant to the context?
- Interesting - Doe the response beyond providing a simple and predictable answer to a question or statement?
- Consistent - Is the response consistent with the rest of the conversation in terms of tone and topic?
- Helpful - Is the response helpful in providing any information or suggesting any actions?
- Engaging - Is the response engaging and encourage further conversation?
- Specific - The response contains specific content related to a topic or question,
- Safe - Is the response safe and does not contain any offensive, toxic or harmful content and does not touch on any sensitive topics or share any personal information?
- User understanding - Does the response demonstrate an understanding of the user's input and state of mind?
- Fluent Is the response fluent and easy to understand?
Context: Speaker A - What are your hobbies?
Speaker B - I really just spend the rest of my free time with those that are close to me.
Speaker A - I'm very much interested in playing table tennis.
Response: That sounds like a lot of fun! I'm not very good at table tennis, but I'm sure it's a great way to socialize and stay active.
* Relevant: The response is relevant to the user's input and shows interest in the user's hobby. 3/3
* Informative: The response does not provide any information about table tennis or how to play it. 1/3
* Interesting: The response is not very interesting or intriguing. 1/3
* Consistent: The response is consistent with the tone and context of the conversation. 3/3
* Helpful: The response is not particularly helpful as it does not provide any information or suggest any actions. 1/3
* Engaging: The response is somewhat engaging. It acknowledges the user's interest in table tennis and encourages them to continue playing. 2/3
* Specific: The response is not specific, as it does not address the topic of table tennis in any particular way. 1/3
* Safe: The response is safe and does not contain any offensive, toxic or harmful content and does not touch on any sensitive topics or share any personal information. 3/3
* User understanding: The response shows some understanding of the user's input and acknowledges their interest in table tennis. 2/3
* Fluent: The response is fluent in terms of grammar and flow of words. 3/3
* Total score: 20/30
New Response: Table tennis is a great hobby! It's a great way to stay active and socialize with others. Have you ever tried it before? If not, I'd suggest starting with some beginner tutorials on Youtube. They are really helpful in getting a good grasp of the basics.
Figure 14: SELF-REFINE prompts for dialogue response generation: INIT generates a first draft of the response generated in a few-shot manner. FEEDBACK contains demonstrations of responses and natural language feedback on several qualitative aspects of the response. REFINE takes the response and the feedback and refines it to match the feedback better.
Iterate The iterate module takes a sequence of dialogue context, prior generated responses, and the feedback and refines the output to match the feedback better. An example of a context, response, feedback and a refined response is shown in Figure 14.
P.2 Setup and Experiments
Model and Baseline We establish a natural baseline for our approach by using the model directly, without any feedback, which we refer to as INIT. Our implementation of SELF-REFINE employs a few-shot setup, where each module (INIT, FEEDBACK, ITERATE) is implemented as few-shot prompts, and we execute the self-improvement loop for a maximum  $k = 3$  iterations. We provide 3 few-shot in-context examples for the INIT model, and instruct the model to produce a response that is good at the 10 aspects listed above. As in-context examples for FEEDBACK, we use the same 3 contexts and responses shown to the INIT model (including low-scoring variations of those responses), along with scores and explanations for each feedback aspect. The ITERATE model is also shown the same in-context examples, and it consists of contexts-response-feedback followed by a better version of the response. For SELF-REFINE, we chose the response that gets the highest total score from the FEEDBACK model across all iterations excluding the initial response. We use text-davinci-003 for all the experiments.
GPT-3.5	ChatGPT	GPT4
SELF-REFINE wins	36.0	48.0	54.0
INIT wins	23.0	18.0	16.0
Both are equal	41.0	50.0	30.0
Table 20: Human evaluation results for dialogue response generation
Evaluation We perform experiments on the FED dataset (Mehri and Eskenazi, 2020). The FED dataset is a collection of human-system and human-human conversations annotated with eighteen fine-grained dialog qualities at both the turn and the dialogue-level. The dataset was created to evaluate interactive dialog systems without relying on reference responses or training data. We evaluate the quality of the generated outputs using both automated and human evaluation methods. For automatic evaluation in Table1, we used zero-shot prompting with text-davinci-003 and evaluate on a test set of 342 instances. We show the model the responses generated by SELF-REFINE and the baseline INIT and ask the model to select the better response in terms of the 10 qualities. We report the win rate. However, we acknowledge that automated metrics may not provide an accurate assessment of text generation tasks and rely on human evaluation instead.
Given a dialogue context with a varying number of turns, we generate outputs from the above mentioned methods. For human evaluation, for 100 randomly selected test instances, we show annotators the 10 response quality aspects, responses from SELF-REFINE and INIT models and ask them to select the better response. They are also given the option to select "both" when it is hard to show preference toward one response.