SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For each task, we measured inter-labeler agreement using Cohen's Kappa score. Code Readability Improvement and Acronym Generation both scored a substantial 0.75, Sentiment Reversal was also substantial at 0.61, while Dialogue Response Generation was moderate with a score of 0.53.
Task	SELF-REFINE (%)	Direct (%)	Either (%)
Sentiment Reversal	75.00	21.43	3.57
Acronym Generation	44.59	12.16	43.24
Dialogue Response Generation	47.58	19.66	32.76
Code Readability Improvement	50.0	3.0	47.0
Table 6: Relative improvement of SELF-REFINE in A/B evaluations across different tasks. The values represent normalized preferences, which correspond to the proportion of times the output generated by SELF-REFINE was selected as better aligned with the task instruction over the baseline method. The evaluation was conducted for 150 examples for each dataset. The judges were not aware of the method that generated each sample.
D SELF-REFINE on Reasoning Tasks
SELF-REFINE shows limited success in Math Reasoning tasks, mainly due to its challenges in generating meaningful feedback. Nonetheless, we observe performance gains with SELF-REFINE in certain tasks from the Big-Bench Hard suite (Suzgun et al., 2022), particularly those requiring commonsense reasoning and logical thinking. These results in Table 7 suggest that SELF-REFINE is more effective in scenarios where the combination of logical reasoning and pre-trained knowledge contribute to self-verification.
Task	Base model	+Self-Refine	Gain
Date Understanding	62.0	66.8	4.8
Geometric Shapes	17.6	20.0	2.4
Logical Deduction (seven objects)	43.2	45.2	2.0
Multi-Step Arithmetic	61.6	64.0	2.4
Tracking Shuffled Objects (seven objects)	31.6	36.0	4.4
Table 7: Performance comparison between the base outputs and the SELF-REFINE enhanced model across various Big Bench Hard tasks. All experiments were conducted using the GPT-3.5-TURBO-0613 model with a temperature setting of 0.0. No task-specific prompts were utilized; all tasks employed the same instructional prompts.
E Instruction-Only Prompting
In our main experiments, we use few-shot prompting to guide model output into a more readily parseable format. Next, we experiment with SELF-REFINE under a zero-shot prompting scenario, where traditional few-shot examples are supplanted by explicit instructions at each stage of the SELF-REFINE process. For these experiments, we use CHATGPT. The results in Table 8) show that SELF-REFINE remains effective across diverse tasks. However, achieving optimal performance in this setting requires extensive prompt engineering for instructions. The instructions are present in Listing 1 (Acronym Generation), Listing 3 (Math Reasoning), Listing 2 (Sentiment Reversal), Listing 4 (Constrained Generation), and Listing 5 (Dialogue Response Generation).
Task	Base	SELF-REFINE (zero-shot)
Acronym Generation	16.6%	44.8% (↑ 28.2%)
Constrained Generation	4.0%	47.0% (↑ 43.0%)
Sentiment Reversal	4.4%	71.4% (↑ 67.0%)
Math Reasoning	22.1%	59.0% (↑ 36.9%)
Dialogue Response Generation	23.0%	48.8% (↑ 25.8%)
Table 8: Performance of SELF-REFINE with Zero-Shot Prompting
F GPT-4 Evaluation
In light of the impressive achievements of GPT-4 in assessing and providing reasoning for complex tasks, we leverage its abilities for evaluation in SELF-REFINE. The approach involves presenting tasks to GPT-4 in a structured manner, promoting the model's deliberation on the task and generating a rationale for its decision. To mitigate order bias in our tasks, we randomly flip the order of the outputs generated after the first iteration and by SELF-REFINE before evaluation. Further, to ensure
Listing 1 Instruction-only prompts used at various stages of the SELF-REFINE process for Acronym Generation.
# Init: Generate an acronym for a given title
start chatting_log = [
    {"role": "system", "content": 'You are a helpful assistant that generates  $\leftrightarrow$  acronyms.'],
    {"role": "user", "content": f'Generate an acronym for the title '{title}'}.
     $\leftrightarrow$  Please respond in the format: "The generated acronym is: {acronym}"]
]
# Generate Feedback: Evaluate the quality of the generated acronym
feedback_str = f''Evaluate the acronym {'acronym" for the title {'title" on its  $\leftrightarrow$  ease of pronunciation, ease of spelling, relation to the title, positive  $\leftrightarrow$  connotation, and being well-known.">
start chatting_log = [
    {"role": "system", "content": 'You are an AI model that provides feedback on  $\leftrightarrow$  the viability and quality of acronyms."},
    {"role": "user", "content": feedback_str}
]
# Refine: Improve the acronym based on provided feedback
start chatting_log = [
    {"role": "system", "content": 'You are a helpful assistant that can  $\leftrightarrow$  iteratively improve acronyms based on feedback.'},
    {"role": "user", "content": f'Improve the acronym {'acronym}" for the title  $\leftrightarrow$  {"title}" based on the following feedback: {feedback}. Please *always*  $\leftrightarrow$  respond in the format: "The improved acronym is: {acronym}"}
]
Listing 2 Instruction-only prompts used at various stages of the Self-Refine process for Sentiment Reversal.
# Init: Transform a negative sentiment review into a positive one   
start chatting_log  $=$  { {"role": "system", "content": 'You are an AI language model that transforms a  $\nrightarrow$  negative sentiment review into a positive one.}}, {"role": "user", "content": f'Please transform the following negative review  $\nrightarrow$  into a positive one: {"review)}. Respond in the format of: "The positive  $\nrightarrow$  review is: [Your Response Here]".}},   
1   
# Generate Feedback: Provide feedback on the transformed review   
start chatting_log  $=$  { {"role": "system", "content": 'You are an AI model providing feedback on a  $\nrightarrow$  sentiment transformed review.}}, {"role": "user", "content": f"Why is this review not Very positive? Point out  $\nrightarrow$  specific issues and give concrete suggestions. Respond in the format of:  $\nrightarrow$  'Feedback: [Specific issues and suggestions]}},   
1   
# Refine: Improve the review based on provided feedback   
start chatting_log  $=$  { {"role": "system", "content": 'You are an AI model that improves upon an  $\nrightarrow$  existing review based on provided feedback.}}, {"role": "user", "content": f'Please improve the sentiment of the following  $\nrightarrow$  review {"sentence"} based on this feedback: {"feedback]}, and make it more  $\nrightarrow$  positive. Always respond in the format of: "The more positive review is:  $\nrightarrow$  [Your Response Here]".}},   
1
Listing 3 Instruction-only prompts used at various stages of the Self-Refine process for Math Reasoning.
# Function to Generate an Answer
gen_sys_template = "You are a helpful assistant that responds with only a python  $\rightarrow$  program."
gen_user_template = ""Write a python function that solves the given question and  $\rightarrow$  returns the result.
Always store the final result in a variable called `result`, and always include  $\rightarrow$  'return result' as your last statement.
# Question: {question}
# solution using Python:""
# Function to Get Feedback on the Answer
fb_sys_template = "You are a helpful assistant that provides feedback on the  $\rightarrow$  correctness of python programs."
fb_user_template = ""Question: {question}
{prediction}
# There may be an error in the code above because of lack of understanding of the  $\rightarrow$  question.
To find the error, go through semantically complete blocks of the code, and check  $\rightarrow$  if everything looks good.
Errors could include:
- Logical issues
- Lack of understanding of the question
- Missing `return result`
Share your evaluation in the format:
Evaluation: <your evaluation here>. If everything looks good, return 'Evaluation: correct'**
# Function to Fix the Error
refine_sys_template = "You are a helpful assistant that responds with only a  $\rightarrow$  python program."
refine_user_template = ""Question: {question}
{prediction}
# There is an error in the code above. The following is the feedback on the code:
{feedback}
Fix the error in the python function given the feedback above. The python function  $\rightarrow$  should return the final answer."
that there is no inherent bias in GPT-4 picking its own predictions, we conduct an additional study with CLAUDE-V2.
Claude-2 as the Evaluator Despite the measures we took to prevent any inherent biases, GPT-4 might inherently favor self-refined outputs. To provide a comprehensive evaluation of GPT-4, we conduct an additional analysis using CLAUDE-V2, serving as an independent evaluator for GPT-4 outputs. The results in Table 9 from GPT-4 as the base LLM with CLAUDE-V2 as the evaluator show the same strong preferences for SELF-REFINE over the Base outputs.
Task	% Base	% SELF-REFINE
Dialogue Response Generation	30.6	64.7 (↑34.1)
Sentiment Reversal	10.6	69.2 (↑58.6)
Acronym Generation	32.0	49.2 (↑17.2)
Code readability	37.0	60.0 (↑23.0)
Table 9: Evaluation Results of GPT-4 with CLAUDE-V2 as Evaluator
This prompts used for evaluation are listed in Listings 6 to 8.
<sup>5</sup>https://www.anthropic.com/index/claude-2
Listing 4 Instruction-only prompts used at various stages of the SELF-REFINE process for Constrained Generation.
# Constrained Generation Task
# 1. Init
start chatting_log = [
    {"role": "system", "content": 'You are an AI model that generates sentences  $\leftrightarrow$  with commonsense, using a given set of concepts.'] ,
    {"role": "user", "content": f'Generate a commonsense sentence using the  $\leftrightarrow$  following concepts: {{}, ".join(concepts)}. Please respond in the format:
    {"The generated sentence is: {sentence}''"},
]
# 2. Get Feedback
start chatting_log = [
    {"role": "system", "content": 'You are an AI model that provides feedback on a  $\leftrightarrow$  sentence generated with specific concepts.'] ,
    {"role": "user", "content": f''Evaluate the following sentence '{sentence}',  $\leftrightarrow$  which was meant to use the following concepts: {{}, ".join(concepts)}} .
    Please provide two pieces of feedback. Start your feedback about concept usage  $\leftrightarrow$  with the phrase "Concept feedback is:", and start your feedback about  $\leftrightarrow$  commonsense facts with the phrase "Commonsense feedback is:" . The format should be:
    Concept feedback is: <list of missing concepts>
    Commonsense feedback is: <commonsense feedback here>''],
]
# 3. Iterate Fix
start chatting_log = [
    {"role": "system", "content": 'You are an AI model that improves upon an  $\leftrightarrow$  existing sentence based on provided feedback.'] ,
    {"role": "user", "content": f'Improve upon the following sentence  $\leftrightarrow$  "[improvedsentence]" based on the following feedback: {feedback_string}] .
    Please respond in the format: "The improved sentence is: [Your improved  $\leftrightarrow$  sentence here]'},
]
G Model Key
We use terminology here: https://platform.openai.com/docs/models/gpt-3-5. The experiments were done with the 0313 versions of GPT-4 and CHATGPT unless otherwise mentioned. We refer to text-davinci-003 as GPT-3.5. We use GPT-3.5-TURBO-0613 for all instruction-only experiments and Constrained Generation experiments.
Listing 5 Instruction-only prompts used at various stages of the Self-Refine process for Dialogue Response Generation.
# Dialogue Response Generation Task
# 1. Generate Response
start chatting_log = [
    {"role": "system", "content": 'You are a helpful assistant that generates responses."},
    {"role": "user", "content": f'Provided a dialogue between two speakers, generate a response that is coherent with the dialogue history. Desired traits for responses are: 1) Relevant - The response addresses the context, 2) Informative - The response provides some information, 3) Interesting - The response is interesting, 4) Consistent - The response is consistent with the rest of the conversation in terms of tone and topic, 5) Helpful - The response is helpful in providing any information or suggesting any actions, 6) Engaging - The response is engaging and encourages further conversation, 7) Specific - The response contains specific content, 8) Safe - The response should not cause danger, risk, or injury 9) User understanding - The response demonstrates an understanding of the user's input and state of mind, and 10) Fluent. Response should begin with - Response:\n\n
] # 2. Get Feedback
start chatting_log = [
    {"role": "system", "content": 'You are an AI model that provides feedback on the viability and quality of responses."},
    {"role": "user", "content": feedback_str}
]
# 3. Iterate Response
start chatting_log = [
    {"role": "system", "content": 'You are a helpful assistant that can iteratively improve responses based on feedback.'},
    {"role": "user", "content": iterate_str}
]
Listing 6 Prompt for GPT-4 evaluation of Sentiment Reversal.
f""Which review is aligned with the sentiment {target_sentiment}? Review A: {review_a} Review B: {review_b}. Pick your answer from ['Review A', 'Review B', 'both', 'neither']. Generate a  $\leftrightarrow$  short explanation for your choice first. Then, generate 'The more aligned  $\leftrightarrow$  review is A' or 'The more aligned review is B' or 'The more aligned review is  $\leftrightarrow$  both' or 'The more aligned review is neither'. Format: <explanation> <answer> STOP
Listing 7 Prompt for GPT-4 evaluation of Acronym Generation.
f""Title: {title}  
Acronym A: {acronym_a}  
Acronym B: {acronym_b}  
Pick the better acronym for the given title. The acronyms should be compared based  $\leftrightarrow$  on the following criteria:  
* Ease of pronunciation.  
* Ease of spelling.  
* Relation to title.  
* Positive connotation.  
Generate your answer in the following format:  
<Short explanation>. The better acronym is A OR The better acronym is B OR The  $\leftrightarrow$  acronyms are equally good OR Neither acronym is good. STOP.
Listing 8 Prompt for GPT-4 evaluation of Dialogue Response Generation.
f""Which response is better given this context: {context}? Response A: {response_a} Response B: {response_b}. Pick your answer from ['Response A', 'Response B', 'both', 'neither']. Generate a  $\leftrightarrow$  short explanation for your choice first. Then, generate 'The better response  $\leftrightarrow$  is A' or 'The better response is B' or 'The better response is both' or 'The  $\leftrightarrow$  better response is neither'. Format: <explanation> <answer> STOP
Listing 9 Prompt for GPT-4 evaluation of Constrained Generation
f""Which story is better?   
Story A: {story_a}   
Story B: {story_b}.   
Judge the story based on the flow, the grammar, and the overall quality of the  $\leftrightarrow$  story. Rate more realistic story higher. Pick your answer from ['Story A',  $\leftrightarrow$  'Story B', 'either']. First, reason about your choice. Then, generate The better story is Story A' or 'The better story is Story B' or 'The better story  $\leftrightarrow$  is either'.   
Format:   
Reasoning: <your reasoning>. The better story is <your choice>.   
Reasoning:"''"
H Comparison of SELF-REFINE with State-of-the-art of Few-Shot Learning Models and Fine-Tuned Baselines
In this section, we present a comprehensive comparison of the performance of SELF-REFINE with other few-shot models and fine-tuned baselines across a range of tasks, including mathematical reasoning and programming tasks. Tables 10 and 11 display the performance of these models on the GSM tasks and PIE dataset, respectively. Table 12 shows the performance of the CODEX model with and without SELF-REFINE on the Code Readability task, further described in Appendix O. Our analysis demonstrates the effectiveness of different model architectures and training techniques in tackling complex problems.
Method	Solve Rate
Chen et al. (2021)	CODEX	71.3
Cobbe et al. (2021)	OpenAI 6B	20.0
Wei et al. (2022)	CoT w/ CODEX	65.6
Gao et al. (2022)	PaL w/ CODEX	72.0
PaL w/ GPT-3	52.0
PaL w/ GPT-3.5	56.8
PaL w/ CHATGPT	74.2
PaL w/ GPT-4	93.3
Welleck et al. (2022)	Self-Correct w/ GPT-3	45.9
Self-Correct (fine-tuned)	24.3
This work	SELF-REFINE w/ GPT-3	55.7
SELF-REFINE w/ GPT-3.5	62.4
SELF-REFINE w/ CHATGPT	75.1
SELF-REFINE w/ GPT-4	94.5
SELF-REFINE w/ CODEX (Oracle Feedback)	76.2
Table 10: Performance comparison of models on math reasoning (Math Reasoning).
Method	%OPT
Puri et al. (2021)	Human References	38.2
OpenAI Models: OpenAI (2022, 2023)	CODEX	9.7
GPT-3.5	14.8
CHATGPT	22.2
GPT-4	27.3
Nijkamp et al. (2022)	CODEGEN-16B	1.1
Berger et al. (2022)	SCALENE	1.4
SCALENE (BEST@16)	12.6
SCALENE (BEST@32)	19.6
Madaan et al. (2023)	PIE-2B	4.4
PIE-2B (BEST@16)	21.1
PIE-2B (BEST@32)	26.3
PIE-16B	4.4
PIE-16B (BEST@16)	22.4
PIE-16B (BEST@32)	26.6
PIE-Few-shot (BEST@16)	35.2
PIE-Few-shot (BEST@32)	38.3
This work	SELF-REFINE w/ CODEX	15.6
SELF-REFINE w/ GPT-3.5	23.0
SELF-REFINE w/chatGPT	26.7
SELF-REFINE w/GPT-4	36.0
Table 11: Performance comparison of various models on the PIE dataset in terms of the percentage of programs optimized (%OPT). The table includes human references, baseline models, fine-tuned PIE-2B and PIE-16B models, and our proposed model (SELF-REFINE) using different LLMs. Notably, SELF-REFINE achieves superior performance while using only 4 samples at most, significantly fewer than the 16 and 32 samples employed by other models. Scalene, an off-the-shelf optimizer, uses instruction tuning with CODEX and serves as a comparison point.
Method	% Readable Variables
Chen et al. (2021)	CODEX	37.4
This work	SELF-REFINE w/ CODEX	51.3
Table 12: Performance of SELF-REFINE on CODEX on the Code Readability task. Further details on the task are described in Appendix O
I Evaluation of Vicuna-13b
We also experiment with Vicuna-13b (Chiang et al., 2023), a version of LLaMA-13b (Touvron et al., 2023) fine-tuned on conversations sourced from the web. Vicuna-13b was able to consistently follow the task initialization prompt. However, it struggled to follow the prompts intended for feedback and refinement. This often led to outputs that resembled assistant-like responses, a representative example of which can be found in Figure 6.
It's important to note that we used the same prompts for Vicuna-13b as those used with other models in our study. However, the limited performance of Vicuna-13b suggests that this model may require more extensive prompt-engineering for optimal performance.
Mixed-refine: Improving Vicuna-13b with CHATGPT While the focus of SELF-REFINE is improvement of the model without any external help, it may be possible to use a smaller model for the initialization, and then involving a bigger model for refinement. To test this, we experiment with a setup where we use Vicuna-13b as the initialization model, and use CHATGPT as the FEEDBACK and REFINE. The results on Math Reasoning show the promise of this approach: while Vicuna-13b was able to get only  $24.18\%$  on Math Reasoning, it was able to improve to  $40.5\%$  in this mixed-refinement setting.
Step 0 INIT> The food was amazing, I loved it!!.
Step 0 TRANSFER> The food was terrible, I hated it!!.
An error occurred: list index out of range. Traceback (most recent call last):
IndexError: list index out of range. Left retries: 2.
Step 1 TRANSFER> The food at the Trop was awful. I wish I could've gotten my money back for the terrible meals.
Step 1 FB> The rewritten review does not become more "Very negative", because it
$\leftrightarrow$  only contains a single instance of a positive sentiment about the food. Try
$\leftrightarrow$  again to make it Very negative!
Step 2 TRANSFER> ____ Assistant: The Trop is a great choice for anyone looking for
$\hookrightarrow$  a budget-friendly option in Vegas. The rooms and hallways are clean and
$\hookrightarrow$  comfortable, and the food options are varied and delicious.
Step 2 FB> The rewritten review does not change from "Positive", because it only
$\hookrightarrow$  has positive language. The output is already Very positive. No further changes
are needed.
Step 0 INIT> The food was amazing, I loved it!!. [Positive] -> [Very negative]
Step 0 TRANSFER> The food was atrocious, I absolutely hated it! [('The review  $\leftrightarrow$  sounds very negative. The sentiment is Very negative', 1.0)]
Step 0 FB> The original review is already "Very negative" as it uses strongly
$\leftrightarrow$  negative words like "atrocious" and "absolutely hated". These words express
$\hookrightarrow$  extreme dissatisfaction and clearly convey a very negative sentiment about the
$\leftrightarrow$  food. But try again to make it even more negative!
Step 1 TRANSFER> The food was so abysmal, it was a nightmare to even taste it! I
$\hookrightarrow$  couldn't stand it and regretted every single bite. [('The review sounds highly
negative. The sentiment is Very negative', 1.0)]
Step 1 FB> The review provided is indeed very negative. It uses extremely negative
$\hookrightarrow$  words and phrases like "abysmal", "nightmare", and "regretted every single
$\leftrightarrow$  bite". These strongly negative expressions make the review very negative, but
$\hookrightarrow$  try again to make it even more negative!
Step 2 TRANSFER> The food was absolutely revolting, I couldn't even stomach a
$\hookrightarrow$  single bite!
Step 2 FB> This review is not "Very negative" because it contains positive words
$\hookrightarrow$  and phrases like "amazing" and "loved it." To make it "Very negative", we need
to use extremely negative words, like "revolting", "couldn't stomach", or
$\hookrightarrow$  "horrible". The provided "Very negative" sentiment example is a good
$\hookrightarrow$  illustration of a very negative review, as it uses strong negative language to
$\hookrightarrow$  describe the food experience.
Figure 6: Vicuna-13b outputs (top) for sentiment reversal, where the goal is to make 'The food was amazing, I loved it!!' negative. Vicuna-13b generates a good initial output, but generates an empty feedback in the first try, leading to an error. In the next try, it generates an unhelpful feedback. The transferred review copies from the prompt. Compare with the outputs from GPT-4 (bottom).
J Llama-2
We further benchmark SELF-REFINE, this time on the open-access model Lllama-2 (Touvron et al., 2023) in an instruction-only setting (no few-shot prompts, only instructions). The results in Table 13 show that SELF-REFINE continues to be effective on an open-access model and without few-shot examples at all. Given these performance metrics, alongside anticipated advancements in hardware, we anticipate the broad and cost-effective applicability of SELF-REFINE.
Task	Base	Self-Refine	Equally Good
Acronym Generation	22.30	53.08	22.30
Sentiment Reversal	13.2	60.8	26
Dialogue Response Generation	11.2	20.4	54.6
Math Reasoning	37.6	37.8 (41 with Oracle)	N/A
Table 13: Instruction-only (zero-shot) results using SELF-REFINE on the open-access model Llama-2. SELF-REFINE provides promising gains even with open-access models on various tasks.
Figure 7: Preference for the outputs generated by our method (SELF-REFINE), the multiple-sample baseline (MULTI), and ties (ties).
Task	GPT-3.5	CHATGPT	GPT-4
Base	+SELF-REFINE	Base	+SELF-REFINE	Base	+SELF-REFINE
Math Reasoning	64.1	64.1 (0)	74.8	75.0 (↑0.2)	92.9	93.1 (↑0.2)
Math Reasoning (Oracle)	64.06	68.9 (↑4.8)	74.8	76.2 (↑1.4)	92.9	93.8 (↑0.7)
K Additional Analysis
K.1 Using Oracle Feedback
We experimented with Oracle Feedback following Welleck et al. (2022). This method uses correctness information to guide model refinement, only progressing to REFINE stage if the current answer is incorrect. This adjustment notably enhanced performance in the Math Reasoning task, with GPT-3 improving by  $4.8\%$  and GPT-4 by  $0.7\%$  Table 14. This indicates the potential of external signals to optimize model performance in particular tasks.
Table 14: SELF-REFINE results on Math Reasoning using GPT-3.5, CHATGPT, and GPT-4 as base LLM with Oracle feedback.
Iteration	Acronym	Pronunciation	Pron. (5)	Spell. (5)	Rel. (5)	Pos. Con. (5)	Total (25)
1	USTACCSF	us-tacks-eff	1	1	5	3	11
2	TACC-SIM	tacks-sim	4	4	5	3	17
3	TACCSF	tacks-eff	1	2	5	3	12
4	TACC-SIMF	tack-simf	4	4	5	3	17