SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Results Automatic evaluation results are shown in Table1 and human evaluation results are shown in Table 20. We experiment on 3 latest versions of GPT models. text-davinci-003 is capable of generating human-like responses of great quality for a wide range of dialogue contexts and hence GPT-3.5 is a strong baseline. Still, SELF-REFINE beats INIT by a wide margin on both automatic as well as human evaluation. Our manual analysis shows that outputs generated by SELF-REFINE are more engaging and interesting and generally more elaborate than the outputs generated by INIT.
Q Code Optimization
Performance-Improving Code Edits or PIE (Madaan et al., 2023) focuses on enhancing the efficiency of functionally correct programs. The primary objective of PIE is to optimize a given program by implementing algorithmic modifications that lead to improved runtime performance.
Given an optimization generated by PIE, SELF-REFINE first generates a natural language feedback on possible improvements Figure 21. Then, the feedback is fed to REFINE Figure 22 for refinement.
Table 21: Main Results and Ablation Analysis
Setup	Iteration	% Optimized	Relative Speedup	Speedup
Direct	-	9.7	62.29	3.09
SELF-REFINE – feedback	1	10.1	62.15	3.03
SELF-REFINE – feedback	2	10.4	61.79	3.01
SELF-REFINE	1	15.3	59.64	2.90
SELF-REFINE	2	15.6	65.60	3.74
Table 22: Performance comparison of SELF-REFINE and ablated variants for code optimization. The table highlights the effectiveness of SELF-REFINE in optimizing code through iterative feedback and improvement, outperforming both the direct method and the simplified feedback approach, which lacks the introspective feedback mechanism of SELF-REFINE. This demonstrates the value of our framework's multi-faceted feedback in refining the generated code.
R Math Reasoning
We use the Grade School Math 8k (GSM-8k) dataset (Cobbe et al., 2021) for evaluating SELF-REFINE on math reasoning. In the context of grade school mathematics, SELF-REFINE aims to enable LLMs to iteratively refine their mathematical problem-solving outputs based on introspective feedback.
Following Gao et al. (2022), we write solutions to the reasoning problems in Python. Consider the following example from the paper, where an error in the code demonstrates a lack of understanding of the problem:
def solution():
    '''Twenty dozen cups cost $1200 less than the total cost of half a dozen plates sold at $6000 each. Calculate the total cost of buying each cup.''' plates = 6
plate_cost = 6000
cups = 12 * 20
cup_cost = plate_cost
result = cup_cost
return result
By using SELF-REFINE, we can identify the error in the code and refine the solution through an iterative process of introspection and feedback:
# Let's go through the error and check step-by-step
plates = 6
plate_cost = 6000
# Let's check the other parts
cups = 12 * 20
cup_cost = plate_cost # wrong! The cost of a cup is not the same as the
 $\leftrightarrow$  cost of a plate.
# The cost of a cup is $1200 less than the total cost of half a dozen
 $\leftrightarrow$  plates sold at $6000 each.
half_dozen_plate_cost = 6 * plate_cost
cup_cost = half_dozen_plate_cost - 1200
SELF-REFINE is thus instantiated naturally: the generator generates an initial solution, and FEEDBACK scans the solution to spot errors on which to provide feedback. The feedback is supplied to REFINE to create a new solution. Following Welleck et al. (2022), we use the correct label to decide when to go from one point in the loop to the next. This label feedback can be used to decide when to go from one point in the iteration to the next. We show results using SELF-REFINE in Figure 15.
Solve rate of SELF-REFINE Over Iterations for GSM-8k
Figure 15: Improvements in accuracy on the GSM-8k math reasoning benchmark as a function of the # of iterations of SELF-REFINE.
S Sentiment Reversal
We consider the task of long-form text style transfer, where given a passage (a few sentences) and an associated sentiment (positive or negative), the task is to re-write the passage to flip its sentiment (positive to negative or vice-versa). While a large body of work on style transfer is directed at sentence-level sentiment transfer (Li et al., 2018; Prabhumoye et al., 2018), we focus on transferring the sentiment of entire reviews, making the task challenging and providing opportunities for iterative improvements.
Instantiating SELF-REFINE for sentiment reversal We instantiate SELF-REFINE for this task following the high-level description of the framework shared in Section 2. Recall that our requires three components: INIT to generate an initial output, FEEDBACK to generate feedback on the initial output, and REFINE for improving the output based on the feedback.
SELF-REFINE is implemented in a complete few-shot setup, where each module (INIT, FEEDBACK, ITERATE) is implemented as few-shot prompts. We execute the self-improvement loop for a maximum of  $k = 4$  iterations. The iterations continue until the target sentiment is reached.
S.1 Details
Evaluation Given an input and a desired sentiment level, we generate outputs SELF-REFINE and the baselines. Then, we measure the  $\%$  of times output from each setup was preferred to better align with the desired sentiment level (see Section 2 for more details).
We also experiment with standard text-classification metric. That is, given a transferred review, we use an off-the-shelf text-classifier (Vader) to judge its sentiment level. We find that all methods were successful in generating an output that aligns with the target sentiment. For instance, when the target sentiment was positive, both GPT-3.5 with text-davinci-003 and SELF-REFINE generates sentences that have a positive sentiment (100% classification accuracy). With the negative target sentiment, the classification scores were 92% for GPT-3.5 and 93.6% for SELF-REFINE.
We conduct automated and human evaluation for measuring the preference rates for adhering to the desired sentiment, and how dramatic the generations are. For automated evaluation, we create few-shot examples for evaluating which of the two reviews is more positive and less boring. We use a separate prompt for each task. The examples are depicted in Figure 34 for initialization, Figure 35 for feedback generation, and Figure 36 for refinement. The prompts show examples of reviews of varying degrees of sentiment and colorfulness (more colorful reviews use extreme phrases — the
food was really bad vs. I wouldn't eat it if they pay me.). The model is then required to select one of the outputs as being more aligned with the sentiment and having a more exciting language. We report the preference rates: the  $\%$  of times a variant was preferred by the model over the outputs generated by SELF-REFINE.
Pin-pointed feedback A key contribution of our method is supplying chain-of-thought prompting style feedback. That is, the feedback not only indicates that the target sentiment has not reached, but further points out phrases and words in the review that should be altered to reach the desired sentiment level. We experiment with an ablation of our setup where the feedback module simply says "something is wrong." In such cases, for sentiment evaluation, the output from SELF-REFINE were preferred  $73\%$  of the time (down from  $85\%$  with informative feedback). For dramatic response evaluation, we found that the preference rate went down drastically to  $58.92\%$ , from  $80.09\%$ . These results clearly indicate the importance of pin-pointed feedback.
Evaluation We evaluate the task using GPT-4. Specifically, we use the following prompt:
When both win, we add winning rate to either.
T Acronym Generation
Good acronyms provide a concise and memorable way to communicate complex ideas, making them easier to understand and remember, ultimately leading to more efficient and effective communication. Like in email writing, acronym generation also requires an iterative refinement process to achieve a concise and memorable representation of a complex term or phrase. Acronyms often involve tradeoffs between length, ease of pronunciation, and relevance to the original term or phrase. Thus, acronym generation is a natural method testbed for our approach.
We source the dataset for this task from https://github.com/krishnakt031990/Crawl-Wiki-For-Acronyms/blob/master/AcronymsFile.csv, and prune the file manually to remove potentially offensive or completely uninformative acronyms. This exercise generated a list of 250 acronyms. The complete list is given in our code repository.
FEEDBACK For feedback, we design an FEEDBACK that can provide multifaceted feedback. Specifically, each acronym is judged along five dimensions:
- Ease of pronunciation: How easy or difficult is it to pronounce the acronym? Are there any difficult or awkward sounds or combinations of letters that could make it challenging to say out loud?
- Ease of spelling: How easy or difficult is it to spell the acronym? Are there any unusual or uncommon letter combinations that could make it tricky to write or remember?
- Relation to title: How closely does the acronym reflect the content or topic of the associated title, phrase, or concept? Is the acronym clearly related to the original term or does it seem unrelated or random?
- Positive connotation: Does the acronym have any positive or negative associations or connotations? Does it sound upbeat, neutral, or negative in tone or meaning?
- Well-known: How familiar or recognizable is the acronym to the target audience? Is it a common or widely-used term, or is it obscure or unfamiliar?
Some of these criteria are difficult to quantify, and are a matter of human preference. As with other modules, we leverage the superior instruction following capabilities of modern LLMs to instead provide a few demonstrations of each task. Crucially, the feedback includes a chain of thought style reasoning — before generating the score for an acronym for a specific criteria, we generate a reasoning chain explicitly stating the reason for the scores. We use human evaluation to judge the final quality of the acronyms. An example of generated acronyms and associated feedback is given in Table 23.
Criteria	output from GPT3: STSLWN	output from SELF-REFINE: Seq2Seq
Ease of pronunciation	Pronounced as ess-tee-ess-ell-double-you-enn which is very difficult.	Pronounced as seq-two-seq which is easy.
Ease of spelling	Very difficult to spell.	Easy to spell.
Relation to title	No relation to the title.	Mentions sequence which is somewhat related to the title.
Positive connotation	Meaningless acronym.	Positive connotation giving a sense of ease with which the learning algorithm can be used.
Well-known	Not a well-known acronym.	Close to the word sequence which is a well-known word.
Total score	5/25	20/25
Table 23: Comparison of acronyms for input = "Sequence to Sequence Learning with Neural Networks"
U Constrained Generation
In this work, we introduce a more challenging variant of the CommonGen task, dubbed "CommonGen-Hard," designed to push the boundaries of state-of-the-art language models. CommonGen-Hard requires models to generate coherent and grammatically correct sentences incorporating 20-30 concepts, as opposed to the original task which presents a set of 3-5 related concepts. This significant increase in the number of concepts tests the model's ability to perform advanced commonsense reasoning, contextual understanding, and creative problem-solving, as it must generate meaningful sentences that encompass a broader range of ideas. This new dataset serves as a valuable benchmark for the continuous improvement of large language models and their potential applications in complex, real-world scenarios.
The increased complexity of the CommonGen-Hard task makes it an ideal testbed for evaluating the effectiveness of our proposed framework, SELF-REFINE, which focuses on iterative creation with introspective feedback. Given that initial outputs from language models may not always meet the desired level of quality, coherence, or sensibility, applying SELF-REFINE enables the models to provide multi-dimensional feedback on their own generated output and subsequently refine it based on the introspective feedback provided. Through iterative creation and self-reflection, the SELF-REFINE framework empowers language models to progressively enhance the quality of their output, closely mimicking the human creative process and demonstrating its ability to improve generated text on complex and demanding natural language generation tasks like CommonGen-Hard (Figure 16).
V Prompts
We include all the prompts used in the experiments in Figures 17-36:
- Acronym Generation: Figures 17-19
Code Optimization: Figures 20-22
Code Readability Improvement: Figures 23-24
- Constrained Generation: Figures 25-27
- Dialogue Response Generation: Figures 28-30
- Math Reasoning: Figures 31-33
- Sentiment Reversal: Figures 34-36
Recall that the Base LLM requires a generation prompt  $p_{gen}$  with input-output pairs  $\langle x_i, y_i \rangle$ , the feedback module requires a feedback prompt  $p_{fb}$  with input-output-feedback triples  $\langle x_i, y_i, fb_i \rangle$ , and the refinement module (REFINE) requires a refinement prompt  $p_{refine}$  with input-output-feedback-refined quadruples  $\langle x_i, y_i, fb_i, y_{i+1} \rangle$ . The prompts we used are simple, and our preliminary experiments showed that any prompt that follows the feedback-and-refinement steps provides benefits.
Figure 16: A comparison of SELF-REFINE and direct generation with GPT-3.5 on CommonGen-Hard.
- Sentiment Reversal We create positive and negative variants of a single review from the training set and manually write a description for converting the negative variant to positive and vice versa. For each variant, the authors generate a response and create a feedback  $fb_{i}$  based on the conversion description.
- Dialogue Response Generation We sample six examples as  $\langle x_i, y_i \rangle$  for the few-shot prompt for the Base LLM. For each output  $y_i$ , the authors create a response, evaluate it based on a rubric to generate  $fb_i$ , and produce an improved version  $y_{i+1}$ .
- Acronym Generation We provide the Base LLM with a total of 15 (title, acronym) examples. Then, for one title  $(x_{i})$  we generate an acronym  $(y_{i})$  using CHATGPT. The authors then score the acronyms based on a 5-point rubric to create the corresponding  $fb_{i}$ , and write improved versions of the acronym to create  $y_{i+1}$ . 3 such examples are used for REFINE and FEEDBACK.
- Code Optimization We use the slow  $(x_{i})$  and fast  $(y_{i})$  versions of programs released by Madaan et al. (2023) for Base LLM. We use their provided explanations (Madaan et al., 2023) for FEEDBACK and REFINE.
- Math Reasoning The prompts for the Base LLM are sourced from PaL (Gao et al., 2022) as  $\langle x_i, y_i \rangle$ . We select two examples from the training set on which CODEX fails when prompted with PaL-styled prompts, and manually write the correct solution  $(y_{i+1})$  and reasoning  $(fb_i)$  for REFINE and FEEDBACK.
- Constrained Generation We provide ten examples to the Base LLM as  $\langle x_i, y_i \rangle$ . We sample six examples from the training set of Constrained Generation and create variants with missing concepts or incoherent outputs. The missing concepts and the reason for incoherence form  $fb$ .
- Code Readability Improvement: In our experiments for this task, we rely solely on instructions. To generate feedback, we use the instruction, I have some code. Can you give one suggestion to improve readability. Don't fix the code, just give a suggestion. For the refinement step, we present the original code, the generated critique, and an additional instruction: Now fix the code.
Title: A Survey of Active Network Research  
Acronym: SONAR
Title: A Scalable, Commutative Replica Dictatorship for Practical Optimistic Replication  
Acronym: SCRATCHPAD
Title: Bidirectional Encoder Representations from Transformers  
Acronym: BERT
Title: Sequence to Sequence Learning with Neural Networks  
Acronym: Seq2Seq
Title: Densely Connected Convolutional Networks for Image Classification  
Acronym: DenseNet
Title: A Dynamic Programming Algorithm for RNA Secondary Structure Prediction  
Acronym: DYNALIGN
Title: Fast Parallel Algorithms for Short-Range Molecular Dynamics  
Acronym: FASTMD
Title: Real-Time Collaborative Editing Systems  
Acronym: COCOON
Title: Efficient Data Structures for Large Scale Graph Processing  
Acronym: EDGE
Title: A program to teach students at UT Southwestern learn about aging  
Acronym: SAGE
Title: Underwater breathing without external accessories  
Acronym: SCUBA
Title: An educational training module for professionals  
Acronym: LEAP
Title: Teaching a leadership program  
Acronym: LEAD
Figure 17: Initial generation prompt for Acronym Generation
Title: Underwater Breathing Product with no Accessories
Acronym: UBPA
Scores:
* Ease of pronunciation: UBPA is pronounced "uhb-puh". This is an easy acronym to pronounce. 4/5
* Ease of spelling: UBPA is easy to spell. 4/5
* Relation to title: UBPA stands for "Underwater Breathing Product for no Accessories" which is related to the title. 5/5
* Positive connotation: UBPA is a positive acronym. 5/5
* Well-known: UBPA is not a well-known acronym. 1/5
* Total score: 19/25
Title: Self-Contained Underwater Breathing Apparatus
Acronym: SCUBA
* Ease of pronunciation: SCUBA is pronounced "skoo-bah". This is an easy acronym to pronounce. 4/5
* Ease of spelling: SCUBA is easy to spell. 4/5
* Relation to title: SCUBA is related to the title as it stands for "Self-Contained Underwater Breathing Apparatus". 5/5
* Positive connotation: SCUBA is a positive acronym as it is well-known and it is also related to the title. 5/5
* Well-known: SCUBA is a very well-known acronym. 5/5
* Total score: 23/25
Figure 18: FEEDBACK prompt for Acronym Generation
Title: Computer Science Conference and Education
Acronym: CSCE
* Ease of pronunciation: CSCE is pronounced "see-CEE". This is an easy acronym to pronounce. 4/5
* Ease of spelling: CSCE is easy to spell. 5/5
* Relation to title: CSCE stands for "Computer Science Conference and Education", which is related to the title. 5/5
* Positive connotation: CSCE is a positive acronym. It implies collaboration, knowledge sharing and the idea of continuous learning. 5/5
* Well-known: CSCE is not a well-known acronym. 2/5
* Total score: 20/25
Okay, let's use this feedback to improve the acronym.
Title: Computer Science Conference and Learning Experience
Acronym: CSCLE
* Ease of pronunciation: CSCLE is pronounced "see-slee". This is an easy acronym to pronounce. 4/5
* Ease of spelling: CSCALE is easy to spell. 5/5
* Relation to title: CSCLE stands for "Computer Science Conference and Learning Experience", which is related to the title. 5/5
* Positive connotation: CSCALE is a positive acronym. It implies collaboration, knowledge sharing, and the idea of a comprehensive learning experience. 5/5
* Well-known: CSCLE is not a well-known acronym. 5/5
* Total score: 23/25
Figure 19: REFINE prompt for Acronym Generation
# slower version:  
import numpy as np  
N, K = map(int, input().split())  
H = np.array(list(map(int, input().split())) + [0] * K, dtype=np.int64)  
table = np.full(N + K, 10 ** 10, dtype=np.int64)  
table[0] = 0  
for i in range(1, N):  
    table[i:i + K] = np.minimum(table[i:i + K], np.abs(H[i:i + K] - H[i - 1]) +  $\rightarrow$  table[i - 1])  
print(table[N - 1])  
# optimized version of the same code:  
N, K = map(int, input().split())  
H = tuple(map(int, input().split()))  
table = [0] * N  
for i in range(1, N):  
    table[i] = min(abs(H[i] - H[j]) + table[j] for j in range(max(0, i - K), i))  
print(table[N-1])
Figure 20: Initial generation prompt for Code Optimization
a, b = input().split()
n = int(a + b)
flag = False
for i in range(n):
    if i ** 2 == n:
        flag = True
        break
print('Yes' if flag else 'No')
# Why is this code slow?
# This code is slow because it is using a brute force approach to find the square  $\leftrightarrow$  root of the input number. It is looping through every possible number  $\leftrightarrow$  starting from 0 until n. Note that the square root will be smaller than n, so  $\leftrightarrow$  at least half of the numbers it is looping through are unnecessary. At most,  $\leftrightarrow$  you need to loop through the numbers up to the square root of n.
Figure 21: FEEDBACK prompt for Code Optimization
a, b = input().split()
n = int(a + b)
flag = False
for i in range(n):
    if i == 2 == n:
        flag = True
        break
print('Yes' if flag else 'No')
# Why is this code slow?
# This code is slow because it is using a brute force approach to find the square  $\leftrightarrow$  root of the input number. It is looping through every possible number  $\leftrightarrow$  starting from 0 until n. Note that the square root will be smaller than n, so  $\leftrightarrow$  at least half of the numbers it is looping through are unnecessary. At most,  $\leftrightarrow$  you need to loop through the numbers up to the square root of n.
# Improved version:
a, b = input().split()
n = int(a + b)
flag = False
for i in range(1000):
    if i == 2 == n:
        flag = True
        break
print('Yes' if flag else 'No')
Figure 22: REFINE prompt for Code Optimization
I have some code. Can you give one suggestion to improve readability. Don't fix the code, just give a suggestion.
{code}
Figure 23: FEEDBACK prompt for Code Readability
I have some code. Can you give one suggestion to improve readability. Don't fix the code, just give a suggestion.   
{code}   
{suggestion}   
Now fix the code.
Figure 24: REFINE prompt for Code Readability
////  
Concepts: ['create', 'ferry', 'silhouette', 'stream', 'terminal']  
Sentence: light streams through windows at the railroad and ferry terminal creating a beautiful silhouette  
////  
Concepts: ['chair', 'couch', 'hang', 'room', 'wall']  
Sentence: A room with a couch, chairs and art hanging on the wall.  
////  
Concepts: ['boat', 'building', 'harbour', 'moor', 'quay']  
Sentence: the harbour and port with fishing boats moored and old buildings on the quay  
////  
Concepts: ['admirer', 'arrive', 'commander', 'crowd', 'greet']  
Sentence: military commander is greeted by a crowd of admirers as he arrives
Figure 25: Initial generation prompt for Constrained Generation (truncated)
Concepts: ['animal', 'catch', 'horse', 'lasso', 'ride']  
Sentence: The horse catches the lasso and rides on it.  
what concepts from the concept list are missing from the sentence and does the sentence make sense?
Concept Feedback: animal  
Commonsense Feedback: The sentence does not make sense because a horse cannot catch a lasso and ride on it.
Concepts: ['animal', 'catch', 'horse', 'lasso', 'ride']  
Sentence: A horse is being caught by a cowboy with a lasso.  
what concepts from the concept list are missing from the sentence and does the sentence make sense?
Concept Feedback: animal, ride
Commonsense Feedback: NONE
Figure 26: FEEDBACK prompt for Constrained Generation (truncated).