MAF: Multi-Aspect Feedback for Improving Reasoning in Large Language Models
Authors: Deepak Nathani
David Wang
Liangming Pan
William Yang Wang
Venue: EMNLP 2023 Main (2023)
Keywords: large language models
feedback generation
reasoning
self-correction
in-context learning
prompting
prompt engineering
error correction
TL;DR: We introduce a multi-aspect feedback approach to refine LM generated solutions and reasoning chains.

Abstract: Language Models (LMs) have shown impressive performance in various natural language tasks. However, when it comes to natural language reasoning, LMs still face challenges such as hallucination, generating incorrect intermediate reasoning steps, and making mathematical errors. Recent research has focused on enhancing LMs through *self-improvement* using feedback. Nevertheless, existing approaches relying on a single generic feedback source fail to address the diverse error types found in LM-generated reasoning chains. In this work, we propose **Multi-Aspect Feedback**, an iterative refinement framework that integrates multiple feedback modules, including frozen LMs and external tools, each focusing on a specific error category. Our experimental results demonstrate the efficacy of our approach to addressing several errors in the LM-generated reasoning chain and thus improving the overall performance of an LM in several reasoning tasks. We see an improvement of up to 20% in Mathematical Reasoning and up to 18% in Logical Entailment.

MAF: Multi-Aspect Feedback for Improving Reasoning in Large Language Models
Deepak Nathani, David Wang, Liangming Pan, William Yang Wang
University of California, Santa Barbara, Santa Barbara, CA {dnathani, d_wang, liangmingpan, william]@ccs ucsb edu
Abstract
design principles. However, previous works on it- erative refinement overlook this requirement and collect generic feedback from multiple sources or collect a single one from the Language Model it- self.
Language Models (LMs) have shown impres- sive performance various natural language tasks. However, when it comes natural lan- guage reasoning, LMs still face challenges such hallucination, generating incorrect interme- diate reasoning steps, and making mathemati- cal errors. Recent research has focused on en- hancing LMs through self-improvement using feedback. Nevertheless, existing approaches relying on a single generic feedback source fail address the diverse error types found in LM- generated reasoning chains. In this work, we propose Multi-Aspect Feedback, an iterative refinement framework that integrates multiple feedback modules, including frozen LMs and external tools, each focusing on specific error category. Our experimental results demonstrate the efficacy of our approach to addressing sev- eral errors in the L.Mggnnrrted reasoning chain and thus improving the overall performance of an LM in several reasoning tasks. We see a rel- ative improvement of up to 20% in Mathemat- ical Reasoning and up to 18% in Logical En- tailment. We release our source code, prompts, and data' accelerate future research.
this work, we first investigate whether the use of generic feedback a bottleneck in addressing the diverse range of errors present in the reasoning chains generated by LMs. We posit that utilizing generic feedback for a broad spectrum of errors may result in vague non non-actionable feedback for specific errors. This is due to main factors: (i) the inclusion of multiple error categories within a single prompt, which not only increases the size of the prompt but also poses a challenge cur- rent models that have a limited context length and struggle with long texts, and (ii) the model is bur- dened with identifying a multitude of eπror types in single instance, which degrades the quality of the generated feedback.
To surmount these challenges, we introduce Multi-Aspect Feedback (MAF), a novel general- purpose iterative refinement framework that em- ploys collection of specialized feedback modules, including pre-trained LMs and external tools, each tailored to address specific eтror category. Our framework includes base model to generate an initial solution, multiple feedback modules, and refiner model to revise the solution using feed- back. Contrary to previous works, our feedback modules can use one of two refinement strategies: Eager Refinement and Lazy Refinement $3.4. When using the Rager-Refine mode, the solution is revised immediately before moving on to the next feedback, however, for Lazy Refinement, we first collect feedback from multiple feedback sources and then refine the solution using this collective feedback.
1 Introduction
Recent research Language Models has focused on augmenting them with external tools (Schick et al., 2023; Paranjape et al., 2023), learning from feedback (Ouyang et al., 2022; Akyurek et al., 2023) and reefeeet (Madaan et al., 2023; Paul et al., 2023; Shinn et al., 2023). Iterative-Refinement has been necessary tool in human evolution and problem-solving. Moreover, humans seek feedback from domain-speciff knowl- edge sources. For instance, an architect tasked with creating an environmentally friendly and struc- turally sound building design will require targeted feedback from civil engineers for structural in- tegrity, and sustainability experts for eco-friendly OOr source code can be found here
In devising MAF, we first classified errors LM- generated reasoning chains based on Golovneva et al. (2022) and also identified some new error

---

Input passed to an initial Model
Initial Output o is then passed to collection of feedback modules F
Input
Model M
Generated feedback and previous solution passed Refiner
Programming Syntax
Redundancy
Refiner R
Calculator
Refined solution is now used to generate new solution until we reach correct solution or
Figure 1: Overview of the Multi-Aspect Feedback framework. We loop over steps 3 and until we reach a final solution 0' or up to a set number of iterations.
categories. Subsequently, we decoupled our feed- back modules such that each module focuses single category. This strategy not only ele- vates performance but also allows one to leverage specialized tools tailored for distinct error types, such as utilizing a code interpreter for syntax er- rors in generated Python programs instead of re- lying on generic LM feedback. Through a mod- ular design, we ensure that each module of our framework is replaceable depending the task. We conduct extensive experimentation of complex reasoning tasks such Mathematical Reasoning (Shi al.. 2023), Logical Reasoning (Dalvi et al., 2021), and Multi-Hop Question An- swering (Dua et al., 2019), and establish that MAF effectively addresses the challenges previously dis- cussed, and significantly enhances the performance of Language Models in complex reasoning tasks. We compare our method against the Base LMs and Reff-Refine (Madaan et al., 2023), which also implements iterative refinement. During our experi- ments, we find that -ven-Refi $4 the solution can lead worse performance, and thus we treat the number of iterations as hyperparameter. We also propose an Oracle Verifier setting $4, where we assume we have access to an "answer verifier" and stop the refining process once we have reached the correct answer. To avoid unfair comparisons,
we test Self-Refine under this setting and find that MAF provides actionable and specific feedback. To summarize, the main contributions of this work are as follows:
We propose a novel framework that decou- ples the feedback generation process for dif- ferent error categories. This allows us to use error-specific tools and LMs. Moreover, the proposed framework is modular and all the modules Plug-and aad--ndd-Pa Play. This allows for better flexibility new tasks and the usage of advanced models as they become available.
variety
Additionally, we propose two different types of refinement strategies, Eager Refinement and Lazy Refinement. Eager Refinement allows immediate revision of solution by a feedback module to avoid conflicts, while Lazy Refinement improves efficiency by per- forming revisions from multiple feedback modules together.
3.
We show that our framework outperforms the base Language Models and similar Iterative Refinement baselines on several complex rea- soning tasks such as Math reasoning (GSM-
IC), Logical Reasoning (Entailment Bank), and Multi-hop Question Answering (DROP).

---

2 Related Work
Task
Feedback Type
Type
ER
Programming Syntax
Interpreter
Chain-of-Thought Reasoning There has been plethora of research prompting Large Lan- guage Models to improve their reasoning capabil- ities. Wei al. (2023) found that prompting the models to generate reasoning chain few-shot setting before solving a problem improves the per- formance by huge margin. Further, Kojima et al. (2022) found that in zero-shot setting prefixing the solution generation with Ler's think step-by step-by-ster by-step has the same effect generating intermediate rea- soning chain and improves the performance. Fol- lowing this work, Wang et al. (2022) proposed sam- pling multiple outputs from the model and select- ing the final answer by majority voting. Zhou et al. (2023) also showed that decomposing the original question into smaller sub-problems can give a per- formance boost. Madaan et al. (2022) and Chen et al. (2022) showed that models trained on code can be used to generate program-of- of-thought simi- lar to chain-of-thoughh which enables the model use language interpreter for translating mathe- matical calculations into code. In this work, we use LMs' in-context learning abilities to implement the LM-based feedback and refiner modules.
Variable Naming
OpenAl
Math
Redundancy
OpenAl
Reasoning
Commonsense
OpenAl
Missing Step
OpenAl
Logical Reasoning
Redundancy
OpenAI
X XXXX
Repetition
OpenAl
Hallucination
OpenAl
Redundancy
OpenAl
Question Answering
Factuality
OpenAl
Commonsense
OpenAl
Missing Step
OpenAI
Table 1: Feedback modules used for each task and their types. OpenAl type modules use LLM to provide feed- back. Feedback Modules with Eager-Refine (ER) en- abled, refine the solution without waiting for feedback from the other feedback modules. ($3.4).
Learning from Feedback Schick et al. (2022) took the first step towards iteratively fixing the problems and introduced the idea of training multi- ple instances of the same model to assist in different stages of problem-solving. collaborative editing framework. (Madaan et al., 2023) proposed to use the same model for gen- erating initial solution, feedback generation, and refiner. Furthermore, recently there have been mul- tiple works exploring the use of natural language feedback to improve performance. Shinn et al. (2023) proposes converting different types of feed- back into natural language and storing the feedback in a memory buffer to revisit later. Akyurek et al. (2023) train critique generator to maximize the end-task performance of a larger base model using reinforcement learning. Paul et al. (2023) proposes a framework to fine-tune LMs for generating inter- mediate reasoning steps while interacting with a trained critique model to bridge the gap between small and large models. Recent research (Wu al., 2023) has also demonstrated the benefits of incor- porating decoupled fine-grained feedback for each error category by using a set of fine-tuned reward models. We defer to Pan et al. (2023) for a com- prehensive discussion of research on automated feedback generation and correction.
was introduced as a
Tool Augmented LLMs However, even with these advanced prompting techniques, the LMs still fail for problems that require external knowledge and are still prone to problems such as hallucina- tion and incorrect reasoning. To circumvent these issues, several recent works have introduced the use of external tools such as calculator, code in- terpreter, knowledge retrieval (Karpas et al., 2022; Chen et al., 2022: Schick et al., 2023). Yang et al. (2023); Patil et al. (2023); Shen et al. (2023) show that one can teach LLMs to use tools by generating API calls. Li et al. (2023) also released large dataset to enable research in the field of augmented Language Models. Hao al. (2023) propose to rep- resent each tool as token and learn tool-specific embeddings to increase the robustness of these lan- guage models for using external APIs. However, reasoning is an iterative task and it is difficult to find plausible answer for several problems in one- shot even when these LLMs augmented with tools. This has inspired some recent works to use iterative refinement frameworks. In our work, use external tools for generating feedback for cer- tain error categorics such as Programming Syntax Errors, Calculator for mathematical equations etc.
Despite these advancements, there is lack of empirical evidence supporting the effectiveness of decoupled multi-aspect feedback for iterative re- finement. Our work addresses this gap by demon- strating that a generic feedback module, is insuffi-

---

cient address the diverse range of potential errors in language model responses. To overcome this lim- itation, we propose a suite of feedback modules, each specifically targeting a particular error cate- gory and providing detailed feedback to improve both reasoning and solution quality.
process. The solution often needs be refined over time with every iteration bringing us closer the correct answer. We follow the same principle in this work, where we initially produce a solution and then proceed to refine it based on actionable feedback.
3 MAF: Mulei-Aspect Feedback
3.3 Feedback Modules
Feedback generation is an involved task, providing comprehensive list of common issues encoun- tered in model outputs. The feedback generation is accomplished through a variety of tools known as feedback modules, which may include external tools, frozen LLMs, fine-tuned models, and scor- ers. These modules are used to provide actionable feedback based on the initial solution.
3.1 Overview
this work, we present an iterative refinement framework with an explicit focus on decoupling feedback generation for different eπror categories. This approach allows us to systematically address different error types in the generated solutions. Our proposed framework has three crucial components: base language model M that generates an ini- tial solution O. A collection of feedback mod- ules {fo, f1, fe...ff., each focusing on a single error category, collectively these modules gener- ate a multi-aspect feedback F. And a Refiner R that generates refined solution O' based on initial solution O and feedback F.
Each type of feedback module is suited to ad- dress specific types of errors. For instance, an external tool like code interpreter would be ideal for providing feedback syntax errors in code, while fine-tuned model could provide more nu- anced feedback on issues such as redundancy or hallucination. This decoupled approach allows us to address errors in a more targeted manner, im- proving the overall quality of the refined solution. Table shows all the feedback modules used in our work.
While there other works that also iterative refinement (Madaan et al., 2023: Akyurek al., 2023; Peng al., 2023). to the best of our knowledge, we are the first to explore the effect of decoupling different types of feedbacks. Taking inspiration from ROSCOE (Golovneva et al., 2022), we categorize feedback into ten distinct categories: arithmetic, programming syntax, variable naming, missing step, coherency, redundancy, repetition, hallucination, commonsense, and factuality. Definitions of these error categories are provided in Appendix A. Moreover, a feedback module can be tool such as a code interpreter for syntax feedback, a calculator for ari thme ic feedback, a knowledge graph for factuality or commonsense feedback, Language Model, or even fine-tuned model. It is important to note that the feedback generation process is not limited these categories and can be extended include other categories as well. The overall process is illustrated in Figure I and Algorithm 1.
Furthermore, it's worth noting that merely speci- fying the error categories in single prompt doesn't yield satisfactory results. We see two main reasons for this, firstly because the model is tasked with focusing on multiple errors simultancously and sec- ondly, as the number of error categories increase, the context length increases as well which results in high-quality feedback for the first few error cat- egories and steep decline in quality for the rest. While some of the previous works (Wu et al., 2023; Paul al., 2023) have explored using fine- ane-grained feedback and achieved promising results.
Moreover, our feedback modules can choose be- tween two refinement strategies: Eager-Refing or Lazy-Refine. We discuss this distinction in more detail in the next section.
3.4 Refinement
In this work, we reuse the initial model M as a Refiner. During the refining phase, the refiner is given a solution and multi-aspect feedback and asked to revise the solution according feedback. We found that these large models are proficient at the refining task, demonstrating a marked improve- ment in the quality of the final solution compared
3.2 Initial Generation
We use a large model such as GPT3.52, GPT4 (Ope- nAI, 2023), to generate an initial solution. How- ever, generating just one solution isn't ideal for the reasoning process since reasoning is an iterative
Shupss//opeaai.co///bbgggagg

---

cunlight the comrer thon the Krong! fhis enteilmert step 15 welld but irrelecen: to the hypotiexia. can rerOve this slop voliauly.
seati: crount diylight
apealext in the sUEAME
sent2: united statee to located the northern henisphere
DAP-REFINED (CORSECT)
sonc3: new yozr new york state 10 5 eraae loosted in the untted
REFETICION PERDRACK
GENERATIOR sent2 and sant3 -> intl: new yark
incl and menté 1862: turn
states of america
the northern
the northeen iuriaphiee
senté: luoe is isring Lhe aumes in the northeen henisphere
wrong!
steply
repeata the premise
int!
int2: Junc
sents: the snount af caylight soletion
cunclusion in.2. Ba can EMEROW thes alep uniecoly.
seati int2 hypothezia: naw yock scate the greatest sunlight during Jane
the
daylightt
MISSING SIEP FREDBACK
inti and seati >> hypothmais: new
munlight dusica June estalinent intornediate pronice that astablishes
sialu bay Ube
INITIAL GENERAZION
nissing
senc2 and sent3 -> iatl: nev yozk the suckiwen lanisoluze
prent your gune,
SELF-RBFINE (INCONRECT) GENERAZIOE
and santé -> int?: June Juring
conslucion.
sentZ and cent? >> intl: new yorr is the northers serispheze
seniesherv
suntd +> iat2: juse in ducing the in
inti and BELP-REPINE PESOBACK the weang! presize
the vintes
secti and int2 hypothezia: york
and sentl -2 hypothesis: nox your state has the greetest aunlight daring June
sunlight during
northern hontophers 6oos antail intermediate conclizaion 1nt2 that June during the aurmer in the northern heni sphere. inti and swnti hypothezis: new yosk sta.e has the groacest auilight during June weong! This stap Le aissing the key oceniew
chst te 1o aurner in new york durtnng jone. ue need to cdd this.
Figure 2: Comparison of the feedback generated by MAF and Self-Refine.
to the initial output.
make it difficult to include all the feedback in the prompt. To address this issue, we use Selective Summarizatin approach. We only select parts of feedback that point a problem. Figure 2 shows the "summarized" feedback generated by our approach. This simple selective summariza- tion approach makes the feedback succinct and also proves to be less distractive to the models dur- ing the refining phase. This approach allows us to effectively combine all feedback together and use models with smaller context lengths to some extent.
As mentioned in the previous section, we use two refinement strategies: Eager-Refinement and Lazy-Refinement. The eager-Refinement approach is used for feedback types that can cause conflicts during refinement. An example of such as feedback module would be Variable Naming (VN), which is used to correct variable names in the generated code. This module can cause conflicts with others because when the other modules are referencing variable that supposed to be changed according the VN feedback and can render the program inexecutable refined incorrectly. Whereas, in the lazy refinement strategy feedback from the multiple modules is concatenated together along with the appropriate eиror categories to make a single multi- aspect feedback. This collective feedback is then passed to the Refiner model in order to get revised solution. Another advantage of this approach is the increased efficiency by refining once for multiple errors and the added flexibility.
Experiments
4.1 Datasets and Metries
Mathematical Reasoning The GSM8K dataset, presented by Cobbe al. (2021), is a comprehen- sive compilation of high-quality grade school math problems. GSM8K has proven to be a great re- source for testing LLMs on mathematical word problems, however, the performance on this dataset has been saturated for a while.
However, we also found that the smaller open- source models like LLaMA (Touvron et al., 2023), Alpaca (Taori et al., 2023) and Vicuna² often fail adhere the feedback provided thus making the refinement process ineffective. These smaller models also have smaller context lengths which
To avoid that problem, conduct experiments on harder variant of this dataset, GSM-IC (Grade- School Math with Irrelevant Context), that was introduced by Shi et al. (2023). We run our main experiments on a randomly sampled subset of 500 problems from GSM-IC and use % solve rate as

---

Algorithm MAF algorithm
Require: Input x, model M, Refiner R, number of iterations T
Require: Eager-refine (81, $2,....8) and m Lazy-Refine {pi, Pm Feedback Modules
1: Initialize output yo from M
2: while T do
3: for j +- to n do
D Eager-refine Feedbacks D Generate Feedback
4:
f8j + senerate_feedbacki 34)
5: if f8j indicates revision is required then
6:
Di + revise(R, (R, fs Vi)
b> Revise solution
7:
F с
Initialize empty feedback
8:
for k + 1 to m do fpk generate_ cedbackkky ys)
>> Lazy-refine Feedbacks
10:
if fpj indicates revision required then F+ F+ fPk
if F not empty then
Mi+1 - revise(R, [Pks V/s)
else
15:
yi+1 = y/
the metric.
nAI API and recently proposed iterative refinement framework, Self-Refine (Madaan et al., 2023). To provide a fair comparison and avoid randomness in the generated answer, we used Greedy Decoding for all our experiments.
Logical Reasoning EntailmentBank, as de- scribed by Dalvi et al. (2021), is datuset that contains multistep entailment trees. We use the validation of Task provided by the authors for our experimentation. For the metrics, we do not use the automated metrics provided by the original work because these metrics expect the trees to be similar in structure to gold trees. However, that is not a fair comparison because we find that there exist multiple correct entailment trees for given hypothesis and information. Thus, conduct human evaluation and ask humans evaluate if the hypothesis can be entailed from the predicted tree.
We follow (Madaan al., 2023) and use 8-shot Program of Thought (Chen et al., 2022) for GSM- IC since the Python program written by the model acts as a "calculator" for mathematical equations.
The input for Entailment Bank includes a hypoth- esis and the supporting text, and we are limited by the context length of the current models. Hence, we use 4-shot prompting for this dataset. We use standard few-shot prompting (Brown et al., 2020), because the Entailment Tree itself acts like rea- soning chain.
Question Answering DROP (Dua al., 2019) is a question-answering dataset. The Discrete Reason- ing Over the Content of Paragraphs (DROP) dataset tasks
Similarly, the examples in the DROP dataset have a passage and an accompanying question, so we use a 3-shot Chain of Thought Wei et al. (2023) prompting. We also provide an instruction speci- fying that the model should select either a number, date, span from the passage to answer the ques- tion as shown in Appendix C.
is designed for complex that require multi-step reasoning over text passages. It presents valuable benchmark for our iterative refinement framework, as the multi-step nature of the questions offers ample opportunitics for gen- erating feedback and refining guide the model toward a correct solution. We use output parsing for answers and use correct answers as the final metric.
we use the prompts provided by the authors for GSM8K in their work for GSM-IC and write ment Bank. Self-Refine has three modules, initial generation, feedback, and refiner. We use the same parameters and prompting strategy as the corre- sponding baseline for the initial generation. The feedback module and refiner module are imple-
For
prompts for DROP and Entail-
4.2 Baselines
this work, we focus on comparing our method with the Base LMs using the Ope-

---

Model
EB
GSMIC
GSM8K
DROP
G
56.1
76.2
69.2
72.3
Base LM
+SR
53.5
|2.6
77.0
10.8
69.2
10.0
62.0
Selt Reff-R (Oracle)
+SR*
54.5
[1.6
87.0
110.8
77.4
18.2
77.5
14.2
MAF
(Oracie)
+MAF
54.5
11.6
77.4
11.2
69.8
10.6
66.2
J6.1
+MAF*
60.4
14.3
91.4
115.2
73.4 14.2
76.4
14.1
Accuracy
ChatGPT
60.4
72.0
71.8
70.7
+SR
65.8 15.4
76.0
74.6
12.8
45.5.25.2
+SR*
7..4
78.076.0
79.4
17.6
73.2
+MAF
68.4
75.8
73.2
11.4
67.912.8
+MAF*
71.7 (11382.8 10.8
76.6
14.8
72.712.0
2 Iterations
Table Experimental results for Entailment Bank (EB), GSMIC, GSMSK, DROP dataset descrihed in $4, SR represents Self-Refine (Madaam et al., 2023), and MAF represents our method. represents the Oracle Verifier setting ($4.4), The best score for standard setting is in bold and underlined for the Oracle Verifier setting.
Figure 3: Accuracy for baselines and MAF on GSM- IC using ChatGPT under Standard and Oracle Verifier setting $4.4.
mented using the standard few-shot prompting with 3 in-context examples for all datasets. Further de- tails about the implementation of these baselines can be found in Appendix C.
strategy summarize feedback. Since our feed- back modules are instructed inspect each inter- mediate step, they also include the steps with no mistakes in the generated feedback. However, feed- back on these lines is not useful as such because there no change in those steps. Thus we look for the steps with feedback "looks good" and re- move those steps. This simple approach helps increase the efficiency of our method by being able to include multiple feedbacks in lazy-refiner.
4.3 Implementation
We implement MAF (Multi-Aspect Feedback) fol- lowing the basic structure defined in $3 and Algo- rithm The iterative refinement process continues until we reach the desired output quality or a task- specific stop criterion, to a maximum of 4 iter- ations. Our method includes an initial generation, refiner, and feedback modules shown in Table 1. The initial generation module uses the same param- eters and prompting strategies as the corresponding baseline. Our Eager-Refine module uses 3-shot standard prompting and our Lazy-Refing which includes feedback from multiple modules, uses a 2-shot standard prompting approach. Our OpenAI- based feedback modules also use 3-shot standard prompting following (Madaan et al., 2023) with an ciror-specific instruction.
4.4 Results
In this section, we discuss the performance our method and compare to the baseline. We also discuss an ablation experiment studying the contri- bution of each feedback module. The main results of this work are shown in Table 2. We find that our method can outperforms the base LMs and Relf-Refine
diverse set of reasoning datasets. For MAF, we report the results after 2 iterations, and following Madaan et al. (2023) report the Self- Refine results after iterations.
To provide fair comparison, prompts for Baselines and Self-Refine wherever possible. We use Greedy Decoding for all our OpenAI-based' modules to avoid any randomness. As mentioned in $3.3, we use a selective summa- rization approach to fit all the lazy-refine feedbacks in our refiner module. We use basic rule- based
the same
We found that GPT3.5 can generate better feedback if an error is present in the solution, however often points out spurious errors even when they not present. On the other hand, ChatGPT is more conservative in its feedback generation and often fails detect the error even in an erroneous solution. Because of this conflicting behavior from the two models, our method able to achieve similar performance using both models,
GPT3.5 vs ChatGPT
"Our experiments with GPT3.5 TText-Davinci-003) and ChatGPT (gpt-3.5-turbo are conducted using OpenAl API hetween April-2023 and August-2023.

---

even though ChatGPT is generally considered to be a better model than GPT3.5.
outcomes of our two proposed refinement strate- gies: Lazy vs Eager Refinement. The ablation studies are conducted on the GSM-IC dataset. Due to resource constraints, we conduct the following ablation studies on a smaller random subset of size 100.
Over-Refining Due to the behavior described above, we face the problem of Over-Re fning. This means that once we have reached an optimal solution for reasoning problem, forcing the LM to refine it further deteriorates the performance and the quality of the reasoning chain. As shown in Table 2 and Figure 3, Self-Refine (Madaan al.. 2023) also suffers from this problem in the DROP dataset. Morcover, all the other itcrative refine- ment framework such as Self-Refine uses a stop condition and stop the refining process if the gen- erated feedback does not point out any problem. We however cannot take advantage of this stop condition because of the interplay between mul- tiple feedback modules. For example, let's say our Missing Step module stops iteration & due to no missing steps at that point in time since the other feedback modules continue to refine the so- lution and self-refinement an imperfect process, it might introduce a Missing Step error in further iterations.
Model
Standard
Oracle
ChatGPT
73.0
+MAF
79.0
84.0
Variable Naming (VN)
79.0 10.0
83.0 U1.0
Redundancy (Red)
70.0 19.0
80.0 (4.0
Commonsense (Com)
73.0 J6.0
80.0 (4.0
Missing Step (MS)
79.0 10.0
84.0 10.0
MAF (VN, Red, Com)
79.0
85.0
Table 3:
This table shows the contribution of each feed- back module in MAF for the GSM-IC dataset. symbol front a module denotes accuracy our method after removing that feedback module. MAF (VN, Red, Com) shows the accuracy of our method using only the best- best- performing feedback modules.
Hence, we treat the number of iterations as hyperparameter and find that 2 iterations work best for our method.
Contribution of different Feedback Modules We the contribution of each feedback mod- ule by removing that module and calculating the accuracy under standard and Oracle verifier set- tings. While the standard setting highlights the potential negative impacts of some of the modules, the oracle verifier setting highlights the absolute contribution of each module. This highlights important finding that using the error categories is paramount gaining performance. Results for this ablation study can be found in Table 3.
Oracle Verifier Since the main focus of this work is to improve the incorrect initial generations from the model, we also evaluate the models under an Oracle Verifier setting. In this setting, assume access "Oracle Verifier" which can judge the final answer generated by the model. If the final answer correct, we stop the refinement process for that test sample, otherwise, we let the model continue refining the solution. It is important to note however that the model is not privy to this verification, and hence can still stop further refine- ment by not generating actionable feedback in the next iteration. Under this setting, we report results for both Self-Refine and MAF after iterations. Even under this modified setting, we see that our method can outperform Self-Refine because we generate diverse feedback and thus can improve more solutions in the GSM-IC dataset.
We also calculate the accuracy of our method when using the three best-performing modules as the only source of feedback. We found that this does recover the performance of our method. Even though Variable Naming and Missing Step modules do not affect MAFs performance by huge margin, still makes method more robust possible distribution shift. Moreover, the Variable Naming module's main contribution not increasing the performance, but rather to increase the readability of the code and not confuse users with unclear names.
4.5 Ablation
In this section, we perform two ablation studies. Firstly, we analyze the impact of each feedback module, identifying those with the most significant effect on performance. Secondly, we compare the
Note that we did not remove the Programming Syntax checker module as we use Program of Thoughts which requires a Python interpreter.

---

number of feedback modules.
Model
Standard
Oracle
G
71.0
71.0
Conclusion
+MAF
67.0 14.0
85.0 114.0
In this work, we present Multi-Aspect Feedback (MAF), a novel iterative refinement framework that decouples the feedback modules and takes advan- tage of error-specific tools to generate feedback. We demonstrate the performance of our framework on set of diverse reasoning tasks and compare it with other iterative refinement baselines.
+ +Only Lazy-Refine
66.0 15500
81.0 110.0
+ Only Rager-Refine
66.0 15.0
85.0 114.0
ChatGPT
73.0
73.0
+MAF
78.0 15.0
81.0 18.0
+Only Lazy-Refine
69.0 [4.0
80.0
17.0
+ +Olly Eager-Refine
73.0 10.0
80.0 17.0
Contrary to previous works, we found that Over- Refinement can be problem in iterative refinement frameworks since models are not certain if their own answer is correct. Our work also draws focus on the necessity devise better feedback methods and call for augmenting Language Models with them. We hope this work will inspire further re- search in this area and to this end, we make all our code, data, and prompts available.
Table 4: Accuracy on GSM-IC when using different refinement strategies under Standard and Oracle set- tings. MAF shows the performance of method when using both Lazy and Eager refine in tandom. Only Lazy- Refine means feedback modules use Lazy-Refine and similarly Only Eager-Refire means all modules Eager- strategy. The best score for each setting is in bold.
Lazy vs Eager Refinement To illustrate the com- plementary nature of our two proposed refinement strategies, evaluate the performance of our it- erative refinement framework using all feedback modules in either Lazy or Eager mode. The re- sults are presented in Table 4, which showeases the performance of our framework under different re- finement settings. In this table, "MAF" corresponds to the results obtained by combining both Lazy and Eager Refinement strategies, as defined in Ta- ble I, while 'Only Lazy/Eager-Reen Refine' displays the results of our framework when utilizing only one type of refinement strategy.
Limitations
The main limitation of our approach is that the base models need have sufficient in-context learning abilities to process the feedback and refine the solu- tion. Even with in-context learning abilities, these models are not perfect and thus can still make mis- takes while refining the solution even when correct feedback is given.
All the experiments conducted in this work use large powerful models provided by OpenAL. We find that open-source LMs such Vicuna, and Alpaca can generate decent initial solutions but are not capable of refining their own solutions. Thus, we leave the investigation of improving open- source models to future work.
The results clearly demonstrate that our frame- work, which leverages a combination eager and lazy refinement, consistently matches or outper- forms using either strategy in isolation, across both the standard and oracle verifier settings.
Another limitation inherent in our approach is the reliance on fixed set of Feedback Modules. Our method necessitates the pre-selection of feed- back modules before execution, which in turn de- mands human intervention to determine the appro- priate feedback categories for each new dataset or domain. Future research could explore novel methods that can dynamically and autonomously determine the most suitable feedback modules for specific problems in real-time.
Practical considerations also favor the use of hybrid approach incorporating both eager and lazy feedback. Relying solely on lazy feedback can lead to a situation where multiple feedback categories are condensed into a single prompt. Despite our feedback summarization technique, this can result in the iteration prompt exceeding the context win- dow limit of many widely available models. Con- versely, exclusively employing cager feedback may result in rewriting the solution for each feedback module, leading high token usage. While us- ing all modules in Rager-Refine mode can closely approach the performance of "MAF' (as shown in Table 4), it is not scalable when dealing with large
Ethics Statement
The experiments in this work were performed with models that are not open-sourced and are ever- changing. Moreover, these models are expensive to use, and thus research using these models requires

---

an enormous amount of funding. The existing liter- ature lacks details about the datasets that are being used to train these huge models or the filtering mechanism that is being used to clean the polluted corpora.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi. Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requir- ing discrete reasoning over paragraphs. In Proceed- ings of the 2019 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies. Volume (Long and Short Papers). pages 2368-2378, Min- neapolis, Minnesota. Association for Computational Linguisties. 2,6 6
Furthermore, there is always possibility for bad actors to use our method to generate more toxic or harmful text. Our approach does not guard against this.
Olga Golovneva, Moya Chen, Spencer Poff, Martin Corredor, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz. 2022. Roscoe: A suite of metrics for scoring step-by-step reasoning. 1, 4, 12, 13
Acknowledgements
This work was supported by the National Science Foundation award #2048 122. The views expressed are those of the authors and do not reflect the offi- cial policy or position of the US government. Ad- ditionally, we thank our reviewers for their detailed and useful feedback.
Shibo Hao, Tianyang Liu, Zhen Wang, and Zhiting Hu. 2023. Toolkengpt: Augmenting frozen lan- guage models with massive tools via tool emboddings. ArXiv, abs/2305.11554, 3
Ehud Karpas, Omri Abend, Yonatan Belinkov, Barak Lenz, Opher Lieher, Nir Ratner, Yoav Shoham, Hofit Bata, Yoav Levine, Kevin Leyton-Brown, Dor Muhl- gay, Noam Rozen, Erez Schwartz, Gal Shachaf, Shai Shalev-Shwartz. Amnon Shashua, and Moshe Tenenholtz. 2022. Mrkl systems: A modular, neuro- symbolic architecture that combines large language models, external knowledge sources and discrete rea- soning.
References
Afra Feyza Akyurek, Ekin Akyürek, Aman Madaan, A. Kalyan, Peter Clark, D. Wijaya, and Niket Tandon. 2023. RI4f: Generating natural language feedback with reinforcement learning for repairing model out- puts. ArXiv, 0bs/2305.08884. 1,3,4
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel HerrbertVVoss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler. Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Πlya Sutskever, and Dario Amodei, 2020. Language models are few-shot learners. ArXiv, abs/2005.14165. 6
Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. In Advances in Neural Information Processing Systems, volume 35, pages 22199 222133 Curran Associates, Inc. 3
Minghao Li, Feifan Song, Bowen Yu, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023. Api- bank: henchmark for tool-augmented llms. ArXiv, Abs/2304.082444 3
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao. Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang. Scan Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with self-feedback. 1, 2, 3, 4, 7,8
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. 2022. Program of thoughts prompting: Disentangling computation from reason- ing for numerical reasoning tasks. 3, 6
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reicchir Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word prob- lems. ArXiv, abs/2110. 14168.
Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang, and Graham Neubig. 2022. Language models of code are few-shot commonsense learners. ArXiv, abs/2210.07128.
OpenAl. 2023. Gpt-4 technical report. ArXiv,
Bhavana Dalvi, Peter Jansen, Oyvind Tafjord, Zhengnan Xie, Hamnah Smith, Leighanna Pipatanangkura, and Peter Clark. 2021. Explaining answers with entail- ment trees. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing. pages 7358-7370 Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics. 2,
abs/2303.08774. 4
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida. Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Pe- ter Welinder, Paul Francis Christiano, Jan Leike, and

---

Ryan J. Lowe. 2022. Training language models to follow instructions with human feedback. ArXiv, abs/2203.02155.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aur' 'eriennnn Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. ArXiv, abs/2302. 13971.
Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathami, Xinyi Wang, and William Yang Wang. 2023. Automatically correcting large language models: Sur- veying the landscape of diverse self-correttion strate- gies. 3
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le. Ed Huai hsin Chi, and Denny Zhou. 2022. Self- consistency improves chain of thought reasoning in language models. ArXiv, 0tns/203..111777
Bhargavi Paranjape, Scott Lundberg, Sameer Singh. Hannaneh Hajishirzi, Luke Zettlemoyer, and Marco Tulio Ribeiro. 2023. Art: Automatic multi- step reasoning and tool-use for large language mod- els.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Fid Chi, Quoe Le, and Denny Zhou. 2023. Chain-of-thougg prompting elic- its reasoning large language models. 6
Shishir G. Patil, Tianjun Zhang, Xin Wang. and Joseph E. Gonzalez. 2023. Gorilla: Large lan- guage model connected with massive apis. ArXiv, 15b/2300511333... 3
Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, and Hanna Hajishirzi. 2023. Fine-graine human feedback gives better rewards for language model training. 3,4
Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beat- riz Borges, Antoine Bosselut, Robert West, and Boi Faltings. 2023. Refiner: Reasoning feedback on in- termediate representations. ArXiv, abs/2304.01904 1,3, 4
Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan. 2023. Gpt4tools: Teaching large language model to use tools via self-instruction. ArXiv, abs/2305.18752. 3
Baolin Peng. Michel Galley, Pengeheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, and Jianfeng Gao. 2023. Check your facts and try again: Improving large language models with extemnl knowledge and automated feed- back.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang. Dale Schuurmans. Claire Cui, Olivier Bousquet, Quoc Lc, and Ed Chi. 2023. Least-to-most prompting enables complex rea- soning in large language models.
Timo Schick, Jane Dwivedi-YY Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. 3,3
Timo Schick, Jane Dwivedi- Yu, Zhengbao Jiang, Fabio Petroni, Patrick Lewis, Gautier Izacard, Qingfei You, Christoforos Nalmpantis, Edouard Grave, and Sebas- tian Riedel. 2022. Peer: A collaborative language model. ArXiv, abs/2208. 11663.
Yongliang Shen, Kaitao Song, Xu Tan, Dong Sheng Li. Weiming Lu, and Yue Ting Zhuang. 2023. Hugging- gpt: Solving ai tasks with chatgpt and its friends in hugging face. ArXiv, abs/2303. 17580.
Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Schärli, and Denny Zhou. 2023, Large language models be easily distracted by irrelevant context. 2.
Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal rein- forcement learning. 3,
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction- following Ilama model. https:// gi thub. com/ tatsu-lttuullt saabstaafaa tanford_aa _ad_apcaa.... 5

---

A Error Categories
times and then selecting the majority answer can improve the results. Thus Hallucination feedback aims to fix any hallucinated facts in the initial gen- eration. This module can be improved by using an external tool such as Knowledge Source instead of a LLM.
In this section we define all the error categories that are implemented in this work. Table 5 shows the enror categories introduced by Golovneva et al. (2022).
A.1 Programming Syntax Feedback
B Implementation Parameters
Programming syntax feedback module is imple- mented as Python Interpreter. This module aims to fix any syntax errors in the generated code. This particular module benefits from Eager- Refinement approach.
Table provides parameters used for Self-Refine and Table 7 provides parameters used for MAF.
C Few-Shot Prompt Examples
We add samples for all the prompts used in work. Complete prompts
be found in source code. Note that for all feedback prompt ex- amples, there at least one example with no errors. in which case the feedback will state that there are ther iterations. Variable Naming Feedback is an- other module which benefits from an cager-refine approach.
A.2 Variable Naming Feedback
Good variable names a code can improve the readability of the code and potentially can improve model's own understanding of the program in fur-
no errors and the reasoning is correct. This helps decrease the likelihood that the model identifies an error in a solution that is actually correct.
A.3 Redundancy Feedback
Redundant information is any information included in the reasoning that doesn't it help answer the ques- tion. This additional information may distract the model from correctly answering and should thus be removed.
A.4 Commonsense Feedback
Commonsense reasoning errors are errors about any relation or knowledge that is should be known from general world such as "all ducks are birds".
A.5 Missing Steps
Missing steps errors are any gaps reasoning or missing information that prevent the reasoning chain from being correct. This also identifies the model saying that the question is unanswerable as a missing step error because that means additional reasoning steps are needed to answer the question from the passage.
A.6 Factuality Feedback
Factuality errors are cases where the answer rea- soning states infactual information. This could be information that contradicts information given in the passage hallucinated.
A.7 Hallucination Feedback
LLMs are prone to hallucination, however, it has been shown that sampling from a LLM multiple

---

Error Type
Definition
Programming Syntax
Syntax errors in code
Arithmetic
Error in math calculations
Grammar
Faulty, unconventional, or controversial grammar usage
Coherency
Steps contradict cach other do follow cohesive story Variable names in program I give full information are wrong Step paraphrases information already mentioned in previous reasoning steps Information is provided the problem statement and irrelevant or wrong Model lacks relations should known from the general world (e.g.. dozen =12)
Variable Naming
Repetition
Hallucination
Commonsense
Factuality
Information about an object (i.e. quantity, characteristics) or named entity doesn't match with the input context
Missing Step
The content the generated reasoning is incomplete and lacks the required infor-
mation to produce the correct answer
Redundancy
Explanation contains redundant information, which even though might be factual, not required to answer the question
Table 5: Error types as defined in Golovneva et al. (2022). Each error category is defined for single step in the reasoning chain.
Dataset
Base LM
Feedback
Refiner
Dataset
Base LM
Feedback
Refiner
EB
300
600
600
EB
300
600
600
GSM-IC
300
600
600
GSM-IC
300
600
600
GSM8K
300
600
600
GSM8K
300
600
600
DROP
450
600
800
DROP
450
600
800
Table 6: Value of Maximum Number of Tokens
Table 7: Value of Maximum Number of Tokens parameter for MAF on various datasets.
parameter for Self-Refin on various datasets.

---

lollipops. gave Denny Lollipops. lollipops. many lollipops Denny?
solution using Python:
def :)lution():
"""Jason had lollipops, Be gave Denny some lollipops. Now Jason has 12 lollipops. How many lollipope give Denny?**
jason_ holllipops_initee
20
jason hollipops_ after
denny 1o11ipops jason_ lollipops initial jason lollipops result jollpppo
returo zesult
There are 15 trees in the Grove workers will plant trees in the today. After they aze done, there will be 21 trees. How many trees did the grove workers plant today? using Python:
def colution():
grove today. they done, there will be 21 trees. How many trees did the grove workers plant today?""r
"""There
grove.
trees initial
trees_added trees_affers initian
result trees added
result
Q: Shawn has five toys. For Christmas, be got two toye each from his mom and ded. Bow many toys does he have solution using Python:
solation():
"u"Shawn has five toys. For Christmas, he got two toys each from his mon and dad. Now many toys does he
toys_iniss initial
non toys = 2 toys total received
dad, toys
total toys toys initial + total received totel_toys
return result
Q: There were computers the server roon, Five more computers were installed each day, from monday to thursday. many computers solution using Python:
def solution():
"""There ware nine computers in the server room. Five more computers ware installed each day, from monday thuraday. many computers computers initial
room?"er
compaters_ per day days par_day day days computere edded
days
monday
computers aandttere_addddaaaec addad computers Bectars_per_dae computers total computers result conputers total return result
Figure 4: Initialization prompt for Mathematical Reasoning

---

colution():
"""Kelly grocesy shopping superarket and itans in her cast. Bec packa of bacon cost $10 in total and she has packets of chicken which each coat pack
she enough in budget for
priced each, packs apples, each priced at half the price of pack of atrawberries. IE Relly's budget is 865 then how much in dollars, does sbe have left in her budget?"
packs
budget
bacon_packs =
bacon total cost
chicken_ Bacaan_p_aa___
chicken_coas = sacon_total_coos
atrawberty_pacce packss_ppps____
strawberzy_cost
ueple_packs==
apple_cost strawberry_comm
total cost bacon cast chicken cost strawberry_cod cost apple noney_ieft left budget total cost
cost cople_cost
money_left.
return result
senantically complete block reasoning errors may include errore in the mathenstical calculations, errors in the order of the steps, azrors the assumptions made. the assumptions you clearly. Ignoze
check any logical reasoning
types errors.
Let us go through the code step-by-step budget
looks good
parts
bacon packs Licon_totallcomm 10 looks good
Let's check other parts chicken packs
chicken_cooe cost = bacon bacon_total_cee tt__ttttlacstttt
wrong! according to context, We should bacon_cost place of bacon_total. caap_tttal_cooe calculate the chicken pack correctly. Let's fix it.
packet
cost packet bacon.
Let's check other parts strawberry_Pan peccberrryppcckssse strawberry_coke
looks good
parts
apple packs 7
apple cost strawberry_com
looks good
parts
total
cost chicken straxberry apple cost
noney_left budgat total cost
result money_left
return result
wrong! we want to calculate the total cost of buying all the items so we should use the total cost of each insteed
Let's fix
K44
Figure 5: Commonsense Feedhack for Mathematical Reasoning

---

colution():
"""Kelly grocesy shopping superarket and itans in her cast. Bec packa of bacon cost $10 in total and she has packets of chicken which each coat pack
she enough in budget for
priced each, packs apples, each priced at half the price of pack of atrawberries. IE Relly's budget is 865 then how much in dollars, does ebe have left in her budget?"
packs
budget
bacon_packs =
bacon total cost
chicken_ Baciann__ak___
chicken_cost = bacon_cost
strawbarty_peace packssppapa____
etrawberzy_cost
uople_packs==
apple_cost strawberry_comm /
total cost bacon cast chicken cost strawberry_cont cost apple cople_cost cost noney_ieft left budget total cost money_
return result
semantically complete then. Ignore all the other types of errore.
any missing steps and auggest way
Let go through the stap-by-stap
budget 65
looks good
Let's cback other parts bacon_packt
bacon total cost 10
looke good
Let's check other parts chicken chicken_packs packs chicken,
wrong! bacon_coos cost is miseing. let's add it. wzong!
cost
calculate remaining budget.
Let's check other parts
Saauberryyppckks strawborry packs strawberry cost
wrong! we need the total cost of strawberries to calculate remaining budgat. Let's add it.
Let's check other parts apple_packs
apple_cost strawberry
wrong! vo need the total cost of apples to calculate remaining budget. Let's add it.
Let's check other total cost bacon cost chicken_cort cost strawberry_coat apple_cost cost
money left budgat total cost result money_left
return result good
## END ###
Figure 6: Missing Step Feedback for Mathematical Reasoning

---

c)lution():
large and small lazge pizza has slices and snall pizza has alices. Albert also has piecea of cake. If he eata it all, day?"er
buying each """Albert is wondering much he
many pieces
large_ pizzas
=
large pizza_ _azge_pizza small mmall_iizzaolloe
cake pieces
slices slarge_pizzas large_ pizza slicea) (small pizzas pizza slices) pieces
return total_elices
Cbeck each semantically complete block of code for any redundancy and suggest fixes. Redundancy errors are stepa that contain redundant information, which even though night be factual, is not zequized to anower question. Ignore
types exrors.
Let's go through the code stap-by-stap large_pizzas
small pizzas
looks good
Let's cbeck other parts large pizza
small
looks good
Let's check other parts
cake pleces-
wrong! while
accurate,
Let's cbeck other parts total_slices (large parpe_iizzs___ee large sarge_iizza___eee slices) (small_ pizzas snall pizza elices) pieces wzong! we should not add cake pieces because we want to calculate the number of pizza pieces he can eat. Let's fix it
##* END 8*#
def ()lution():
"u"Twenty dozen cups cost $1200 less than the total cost of halé dozen plates sold at 56000 each. Two party cost $400 less cost
plates.
logical reasoning plates
plato cost 6000
plate plate_total tat__tttll_oooo cost: = plate_cost plates
party_hate
party_bats__
barty_bats_tottl__cors total_cost = plate tatal_ttaoooooeeee joost 400
party hat_cost cost party hats batas cost party_hat hats cups
cup_ hip_total_ccom plate_total cost 1200 aup_cost cup total cups
result
return result
Check each semantically complete block of code for redundancy arrora and suggest fixes. Redundancy arrors are steps that contain redundant information, which even though night be factual, is not required to question. Ignore
types
Let's go through the code step-by-stap
plates
plate_ cost 6000
looks good
Figure 7: Redundancy Feedback for Mathematical Reasoning

---

def solution():
"""Twenty cupa
plates
Calculate the total coat of buying each ne* plates
plate 12 20
cup_cost cost plats_ccat_se cup_cost
return result
Check each semantically complete block of code and identify the variables that naned correctly may cause confusion and fix the issues. State the assumptiona you made when renaming the variables clearly. type exrore.
Ignore
Let us go through each variable atap-by-stap
plates
plate cost 5000
looks good
Let's cbeck other parts cupa
eup_cost cost plate_ces rename the variable cup_cost to cups_total_cost because we are given information about total cost of cupe. cupa
totttootlooooooo cost
parts
result cup_cost
return result
looke good
Okay, here is the rewrite:
def solution():
"""Twenty cups
dozen plates
Calculate the total of buying each nee plates
plate cost cups 12 20
cups total cost plate_cost cup_cost
return result
solution():
"""Carly had cards,
the cards had the
of had the letter
5/8 of the rest had the letter C on then, and the others had the letter D. Hlow many of the carde had the
parde_initial cards initial 80
cards_ cards initial 2 /
cards_a
cards b cards left
cards carda_left cards left cards_b
cards d = hards_lafft result cards result
Check each semantically complete block code identify the aze not naned correctly confusion and fix the issues. the assumptions you renaming the variables clearly. Ignore all the other type of ecrors.
Figure 8: Variable Naming Feedback for Mathematical Reasoning

---

def solution():
"""Twenty cupa
plates
Calculate the total coat of buying each ne* plates 0.5
plate 12 20
cupa total cost plate_coat
cup_
return result
Feedback: Let ua go through the code atap-by-atap
plates
plate cost 5000
wrong! The nunber of plates is wrong, The question nantions half dozen plates. dozen is 12. Bo half
Let's cbeck other parts cupa
oup oup_total_coct plate_cost
wzong! questions
plate and subtract 1200 Let's fix plates.
cupa than
plate total
Missing Step Feadback:
Let through the code step-by-step
plates 0.5
plate_cost
wzong! we need to calculate the total cost of plates to calculate the cost of each Let's add it.
parts
result cost
return result
wrong!
calculate
cup question. add
feedback the original code,
incorporate
feedback.
change anything unless it mentioned the feedback.
Okay,
def solution():
"""Tuenty cups
half plates
Calculate the total cast of buying each cup. ne*
logical reasoning plates
plate_ cost 6000
missing step
total
plates
cupe 12 20
logical zeasoning
total_otaac cost plate total 1200 missing stop
oup cost cup total cups result cup_cost
return result
Figure 9: Our Iterative Refinement prompt for Mathematical Reasoning

---

def solution():
"""Twenty cupa
plates
Calculate the total coat of buying each ne* plates
plate cups 12 20
eup_cost cost plate_cost cost
cup_cop cos_cost return result
code above because of lack of underetanding question. What is the find the arrec, through semantically complete blocks of the code, and chack if everything looks good.
there
Let go through the arzor and check step-by-step
plates
plate_con looks good
Let's
other
cups 12 20
cop_cost plate cost
plate. cup $1200 the total cost of half dozen plates sold at $6000 each. So we need to calculate the cost of first (total of dozen plates at $6000 $1200)
wxong!
of cup
Okay! Here is the zewrite:
def s)lution():
"""Twenty dozen cups cost $1200 less than the total cost of halé dozen plates sold at $6000 each. buying cup
Calculate
plates =
plate_cost 6000
cupa
total_oup cup_cost (plate (plate_cost cost plates) 1200 cost_per_ cup total cup__csstt cost cupa return cost_per_cup
daf solution():
"""Carly
carde
remsining
5/8 of the rest had the letter C on then, and the others had the letter D. How many of the cards had the letter D on them?'on
cards_ initial
cards cards initial 2 /
cards left cards initial carda_ left
cards
cards_left left cards left cards_b cards cards
cards d cards_leff result cards_d return
There is an error in the code above because of lack of understanding of the question. What is the error? To find arror, go through senantically complete of code, everything good.
Let us go through the erroc check step
cards initial
looks good
Figure 10: Iterative Refinement prompt from Self-Refine for Mathematical Reasoning

---

million and allowed for hostages beld king, II, million and allowed for hostages beld lieu of John. The hostages included of his several princes and nobles, four inhabitanta of Pacia, and bwo citizens fron each of the nineteen principal hostages ranson. In 1362 John's Louis of Anjou, hostage in English-held Calais, ascaged captivity. še, with his stand-in in hostage gone, John felt honor-bound to return to captivity in England. The French crow had been at odds with Navarze since political weakness of the Dauphin to to seize power. Although there was formal treaty, Edward III supported the Navarrese noves, particularly there was prospect that night gain control over consequence.
Fassage:
captive England. Treaty Brétigmy
at
try
pay
the captivity
Edward deliberately slowed the peace negotiations. In 1364, John II died in London, while still in honourable captivity. Charles V succeeded hin as king of France. May one after dauphin's accession and three days before coronation cuffered crushing defeat at the Battle of Cocherel.
provingea
Charles the
Question: How many groupe
hostage
Anaver:
The Treaty of Brétigny set John II's ransom at million crowns and allowed for hestages to be held in sons, several princes two citizens from each of the nineteen principal towns of France. Thus, there ware total of twenty-four four groupe held hostage in lieu of John II. final_ancen anewer:
bostages included
inhabitants Paris,
Check each semantically complete block of the reasoning to check for any commonsense errore. Commonsense reasoning arrors ezrors about any zelation or knowledge is sbould "all ducks birds". State assumptions you clearly. Ignore other typas
general
Let's go by The Treaty of Brétigny John II's
1ieu of him.
good.
Let's cbeck the next line hostages citisens from each of the nineteen principal towns of France. looke good
sons, princes nobles,
Paris,
let's cbeck the next line
Thus, thare counte each beld hostage group, which doesn't t make since group wrong! This
tventy-four groups hostage
generally isn't single person. The different groups that are held hostage in lieu of John II are his sons, princes nobles, Thus, there four held hostage in lieu John II.
Paris, citirens
of nineteon principal
Passage: As the census 2000, there were 218,590 people, 79,667 householde, and 60, families residing county. population density
people per square mile (192/km\u00622.
housing units at average density of 189 per square mile (73/km)u00b2). The racial makeup of the county was 86. 774 Race (United States Census), 9.278 Race (United States Census), 0.234 Race (United States Censue). (United states Census), and 4,478 from two or more races. 1,918 of the population were Race (United States Census) or Race (United people, 9.21
1.52% (United
Census), 0.06%
(United
Consus), 0.694
Consus) of any race. 22.5% English, 8.14 \"Anerican)" and 6. Polish ancestry.
people, 13.18 paople,
Question: How many more housing
axe
families?
The passage states that there were 79,667 households and 60, families residing in the county. families eguals 280. Thus, families. final answer: 19,280
Anzwer:
housing units than
check each semantically complete block of the reasoning cbeck for any reasoning errors are errors about any relation or knowledge that is sbould be known from general world such as ducks are birds", State the assumptions you clearly. Ignore other types errors.
Commonsense
Figure 1I: Commonsense Feedback for Question Answering

---

king, II, 3-million-croms and allowed for hostages beld lieu of John. The hostages included of his several princes and nobles, four inhabitanta of Pacia, and bwo citizens fron each of the nineteen principal hostages ranson. In 1362 John's Louis of Anjou, hostage in English-held Calais, ascaged captivity. še, with his stand-in in hostage gone, John felt honor-bound to return to captivity in England. The French crow had been at odds with Navarze since political weakness of the Dauphin to to seize power. Although there was formal treaty, Edward III supported the Navarrese noves, particularly there was prospect that night gain control over consequence.
Fassage:
captive England. Treaty Brétigmy
at
try
pay
1363
the captivity
London
nind, Edward deliberately slowed the peace negotiations. In 1364, John II died in London, while still in honourable captivity. Charles V succeeded hin as king of France. May one suffered crushing defeat at the Battle of Cocherel.
provingea
dauphin's accession and thres days before coronation
Charles the
Question: many years Mavarrace
captivity John
Anawer: The French crown had been at odds with Havarre since 1354. Thus, Navarrase used the captivity of John years
Simal_anewers
check each semantically complete block of reasoning for any infactual information and renove or fix it. Any information present
considered infactual. Ignoze all other types
Let's go by French had odds with looks good
1354.
Let's check the next line
Thus, Havarrase used the captivity of John II 10 years later in 1364. used captivity
passage, 1363 states that the Navarrese used the captivity of John II in 1364 whan they actually used it in 1363. Let's fix it
wrong!
line falsely
## END ###
Tampa, Florida The Lions scored firet in the first quarter with a 23-yard field goal by Jason Hanson. The Buccaneers tied it took
Fassage:
Tampa Bay
up with 38-yard
goal by Connor Barth, Matthew Stafford and ran in 28 yards. The Lions responded with 26-yard field goal. In the second Detroit took the lead with 36-yard touchdown catch by Calvin Johnson, and later added more points when Tony 31-yard
when Agib Talib intercapted pass
Schefflor eaught 1l-yard TO pass. Tampa Bay respondad only scoring touchdown. First, Detroit's Calvin Johnson
goal just
second half relatively quiet, with each caught a 1-yard pass in the third quartor. The game 's final points cane when Mike Williams of Tampa Bay caught opener the
5-yard paes.
regular
2007
Question: Nlw many field goals did the Lions score?
Answer:
The first was 23-yard field goal by Jason Hanson in the first quarter. The second was 28-yard field goal response to the Buccaneers' interoeption. And the third was 36-yard field goal the second they scored three field goals. final,
somantically complete information not in the context considered infactual. Ignore all other types of errors.
zeasoning any
zenove Any
go by line
The first 23-yard field Jaon Hanson in the first quartar.
looks good
Let's cbeck the next line
second was 28-yard field goal reaponse to Buccaneers' interception.
looks good
Let'a check next
And the third 36-yard field the second quarter. wrong! This anaver count the 36-yard touchdown catch by Calvin Johnson in second quarter as field goal. goal. Thus,
wrong because
in answer.
Figure 12: Factuality Feedback for Question Answering

---

Tampa, Florida Lions scored first the first quarter with 23-yard field by Hanson. The up with a 38-yard field goal by Connor Barth, than took the lead when Agib 7alib interceptad pass from 28-yard
Fassage:
season,
Tampa Bay Buccaneers. tied it
Katthew Stafford Detroit took the lead with 36-yard touchdown catch Calvin Johnson, and later added more points when Tony Scheffler caught an 11-yard TO paes. Tampa Bay responded with a 31-yard field goal just before halftime. The only scoring
ran
yards.
responded
goal. In second quarter,
relatively quiet, caught 1-yard pacs in the third quarter. The game's final points came when Mike Williams of Tanpa Bay caught 5-yard pass. The Lions won their regular season opener for the
touchdown.
Detzoit's
Johnson
2007.
Question: Nho caught the chortest touchdown pass?
The shortest touchdow
caught by Scheffler. caught 11-yard final_ann
Tony Scheffler
Check each semantically complete block of reasoning for any missing staps and auggest the correct add then. Ignore all the other types of ezrors.
Lat's check the answer step-by-stap The sbortest touchdown pass was caught by Tony Scheffler. Be caught an 11-yard zo pass. final_ answer: Tony Scheffler
wrong! This answer missed the 1-yard paes caught by Calvin Johnson and 5-yard pass caught by Mike Williams. So ansver should Let's add information to anaver.
## END H+
The French John had been held captive in England. The Treaty of BriuD0eStigny his zansom at muutaonilleenwuaaaeeeee and allowed for hostages to be held in 1ieu of John. The hostages sons,
princes nobles, the ninateen principal towns of France. While these hostages were held, John returned France to try and raise funds to pay the ransom. In 1362 John's son Louis of Anjou, hostage Englich-held Calais, escaped captivity.
captivity England. French crown had been at odds with Navarre since 1354, and in 1363 the Navarrese used the captivity of John II London and the political weakness the Dauphin to try to seize power. Although there was no formal treaty,
hostage gone,
supported the control over the northern and western provinces ae consequence. With this in mind, Edward deliberately 1364, John
particularly
prospect
might gain
slowed peace begotiations. succeeded him king of France. On 7 1364, month after the dauphin's accession and three days before his coronation as Charles V, the Navacrese suffered crushing defeat at the Battle of Cocherel.
London,
still honourable captivity.
Question: At least how
held hostage place of John II?
hostages held in lieu of John. The hostages included two of his sons, several princes and nables, four inhabitants of Paris, and two citisens from each the nineteen principal towns of France. Thus least 2 2 people
Answer: Treaty of BelubdeRtigny
hostage place
final_anasn answer;
Check each semantically complete block reasoning for any missing steps and suggest the correct way add thon. Ignore
other types of
Let's
answer step-by-step
The Treaty of Briudde9tigny set John It's ransom at million crowns and allowed for hostages to be held in 1ieu of John.
good
Let's check other parts
hostages included
sons, princes
inhabi and
citisens from each of the nineteen principal towps of France.
looks good
Let's cbeck other parts
people were hald hostage place final, anaver: this missed counting the inhabitants Paris. the should be changed. the will be 40 44, Let a add this.
Thua at least
Figure 13: Missing Step Feedback for Question Answering

---

captive England. Treaty Brétigmy million and allowed for hostages beld lieu of John. The hostages included of his several princes and nobles, four inhabitanta of Pacis, and bwo citizens fron each of the nineteen principal hostages
Fassage: French king, II,
ranson at
pay the ranson. In 1362 John's Louis of Anjou, hostage in English-held Calais, ascaged captivity. Se, with his stand-in in hostage gone, John felt honor-bound to return to captivity in England. The French crow had been at odds with Navarze since political weakness of the Dauphin to to seize power. Although there was formal treaty, Edward III supported the Navarrese noves, particularly there was prospect that night gain control over consequence.
try
Navarrese
the captivity
London
this nind, Edward deliberately slowed the peace negotiations. In 1364, John II died in London, while still in honourable captivity. Charles V succeeded hin as king of France. May 1364, one after dauphin's accession and three days before coronation Charles the Navarsese cuffered crushing defeat at the Battle of Cocherel.
provingea
Question: How many groupe
hostage
John
Anaver:
The Treaty of Brétigny set John II's ransom at million crowns and allowed for hestages to be held in sons, several princes nobles, inhabitants Paris,
bostages included two citizens from each of the nineteen principal towns of France. Thus, there ware total of twenty-four four groupe held hostage in lieu of John II. final anewer:
Check each semantically complete block of reasoning for any redundancy errore and suggest fixes. Redundancy ezrors are stepa that contain zedundant information, which even though might factual, is requized quastion. Ignore other types
Let's go by
million and allowed for bostages be beld crowns to the question of how many groups ware hold hostage in 1ieu of him. Let's renove that information.
The Treaty of Brétigny John II's
1ieu of him. wrong!
Let's
The hostages included two of his sons, several princes and nobles, four inhabitants of Paris, and two citizens from each of the nineteen principal towne of France. wrong!
number hostages each group held hostage in lieu of John II. let's renove that information.
the question nany groups
Let's check the next line Thus, there were total of tventy-four groupe held hostage in lieu of John 1I.
looks good
## END ###
Fassago: Trying to two-game skid, the Bills flom to Gillette Stadium for Week 3 divisional fight with the New England Patriots. In the first quarter, qB J. P. Losman was imediately injured on the first offensive play
but ended up After New England took the lead with kicker Stepben Gostkowski's 24-yard field goal, rookie Qs Trent Edwards played the rest the gane for Buffalo. The Bills would get their only score of game RB Marshawn point put Bills
gane.
finish
game.
Lynch got 8-yard the second quarter, the Patriots were able to open up their running game when Bills rookie standout Paul Foslusany lost to Patriots dominated. QB Ton Brady's 8-yard To passing Benjanin Watson and 3-yard TD pass to WR Randy Noss made it 17-7 at the half. In the third quarter, New England continued its conquest with Brady's 4-yard TO pass to with Brady and Mosa hooking with each other again 45-yard TO
and
Lindell
suzprisingly 7-3.
in
This
open,
rest the game,
Gaffney
Sanmy Morris' 4-yard run.
quarter, the Fatriots ended the
Question: many points
by
Answer: the first half, Bills acored 7 points with Marshawn Lynch 8-yard run and Lindell point. Fatriots scored 17 points with Stephen Gostkowski 24-yard field goal, Brady 8-yard pass to TE Benjamin Natson, and 3-yard TD pass to HR Randy Moss. Thus, total of 24 points were scored in half, final ansver:
Figure 14: Redundancy Feedhack for Question Answering

---

captive England. Treaty Brétigmy million and allowed for hostages beld lieu of John. The hostages included of his several princes and nobles, four inhabitanta of Pacis, and bwo citizens fron each of the nineteen principal hostages
Fassage: French king, II,
ranson at
pay the ranson. In 1362 John's Louis of Anjou, hostage in English-held Calais, ascaged captivity. Se, with his stand-in in hostage gone, John felt honor-bound to return to captivity in England. The French crow had been at odds with Navarze since political weakness of the Dauphin to to seize power. Although there was formal treaty, Edward III supported the Navarrese noves, particularly there was prospect that night gain control over consequence.
try
Navarrese
the captivity
London
nind, Edward deliberately slowed the peace negotiations. In 1364, John II died in London, while still in honourable captivity. Charles V succeeded hin as king of France. May 1364, one after dauphin's accession and three days before coronation cuffered crushing defeat at the Battle of Cocherel.
provingea
Charles the
Question: How many groupe
hostage
John
Anaver:
of The Treaty of Brétigny set John II's ransom at million crowns and allowed for hestages to be held in sons, several princes
bostages included two citizens from each of the nineteen principal towns of France. Thus, there ware total of twenty-four Four groupe held hostage in lieu of John II. final anewer:
inhabitants Paris,
There is error in the answer above. what is the error? To find the erzor through the answer line by everything looks good.
line and
Let's go line by line Treaty of Brétigny John II's ranson million crowns allowed for hostages to lieu of hin.
# wrong! this information is not neaded to ansver the question. Let's remove it.
Let's check the next line
The hostages included two his sone, several princes and nobles, four inhsbitante of Paris, and two principal
# looks good
Let's
Thus, there were total of tventy-four groupe held hostage in lieu of John II.
wrong! This anawer counts each person bold hostage group, which doesn't it make sense, generally ien't single
group
The different princes and nobles, inhabitants of Paris, and citizens from each of the nineteen principal towns of France. there four groups hostage
that held hostage in lieu of John his
## END ##
Fassage: To start the season, the Lions traveled south to Tampa, Florida to take the Tampa Bay Buccaneers. The Lions scored first in the first quarter with a 23-yard field goal by Jason Hanson, The Buccaneers tied it took lead when Agib Matthew Stafford and ran it in 28 yards. The Lions zesponded with 28-yard field goal. In the second quarter, Detroit took the lead with 36-yard touchdown eatch by Calvin Johnson, and later added more points when Tony 31-yard field goal just before halftime. The
up 38-yard field goal by
intercepted pass from
Scbeffler caught 11-yard тO pass. Tampa Bay responded second half was relatively quiet, with each tean only scoring one touchdown. First, Detroit's Calvin Johnson Killiams Tampa Bay caught for the first time since 2007
caught 1-yard pass the third quartor. 5-yard The Lions their regular season
game's 's points
Question: How many
goals
Lions score?
Anzwer:
The first was a 23-yard field goal by Jason Hanson in the first quarter. The second was a 28-yard interception. And third 36-yard goal the second
goal response the quarter. 3o they scored three field goals. final, answer:
What the line and check that everything looks good.
There
in
find the through the
Let's line line The first was 23-yard field goal by Jaon Banson in the first quarter. good
Figure 15: Self-Refine style Iterative Refinement prompt for Question Answering

---

season, Lions Lions scored first the first quarter with 23-yard field by Hanson. The up with a 38-yard field goal by Connor Barth, then took the lead when Agib 7alib interceptad pass from 28-yard
Fassage:
Tampa, Florida
Tampa Bay Buccaneers. tied it
Katthew Stafford Detroit took the lead with 36-yard touchdown catch Calvin Johnson, and later added more points when Tony Scheffler caught an 11-yard TO paes. Tampa Bay responded with a 31-yard field goal just before halftime. The only scoring
ran
yards.
responded
goal. In
quarter,
relatively quiet, caught 1-yard pass in the third quarter. The game's final points came when Mike Williams of Tanpa Bay caught 5-yard pass. The Lions won their regular season opener for the first time 2007.
touchdown.
Detzoit's
Johnson
Question: How many points did the buccaneers need to tie in the first?
Answer:
The Buccaneers tied the in firet with 38-yard field goal Barth". field geal is worth points in American football, Thus, the bucaneers needed points to tie the scoze in the quarter. final_ answer:
## END ###
Passage: Pack also received Oscar nominations for his roles in The Kays of the Kingdon (filn) (1944), The Yearling (film) (1946), Gentlemane Agreement (1947) and Twelve oclock High (1949). Other notable films in Holiday (1953), Moby Dick (1956 film) (1956, and its Moby Dick (1998 miniseries)), , The Big Country (1958), The Bravados (1958), Pork Chop (1959), Guna Navazone (film) (1961), Cape Fear (1962 film) (1962, West
which he appeared include Spellbound (1945 film) (1945).
Gunfighter (1950),
Cape Fear (film) (1962), Oman (1976) and Boys Brazil (film) (1978)..
(1991
Question:
did zeceive an Oscar nomination first: Keys the Kingdom or Yearling?
Answer: Pack zeceived an Oscar nomination for The Kays of the Ringdon in 1944 and the Yearling 1946. 1944 1946, Kays of the Kingdon"
an
Keys Kingdom
sinal_answer:
Passage:
The Spanieh were able to complete their escape to Tlaxcala, there, they were given assistance, since wounded, only
Maxixcatrin, Xicotencati Eldar Chichinecatecle told Cortés's e men: "Consider yourselves at home. Rest...do not think it small thing that you have escaped with your lives
strong city...if sе thought you reinforcements when the Panuco River settlement abandoned, and supply ships
before, consider you much
":306-07 Cortés
arrived from Cuba and Spain, Cortés also had built 13 brigantines then had then mounted with cannons, turning Lake Texcoco into stratogic body water assault Tenochtitlan. Xiostencatl alliance with the Mexicans, but opposed.:309e :309-11 Cortés Navaror's a mon, on a ship to Spain, and Alonso de Ávila on a ship to Santo Domingo to represent his case in the pacify country, end to the and robbery that the Mexicans practised." Finally, Xicotencatl the Elder. baptized as Don Lorenzo Vargas, agreed to support Cortés's expedition against Texcoco. According to Bernal Diaz, he sent narched day
Youngor, howevor, sought Diego de Ordaz, and the remnants
Royal
Cortés
indigenous
Spaniards put
thousand Christmas :309, 311-12
Chichinecatecle
Question: what
10,000
march under Chichinecatecle?
Chichinecatacle "the day 1520". The day after christmas is 26 December. Thus, 10,000 warriors marched under Chichimecatecle 26 December 1520. final_ answer: 26-Decamber-1520
Answer: According to Diaz, 10,000 warriors
## END 144
with either passage. Format your answer with "final answer: and then your final answer. If your answer is number, give just the numeric answer. your ansver date, then give the day in numbers, the month by its name, and year in numbers. your span, surround it quotes.
following passage
questions
span
Figure 16: Initialization prompt for Question Answering


Reviews:
[{"review_id": "VvfvoxE2vY", "reviewer": "Reviewer_zRzu", "summary": "", "questions": "A. The authors took the categories from Golovneva et al. (2022) and made a new classification for defining feedback types. I wonder if the different types of feedback have different effects on the performance improvement. And, what additional kinds of feedback can be added to improve the performance of different tasks using LLMs?\nB. What other NLP tasks can be benefited from MAF? And why do you think so?\nC. Could you explain more about over-refinement problem and in what conditions the problem occurs and how your MAF can mitigate the problem?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comprehensive review and for expressing interest in our work. We’ve considered your comments carefully and provide the following responses:\n\n**1A. The authors took the categories from Golovneva et al. (2022) and made a new classification for defining feedback types. I wonder if the different types of feedback have different effects on performance improvement. And, what additional kinds of feedback can be added to improve the performance of different tasks using LLMs?**\n\n> Different types of feedback do result in different performance improvements. One of the main strengths of our framework is that it leverages specific feedback in several error categories rather than a single general feedback such as in Self-Refine (Madaan et al. 2023). This results in performance improvement compared to Self-Refine as shown in Table 2. \nFurthermore, different feedback modules have varying effects on the performance. We have conducted an ablation study in which we removed feedback modules one by one from the feedback framework and tested the performance on a subset of the GSM-IC dataset, with results shown in Table 3. It can be seen that for this dataset, redundancy has the largest impact on performance while variable naming and missing step have a much smaller impact.\n>\n> For future work, there are many possible feedback modules that can be added to improve LLM performance. Notably, in the future, we would like to add external tools such as a calculator or entailment checker that can provide more concrete feedback on generated answers. Furthermore, the benchmarks and feedback modules in this paper focus on logical reasoning and question-answering. For generative tasks, feedback modules that focus on conciseness, grammar, style, informativeness, etc. may help generate higher-quality texts.\n\n**1B. What other NLP tasks can be benefited from MAF? And why do you think so?**\n\n> We believe that MAF is particularly applicable to problems that may be solved by generating reasoning chains. This is because MAF is able to examine individual reasoning steps and determine if they fall into any of the error categories as well as suggest specific fixes for it. \n>\n> One area that we would like to explore using MAF in scenarios where an LLM generates new content from a prompt, such as dialogue generation or summarization. As mentioned above, these tasks could benefit from feedback and refinement specifically targeted towards categories such as conciseness, grammar, style, informativeness, etc. depending on what the user prioritizes in their response.\n\n**1C. Could you explain more about over-refinement problem and in what conditions the problem occurs and how your MAF can mitigate the problem?**\n\n> Over-refinement occurs when the model reaches a correct solution through iterative refinement but cannot successfully detect to stop correcting the solution. In this case, feedback with a spurious error is generated and results in an incorrectly refined solution.\n>\n> To clarify, MAF does not mitigate the problem and we notice that our framework is also susceptible to the Over-Refinement problem. In fact, the interplay of different feedback modules exacerbates the problem as we increase the number of refinement steps. For this reason, we report all MAF results under the standard setting after 2 iterations, as compared to after 4 iterations for Self-Refine. The number of iterations is treated as a hyperparameter. \nHowever, the Self-Refine framework also suffers from the problem on the DROP dataset, which results in a drop in performance after using Self-Refine.\n>\n> We wanted to highlight the problem of Over-Refinement to encourage future works to study the Over-Refinement problem in iterative frameworks more closely.\n\nAdditionally, we would like to acknowledge that we have considered all items in the ACL Responsible NLP checklist. We hope this response adequately addresses your queries. Thank you once more for your constructive feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This long paper can contribute to the improvement of reasoning using LLMs. MAF is a novel iterative refinement framework that decouples the feedback modules and takes advantage of error-specific tools to generate feedback. Its performance seems to be not only useful to mathematical reasoning, logical reasoning, and question answering that the authors examined, but also useful to other tasks in NLP. Additionally, the authors stated that overstatement is a problem. This problem can be widely discussed in the NLP community.", "reasons_to_reject": "The authors may confirm if they considered all items in the responsible NLP checklist provided by ACL.", "questions_to_authors": "A. The authors took the categories from Golovneva et al. (2022) and made a new classification for defining feedback types. I wonder if the different types of feedback have different effects on the performance improvement. And, what additional kinds of feedback can be added to improve the performance of different tasks using LLMs?\nB. What other NLP tasks can be benefited from MAF? And why do you think so?\nC. Could you explain more about over-refinement problem and in what conditions the problem occurs and how your MAF can mitigate the problem?", "paper_topic_and_main_contributions": "This long paper entitled “MAF: Multi-Aspect Feedback for Improving Reasoning in Large Language Models” can contribute to NLP engineering. In this paper, authors consider not only self-refine feedback but also redundancy, repetition, and missing step feedback iteratively for self-improvement in mathematical reasoning, logical reasoning, and question answering. They found that improvements up to about 20% were achieved through their approach, MAF. Additionally, MAF seems to be useful and promising to any other tasks in NLP.", "excitement": 5, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GILUCuStME", "reviewer": "Reviewer_Bg9c", "summary": "", "questions": "1. The \"redundancy\" module is used in all three evaluated tasks. Does this module share the same prompt across different types of tasks?\n2. How do you implement \"code interpreter\" module?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comprehensive review and insightful questions! We carefully address the questions and concerns below.\n\n**2A. The proposed MAF pipeline only supports generating feedback for pre-defined error types, and it might miss other error types that are not covered.**\n> We would like to emphasize that we have considered most of the error categories found in reasoning problems. However, for applying MAF to non-reasoning (Style Transfer, Summarization etc.) problems, we agree with the reviewer that our framework supports generating feedback only for pre-defined error types and it can be limiting in those cases. \n>\n> For those cases, We would like to assert that since our model supports plug-and-play style feedback modules, a new feedback module can be designed to target a task-specific error category. Moreover, our framework also supports using a fine-tuned model as a feedback module which provides even more freedom in fine-tuning smaller models for specific error categories that cannot be successfully addressed by in-context learning or external tool-based feedback modules. \n>\n> Overall, we believe that our framework presents a holistic approach to improving reasoning capabilities in LLMs by presenting a varied set of error categories that are often encountered in reasoning tasks and also provides extensibility by designing new task-specific feedback modules.\n\n**2B. Human evaluation and case studies should be included in the final version.**\n> Currently, we incorporate human evaluation into our metrics for the Entailment Bank dataset, where we asked human annotators to evaluate whether each entailment was valid or not. For GSM-IC and DROP, because the answers are objectively defined, we believe that using automatic evaluation is a sufficient measure of performance.\n>\n> If possible, we would like for the reviewer to clarify what additional areas they believe could be improved through human evaluation.\n>\n> In regards to case studies, we agree that qualitative examples of how MAF iteratively improves a generated solution would benefit our manuscript. We commit to adding such examples from our experiments in the appendix of our revised version.\n\n**2C. The \"redundancy\" module is used in all three evaluated tasks. Does this module share the same prompt across different types of tasks?**\n> In our work, all the in-context learning-based feedback modules are defined as a combination of a Task Instruction and Prompt Examples. When a feedback module for an error category is used across multiple tasks, the Task Instruction remains the same, however, the Prompt Examples are taken from the training set of the corresponding task. \n>\t\t\n>We provide the task instruction used for the Redundancy module below. This task instruction remains the same in all the tasks. \n>\n> ```Check each step/semantically complete block of code for any redundancy errors and suggest fixes. Redundancy errors are steps that contain redundant information, which even though might be factual, is not required to answer the question. Ignore all other types of errors.```\n\n**2D. How do you implement the \"code interpreter\" module?**\n> The generated code is passed as a string using Python Repl. This module requires the generated code to follow a specific format. For example, in our case we expect the model to generate a function named ‘solution’ which contains the correct logic. The stdout and stderr from the execution are then used as feedback. In a case, where the model generates no error, the feedback contains no actionable items and thus the generated code remains the same after refinement. If there is an error, we use that error message as a feedback for the model. \n\n**2E. The feedback module named \"calculator\" is illustrated in Figure 1, but it seems that this module is actually unimplemented (see Table 1).**\n> We would like to note that for GSM-IC since we are generating a Python program, we do not need a calculator separately. Since all the mathematical calculations are conducted as part of the program logic, there is no scope for calculation errors. The figure shows the calculator module separately for clarity and possibilities of different types of feedback modules. \n\nWe are also grateful for the reviewer’s feedback on the overall presentation of our manuscript and will revise the manuscript accordingly.\nWe hope this response adequately addresses your queries. Thank you once more for your constructive feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper improves self-Refine framework by decoupling feedback generation for different error categories, allowing for error-specific tools to be used.\n2. The experimental results demonstrate the effectiveness of MAF in improving the performance of LLMs on complex reasoning tasks.", "reasons_to_reject": "1. The proposed MAF pipeline only supports generating feedback for pre-defined error types, and it might miss other error types that are not covered.\n2. Human evaluation and case studies should be included in the final version.", "questions_to_authors": "1. The \"redundancy\" module is used in all three evaluated tasks. Does this module share the same prompt across different types of tasks?\n2. How do you implement \"code interpreter\" module?", "paper_topic_and_main_contributions": "This paper introduces a framework called Multi-Aspect Feedback (MAF) for improving reasoning in large language models. The framework decouples feedback generation for different error categories and uses error-specific tools to provide feedback and refine the solution. The paper demonstrates the efficacy of MAF on various reasoning tasks and compares it with other iterative refinement baselines. The experimental results show that MAF outperforms the base language models and Self-Refine on complex reasoning tasks.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. The feedback module named \"calculator\" is illustrated in Figure 1, but it seems that this module is actually unimplemented (see Table 1).\n2. Figure captions are needed for the figures in the Appendix to reduce the reading difficulty.\n3. The author should use \\paragraph{} instead of \\textbf{} in latex to start a new paragraph with title (e.g., line 350, 377)"}}, {"review_id": "4geHwFxKCW", "reviewer": "Reviewer_J3st", "summary": "", "questions": "Question A: How did you choose the order of feedback type in the experiment? Do you think different orders of feedback will have impacts on model performance? Like doing the missing step first, and then redundancy, or the reverse order?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comprehensive review and actionable feedback! We carefully consider your concerns and address them below.\n\n**3A. In the contributions, the authors claim their proposed framework outperforms baselines on Math reasoning, such as GSM8K and GSM-IC, but in Sec 4.1 they say that the performance on GSM8K has been saturated and use GSM-IC for the experiment (they indeed show the results of GSM-IC in the paper only).**\n> This is indeed a mistake from our side. In the contribution section, it should be “mathematical reasoning (GSM-IC)” rather than “mathematical reasoning (GSM8K and GSM-IC)“. As stated in our paper Lines 350 - 355, we did not choose GSM8K because the performance on GSM8K has been too saturated. Thanks for noticing this inconsistency.\n>\n> Nonetheless, we still conduct additional experiments to investigate the effects of MAF on GSM8k. We will add the results presented below to the revised version of the manuscript. \n\n\n| Model | GSM8K |\n|--------|-----------|\n| GPT3.5           | 69.20 |\n| \\+ Self-Refine | 69.20 |\n| \\+ Self-Refine (Oracle) | *77.40* |\n| \\+ MAF | **69.80** |\n|\\+ MAF (Oracle) | 73.40 |\n| ChatGPT | 71.80 |\n| \\+ Self-Refine | **74.60** |\n| \\+ Self-Refine (Oracle) | *79.40* |\n| \\+ MAF | 73.20 |\n| \\+ MAF (Oracle) | 76.60 |\n\n**3B. For the two proposed refinement strategies: Eager Refinement and Lazy Refinement, although the author gives an example to show that these two methods are ‘complementary’, I personally think that a detailed comparison of them should be included in the experiments.**\n> We agree that elaborating on the interplay between eager and lazy refinement modules would improve this work. Following suit, we conduct a new experiment on a subset of the GSM-IC dataset where we compare our proposed setting with all modules in either lazy or eager mode. \n\n| Model | GSM-IC 100 | GSM-IC 100 (Oracle) | Over-Refined Instances |\n|--------|-----------------|-------------------------|-----------------------------------------|\n| GPT-3.5 | **71** | 71 | NA |\n| \\+ MAF | 67 | *85* | 13 |\n| \\+ All Lazy | 66 | 81 | 10 |\n| \\+ All Eager | 66 | *85* | 17 |\n| ChatGPT | 73 | 73 | NA |\n| \\+ MAF | **78** | *81* | 3 |\n| \\+ All Lazy | 69 | 80 | 9 |\n| \\+ All Eager | 73 | 80 | 6 |\n\t\n> As can be seen from the results, our framework which combines eager and lazy refinement does indeed outperform all lazy feedback and all eager feedback across both the standard and Oracle verifier settings. Over-Refined Instances in the above table show the number of instances where the model was able to generate the correct solution at Time Step $i$, but regressed at some future time step. We can also see that using all feedback modules in either lazy or eager mode also exacerbates this problem.\n> \n> Logically, one reason that we do not combine all feedbacks in one lazy feedback is that different feedbacks may result in revisions that interfere with each other. For example, Variable Naming feedback may suggest renaming a variable while Missing Step feedback suggests adding a line that uses that variable, resulting in an incorrect final answer. Having all feedback modules be eager also has its weaknesses, as any errors caused by the last feedback module in the sequence will not have a chance to be corrected. Overall, different feedback modules may be better suited to either eager or lazy refinement, so using a mix of them maximizes performance.\n\n> There are also practical concerns that favor using a mix of eager and lazy feedback. Using only lazy feedback can result in many feedback categories being compiled into one prompt. Even with our feedback summarization technique, this can lead to the iteration prompt being longer than the context window of many widely available models. On the other hand, using only eager feedback will result in the solution being rewritten for every feedback module, which will result in high token usage.\n>\n> We will add these details and a new ablation study comparing Lazy and Eager refinement strategies in our revised manuscript.\n\n**3C. In Table 3, the results show that some modules are not necessary, and even might hurt the framework according to the results of MAF (VN, Red, Com). In this regard, and as a long paper, the authors might need to provide more comprehensive experiments to show the effectiveness and necessity of each module for each task.**\n> We agree with the reviewer that some modules have small to no effect on the final results. One of the potential reasons for this is the provided ablation study (removing one feedback module at a time) does not capture the interplay of different feedback modules together. However, due to limited time, we were not able to do a detailed ablation study at this time on the interplay between multiple feedback modules. We will add an ablation on the importance of different feedback modules on each task in our final version. \n\n**3D. How did you choose the order of feedback type in the experiment? Do you think different orders of feedback will have impacts on model performance? Like doing the missing step first, and then redundancy, or the reverse order?**\n> Thank you for raising an interesting and important question. We chose the order of the feedback types intuitively. However, we agree that this is an important question and should be answered in this work. \n> To investigate how the order of various feedback modules affects our model’s performance, we conduct a new ablation study where we try multiple orders for our feedback modules. Specifically, we reverse the order of the Eager Refine modules, and Lazy Refine Modules respectively. We also try reversing the execution order of Eager and Lazy modules, i.e. instead of first applying the eager feedback we first apply all the lazy feedback modules and then apply Eager modules.\n\n| Model | GSM-IC 100 | GSM-IC 100 (Oracle) |\n|--------|----------------|--------------------------|\n| ChatGPT | 73 | 73 |\n| + MAF [Reported] - VN, Syntax, (MS, Com, Red) | **78** | **81** |\n| + MAF - Syntax, VN, (MS, Com, Red) | 73 | 77 |\n| + MAF - VN, Syntax (Red, Com, MS) | 75 | 78 |\n| + MAF - (MS, Com, Red), VN, Syntax | 74 | 80 |\n> VN refers to Variable Naming, MS refers to Missing Step, Com refers to Commonsense, and Red refers to Redundancy. All the modules inside parentheses are Lazy modules and other modules use eager refine strategy.\n\n> As seen from the table above, reversing the order of Eager Refine modules does affect the performance. This is expected because eager modules by design generate refined solutions immediately and in this particular case, generating Variable Naming feedback after fixing syntax errors can introduce new syntax errors. Whereas, changing the order of Lazy feedback modules ideally should have minimal effect on the performance because the feedback from all the modules is passed as one long feedback to the model. However, we note that this is not the case and the model’s performance does depend on the order of lazy feedback modules. We posit that this may be due to the length of the feedback. We will provide a more detailed study on the order of Lazy feedback modules in our revised manuscript.\n\n> Furthermore, in the reported setting, we execute all Eager Refine feedback before moving on to lazy modules because of the nature of the eager feedback modules. The syntactic eager feedback modules (Variable Naming and Syntax) primarily aim to make the generated program valid Python code. We hypothesized that placing these two modules first would result in higher quality feedback from the semantic modules (redundancy, commonsense, missing step) because the variable names and correct syntax could help align the feedback generation with the original problem. This is supported experimentally by testing our MAF framework on a subset of GSM-IC problems with the feedback modules reversed, such that the lazy modules come first, then variable naming, then syntax, which resulted in decreased performance. We will add these details in the revised manuscript. \n\nWe are also grateful for the reviewer’s feedback on the overall presentation of our manuscript and will revise the manuscript accordingly. We hope this response adequately addresses your queries. Thank you once more for your constructive feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Overall experimental results show the effectiveness of the proposed framework in several reasoning tasks.\n- The proposed framework is modular and the modules are plug-and-play, which provides good flexibility for using new models and tools in new tasks.\n- This paper found that over-refinement can be a problem in iterative refinement frameworks, which might be interesting and useful to the community.", "reasons_to_reject": "- In the contributions, the authors claim their proposed framework outperforms baselines on Math reasoning, such as GSM8K and GSM-IC, but in Sec 4.1 they say that the performance on GSM8K has been saturated and use GSM-IC for the experiment (they indeed show the results of GSM-IC in the paper only). \n- For the two proposed refinement strategies: Eager Refinement and Lazy Refinement, although the author gives an example to show that these two methods are ‘complementary’,  I personally think that a detailed comparison of them should be included in the experiments.\n- In Table 3, the results show that some modules are not necessary, and even might hurt the framework according to the results of MAF (VN, Red, Com). In this regard, and as a long paper, the authors might need to provide more comprehensive experiments to show the effectiveness and necessity of each module for each task.", "questions_to_authors": "Question A: How did you choose the order of feedback type in the experiment? Do you think different orders of feedback will have impacts on model performance? Like doing the missing step first, and then redundancy, or the reverse order?", "paper_topic_and_main_contributions": "Given that existing approaches rely on single generic feedback and fail to address the diverse error types in LLM reasoning, this paper proposes a Multi-Aspect Feedback framework based on multiple feedback modules, including LMs and external tools. Overall, this paper might contribute to the \"NLP engineering experiment\".", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Table 1 is placed on page 3, but is not mentioned until page 7, and the table needs more clarification for easy reading.\n- If I understand correctly, Table 3 shows the ablation study results for Sec 4.5, but that section doesn't mention it.\n- In the related work, ‘prefixing the solution generation with Let’s think step-by-step’ is not proposed by Wei et al. 2023 but Kojima et al. 2023.\n- The pictures in the appendix are very blurry and unreadable."}}]

Author rebuttal:

Meta-review: This paper focuses on the problems of integrating multiple feedback modules (including frozen LMs and external tools) for improving reasoning in LLMs. 

The reviewers agree that the problem is interesting and the method is effective. The one major weakness is that the ablation studies suggest that some modules have small to no effect on the final results. Overall, this is above the threshold for acceptance.
Decision: Accept-Main