Large Language Models Cannot Self-Correct Reasoning Yet
Authors: Jie Huang
Xinyun Chen
Swaroop Mishra
Huaixiu Steven Zheng
Adams Wei Yu
Xinying Song
Denny Zhou
Venue: ICLR 2024 poster (2024)
Keywords: Large Language Models
Self-Correction
Reasoning
Primary area: general machine learning (i.e., none of the above)
TL;DR: This paper critically examines the role and efficacy of self-correction within LLMs, shedding light on its true potential and limitations.

Abstract: Large Language Models (LLMs) have emerged as a groundbreaking technology with their unparalleled text generation capabilities across various applications. Nevertheless, concerns persist regarding the accuracy and appropriateness of their generated content. A contemporary methodology, self-correction, has been proposed as a remedy to these issues. Building upon this premise, this paper critically examines the role and efficacy of self-correction within LLMs, shedding light on its true potential and limitations. Central to our investigation is the notion of intrinsic self-correction, whereby an LLM attempts to correct its initial responses based solely on its inherent capabilities, without the crutch of external feedback. In the context of reasoning, our research indicates that LLMs struggle to self-correct their responses without external feedback, and at times, their performance even degrades after self-correction. Drawing from these insights, we offer suggestions for future research and practical applications in this field.

Published as a conference paper at ICLR 2024
LARGE LANGUAGE MODELS CANNOT SELF-CORRECT REASONING YET
Jie Huang Xinyum Chen Swaroop Mishra' Xinying Song' Denny Zhou'
Huaixiu Steven Zheng' Adams Wei Yu!
Google DeepMind University of Illinois at Urbana-Champaign
jeffh 11 inoi edu, (xinyunchen, dennyzhou]
ABSTRACT
Large Language Models (LLMs) have emerged as groundbreaking technology with their unparalleled text generation capabilities across various applications. Nevertheless, concerns persist regarding the accuracy and appropriateness of their generated content. A contemporary methodology. self-correction, has been pro- posed as remedy to these issues. Building upon this premise, this paper critically examines the role and efficacy of self-correction within LLMs, shedding light on its true potential and limitations. Central to our investigation is the notion of in- trinsic self-cornn correctio,, whereby LLM attempts to correct its initial responses based solely on its inherent capabilities, without the crutch of external feedback. In the context of reasoning, correct their responses without external feedback, and at times, their performance even degrades after self-correction. Drawing from these insights, we offer sugges- tions for future research and practical applications in this field.
research indicates that L.Mss struggle self-
INTRODUCTION
The rapid advancements in the domain of artificial intelligence have ushered in the era of Large Language Models (LLMss.. These models, characterized by their expansive parameter counts and unparalleled capabílities text generation, have showcased promising results across a multitude of applications (Chowdhery et al. 2023 Anil et al. 2023 OpenAl 2023 inter alia). However, concerns about their accuracy, reasoning capabilities, and the safety of their generated content have
drawn significant attention from the community (Bang al. 2023 Alkaissi & McFarlane 2023
Zheng et al.) [023] Shi et al.) 2023] Carlini et al. 2002 Huang et al. [022]] Shao et al.] 2023
al. 2023
Wei al.) 2023 Zhou et al.) 2023b; Zou al. [002]] inter alia).
Amidst this backdrop. the concept of "self-correction" has emerged as a promising solution, where
LLMs refine their responses based feedback to their previous outputs (Madaan al.) 2023
Welleck et al. 2023] Shinn et al. 2023 Kim et al. [023 Bai et al. [022]] Ganguli et al., 2023
Gao et al.] 2023 Paul al. 2023 Chen et al. [203bb Pan et al. 2023] inter alia). However, the
underlying mechanics and efficacy of self-correction in LLMs remain underexplored. A fundamen-
tal question arises: If LLM possesses the ability self-correct, why doesn't 't it simply offer the correct answer in its initial attempt? This paper delves deeply into this paradox, critically examining the self-correction capabilities of I.L.Ms, with a particular emphasis on reasoning (Wei et al. 2022
Z.hou et al.) 2023b Huang & Chang 2023).
To study this, we first define the concept of intrinsic self-correction, a scenario wherein the model endeavors rectify its initial responses based solely on its inherent capabilities, without the crutch of external feedback. Such setting is crucial because high-quality external feedback often un- available in many real-world applications. Moreover, it is vital to understand the intrinsic capabilities of LLMs. Contrary to the optimism surrounding self-correction (Madaan et al.) 2023 Kim et al.]
2023 Shinn al. 2023: Pan et al. 2023] inter alia), our findings indicate that LLMs struggle to self-correct their reasoning in this setting. In most instances, the performance after self-correction
"Equal contribution.

---

Published as a conference paper at ICLR 2024
even deteriorates. This observation is in contrast prior research such Kim al. (2023) Shinn jet al. (2023). Upon closer examination, we observe that the improvements in these studies result from using oracle labels to guide the self-correction process, and the improvements vanish when oracle labels are not available.
Besides the reliance on oracle labels, also identify other issues in the literature regarding mea- suring the improvement achieved by self-correction. First, we note that self-correction, by design, utilizes multiple LLM responses, thus making it crucial to compare inference costs. From this perspective, we investigate multi-agent debate (Du et al., 2023 Liang al. 2023) as means to improve reasoning. where multiple LLM instances (can be multiple copies of the same LLM) critique each other's 's responses. However, our results reveal that its effi- cacy no better than ceff-consitteey [Wang et al. 2022) when considering an equivalent number of responses, highlighting the limitations of such an approach.
baselines with equivalent
Another important consideration for self-correction involves prompt design. Specifically, each self- correction process involves designing prompts for both the initial response generation and the self- correction steps. Our evaluation reveals that the self-correction improvement claimed by some exist- ing work stems from the sub-optimal prompt for generating initial responses, where self-correction corrects these responses with more informative instructions about the initial task in the feedback prompt. In such cases, simply integrating the feedback into the initial instruction can yield better results, and self-correction again decreases performance.
In light of our findings, we provide insights into the nuances of LLMs' self-correction capabilities and initiate discussions to encourage future research focused on exploring methods that can gen- uinely correct reasoning.
2 BACKGROUND AND RELATED WORK
With the LLM evolution, the notion of self-correction gained prominence. The discourse on self- correction pivots around whether these advanced models can recognize the correctness of their out- puts and provide refined answers (Bai et al. [2022 Madaan et al. 2023: Welleck et al. 2023 inter alia). For example, in the context of mathematical reasoning. an LLM might initially solve a com- plex problem but make an error in one of the calculation steps. In an ideal self-correction scenario, the model is expected to recognize the potential mistake, revisit the problem, correct the error, and consequently produce a more accurate solution.
Yet, the definition of "seeff-correctoon" varies across the literature, leading to ambiguity. pivotal distinction lies in the source of feedback (Pan et al. (202)), which can purely come from the L.M,, or can be drawn from external inputs. Internal feedback relies on the model's inherent knowledge and parameters to reassess its outputs. In contrast, external feedback incorporates inputs from humans,
other models (Wang et al. 2023b Paul et all
2023 inter alia), or external tools and knowledge
sources (Gou al. 2023 hen et al. 2023b O Olausson al. 2023 Gao et al. 2023] inter alia).
In this work, we focus on examining the self-correction capability of LLMs for reasoning. Reasoning fundamental aspect of human cognition, enabling to understand the world, draw inferences, make decisions, and solve problems. To enhance the reasoning performance of L.Mss, Kim et al. (2023); et al. (2023) use oracle labels about the answer correctness to guide the self-correction process. However, in practice, high-quality external feedback such as answer correctness is often unavailable. For effective self-correction, the ability judge the correctness of answer is crucial and should ideally be performed by the LLM itself. Consequently, our focus shifts to self-correction without any external or human feedback. We term this setting intrinsic self-correction. For brevity, unless explicitly stated otherwise (e.g., self-correction with oracle feedback), all references to "self- correction" in the remainder of this paper pertain intrinsic self-correction.
In the following sections, we will evaluate a variety of existing self-correction techniques. We demonstrate that existing techniques actually decrease reasoning performance when oracle labels are not used (Section 3). perform worse than methods without self-correction when utilizing the same number of model responses (Section 4. and lead to less effective outcomes when using infor- mative prompts for generating initial responses (Section We present overview of issues in the evaluation setups of previous LLM self-correction works in Table with detailed discussions in the corresponding sections.
2

---

Published as a conference paper at ICLR 2024
Table 1: Summary of issues in previous L.MM self-correction evaluation.
Method
Issue
RCI (Kim et al. 2023:: Reflexion (Shinn et al.
2023
Use of oracle labels (Section Unfair comparison to self-consistency (Section
Multi-Agent Debate (Du al. 2025
Self-Refine (Madaan 2023
Sub-optimal prompt design (Section
3 LLMs CANNOT SELF-CORRECH REASONING INTRINSICALLY
In this section, we evaluate existing self-correction methods and compare their performance with and without oracle labels regarding the answer correctness.
3.1 EXPERIMENTAL SETUP
Benchmarks. We use datasets where existing self-correction methods with oracle labels have demonstrated significant performance improvement, including
GSM8K (Cobbe et al., 2021) GSM8K comprises test set of 1,319 linguistically diverse grade
school math word problems, curated by human problem writers. There is notable improvement of approximately 7% evidenced by Kim et al. (2023) after self-correction.
CommonSenseQA (Talmor et al. 2019): This dataset offers collection of multi-choice questions that test commonsense reasoning. An impressive increase of around 15% showcased through the self-correction process, as demonstrated by Kim et al. (202)). Following Kojima et al. (2022); Kim et al. (2023), we utilize the dev for our evaluation, which encompasses 1,221 questions.
HotpotQA (Yang et al. 2018): HotpotQA is an open- open-domain multi-hop question answering dataset. Shinn et al.J (2023) demonstrate significant performance improvement through self- correction. We test models' performance in a closed-book setting and evaluate them using the same set as Shinn et al. (023). This set contains 100 questions, with exact match serving as the
evaluation metric.
Test Models and Setup. We first follow Kim et al. (023); Shinn et al. (2023) to evaluate the per- formance of self-correction with oracle labels, using EPT--35-Turte 3 3-tuuboo-66111 and GPT-4 accessed 2023/08/29.
For intrinsic self-correction, to provide more thorough analysis, we also evaluate GPT-4-Turbo (I.l (1llama---70-chea (Touvron al., 2023). For GPT-3.5-Turbo, we employ the full evalua- tion set. For other models, to reduce the cost, we randomly sample 200 questions for each dataset (100 for HotpotQA) for testing. We prompt the models to undergo maximum of two rounds of self-correction. We use temperature of for GPT-3.5-Turbo and GPT-4, and temperature of 0 for GPT-4-Turbo and 2lama-2, provide evaluation across different decoding algorithms.
and Llama-2
Following Kim et al. (2023); Shinn et al. (2023). we apply a three-step prompting strategy for self-correction: 1) prompt the model to perform initial generation (which also serves as the results for Standard Prompting); 2) prompt the model to review its previous generation and produce feedback; 3) prompt the model to answer the original question again with the feedback.
Prompts.
For our experiments, we mostly adhere to the prompts from the source papers. For GSM8K and CommonSenseQAA, we integrate format instructions into the prompts of Kim et al. (2023) to facilitate a more precise automatic evaluation (detailed prompts can be found in Appendix Aj. For HotpotQA, the same prompt as Shinn al. (2023). We also assess the performance of various self- correction prompts for intrinsic self-correction. For example, we use "Assume that this answer could be either correct or incorrect. Review the answer carefully and report any serious problems you find." as the default feedback prompt for the evaluation on GPT-4-Turbe and Llama-2.
3.2 RESULTS
Correctrrttion with Oracle Labels. Following previous works (Kim et al., 2023 Shinn et al. the correct label to determine when to stop the self-correction loop. This means
2023
3

---

Published as a conference paper at ICLR 2024
Table 2: Results of GPT-3.5 and GPT-4 on reasoning henchmarks with oracle labels.
GSM8K
ConmonSenseQQ
HotpotQA
Standard Prompting
75.9
75.8
26.0
GPT-3.5
Self-Correct (Oracle)
84.3
89.7
29.0
Standard Prompting
95.5
82.0
49.0
GPT-4
Self-Correct (Oracle)
97.5
85.5
59.0
Table 3: Results of GPT-3.5 and GPT-4 on reasoning benchmarks with intrinsic self-correction.
# calls
GSM8K
CommonSenseQAA
HotporQA
Standard Prompting
1
75.9
75.8
26.0
GPT-3.5
Self-Correct (round 1)
3
75.1
38.1
25.0
Self-Correct (round
5
74.7
41.8
25.0
Standard Prompting
1
95.5
82.0
49.0
GPT-4
Self-Correct (round 1)
3
91.5
79.5
49.0
Self-Correct (round 2)
5
89.0
80.0
43.0
utilize the ground-truth label to verify whether each step's generated answer correct. If the answer already correct, no (further) self-correction will be performed. Table summarizes the results of self-correction under this setting, showcasing significant performance improvements, consistent with the findings presented in Kim et al. (2023) Shhnn al. (2023).
However, these results require careful consideration. For reasoning tasks, like solving mathematical problems, the availability of oracle labels seems counter-intuitive. If we are already possession of the ground truth, there seems be little reason to deploy LLMs for problem-solving. Therefore, the results can only be regarded as indicative of an oracle's performance.
Intrinsic Self-Correction. Per the above discussion, performance improvements achieved using oracle labels do not necessarily reflect true self-correction ability. Therefore, we turn our focus to the results in the intrinsic self-correction setting as defined Section [ To achieve this, we elimi- nate the use of labels, requiring LLMS to independently determine when to stop the self-correction process, i.e., whether retain their previous answers.
Tables and 4 report the accuracies and the number of model calls. We observe that, after self- correction, the accuracies all models drop across all benchmarks.
To provide more comprehensive assessment, we also design several different self-correction prompts to determine there are better prompts that could enhance reasoning performance. Nonetheless, shown in Tables 5 and 0 without the use of oracle labels, self-correction consis- tently results in a decrease in performance.
3.3 WIYY DOES THE PERFORMANCE NOT INCREASE, BUT INSTEAD DECREASE?
Empirical Analysis. Figure [ summarizes the results of changes in answers after two rounds of self-correction, with two examples of GPT-3.5 illustrated Figure 2 For GSM8K, 74.7% of the time, GPT-3.5 retains its initial answer. Among the remaining instances, the model is more likely to modify correct answer to an incorrect one than to revise an incorrect answer correct one. The fundamental issue is that LLMs cannot properly judge the correctness of their reasoning. For CAnmonSenseeAA, there higher chance that GPT-3.5 alters its answer. The primary reason for this is that false answer options in CommonSenseeA often appear somewhat relevant to the ques- tion, and using the self-correction prompt might bias the model to choose another option, leading to high "correct incorrect" ratio. Similarly, Llama-2 also frequently converts correct answer into an incorrect one. Compared to GPT-3.5 and Llama-2, both GPT-4 and GPT-4-Turbo are more likely retain their initial answers. This may be because GPT-4 and GPT-4-Turbo have higher confidence

---

Published as a conference paper at ICLR 2024
Table 4: Results of GPT-4-Turbo and Llama-2 with intrinsic self-correction.
calls
GSM8K
CommonSenseQA
Standard Prompting
1
91.5
84.0
GPT-4-Turbo
Self-Correct (round I)
88.0
81.5
Belf-Correct (round 2)
5
90.0
83.0
Standard Prompting
1
62.0
64.0
Llama-2
Self- CelreCorrect (round 1)
3
43.5
37.5
Self-Correct (round 2)
5
36.5
36.5
Table 5: Results of GPT-4-Turbo with different feedback prompts.
# calls
GSM8K
CommonSenseQQ
Standard Prompting
1
91.5
84.0
Feedback Prompt:
Assume that this answer could be either correct or incorrect. Review the answer carefully and report any serious problems you find.
Self-Correct (round 1)
3
88.0
81.5
Self-Correct (round 2)
5
90.0
83.0
Feedback Prompt: Review your previous answer and determine whether it's correct. If wrong, find the problems with your answer.
Self-Corred (round 1)
3
90.0
74.5
Self-Correct (round 2)
5
90.0
81.0
Feedback Prompt: Verify whether your answer is correct, and provide an explanation.
Self-Correct (round 1)
3
91.0
81.5
Self-Correct (round 2)
5
91.0
83.5
Table 6: Results of Llama-2 with different feedback prompts.
# calls
GSMSK
CommonSenseQQA
Standard Prompting
1
62.0
64.0
Feedback Prompr:
Assume that this answer could be either correct incorrect. Review the answer carefully and report any serious problems you find.
Self-Correct (round 1)
3
43.5
37.5
Celf-Correct (round 2)
5
36.5
36.5
Feedback Prompt: Review your previous answer and determine whether it's correct. If wrong, find the problems with your answer.
Self-Correct (round 1)
3
46.5
26.0
Self-Correct (round 2)
5
30.5
37.0
Feedback Prompt: Verify whether your answer is correct, and provide an explanation.
Self-Correct (round 1)
3
58.0
24.0
Self-Correct (round 2)
5
41.5
43.0
in their initial answers, because they more robust and thus less prone to being biased by the self-correction prompt]
We omit the analysis on HotpotQA because the sample size used in the source paper is quite small, which may not produce meaningful statistics.
5

---

Published as a conference paper at ICLR 2024
Incorrect in Incorrect
Incorrect Correct
39.0%
Correct - Incorrect
1.5%
8.0%
6%
90.5%
90.5%
74.7%
42.
No Change
8K (GPT-3.5)
CormonSense@AA (GPT-3.5)
GSMBK (GPT-4)
CommonSenseeA (GPT-4)
31.0%
5.5%
35.5%
2.0%
8.0%
6.0%
95.0%
26.5%
88. 0%
40.0%
40.0%
GSM
CommonSense@A. (TPT--Turbb)
GSMBK
CommenSense@A (Lisma
Figure 1: Analysis of the changes in answers after two rounds of self-correction. No Change: The answer remains unchanged; Correct = Incorrect: A correct answer is changed to an incorrect one; Incorrect Correct: An incorrect answer is revised to correct one; Incorrect Incorrect: An incorrect answer is altered but remains incorrect.
Christina is planning birthday party and needs .75 gift bags per invited guest, because 1/4 of attendees don't show up. She invited 16 friends. Gift bugs are $2 euch. How much will she spend?
Terry cats 2 yogurts day. They currently sale 4 yogurts for $5.00. How much does he spend on yogurt over 30 days?
So. Terry spends Siboxed (75.005$ on yogurt over 30 days, (Correct)
Therefore, Christina will spend $18 on gift bags.
Answer: (boxed(18). (Incorrect)
Review your previous answer find problems your answer.
your previous answer find problems with your
Upon reviewing my previous answer, realized that I made mistake in caleulating Terry's spending yogurt days. incorrectly stated that Terry spends $2.50 per day for 30 days, resulting in total of $75.00...
Upon reviewing my previous answer, recognize mistake. incorrectly stated that Christina needs 0,75 gift bags invited guest. The given information states that 1/4 of the attendees won't show up...
Based on the problems you found, improve your answer.
Based on the problems you found, improve your answer.
... Therefore, Christina will spend $24 gift bags. (boxed(24). (Correct)
Therefore, answer Terry spends $\boxed(37.50)) yogurt over days. (Incorrect)
Figure 2: Examples on GSM8K with GPT-3.5. Left: successful self-correction; Right: failed self- correction. Full prompts and responses can be viewed in Figures and of Appendix
Let's take another look the results presented in Table These results use ground -ruth labels to prevent the model from altering a correct answer to an incorrect one. However, determining how to prevent such mischanges is, fact, the key ensuring the success of self-correction.
Intuitive Explanation. If the model is well-aligned and paired with thoughtfully designed ini- tial prompt, the initial response should already be optimal relative to the prompt and the specific decoding algorithm. Introducing feedback can be viewed adding additional prompt, poten- tially skewing the model towards generating a response that is tailored to this combined input. In an intrinsic self-correction setting, on the reasoning tasks, this supplementary prompt may not offer any extra advantage for answering the question. fact, might even bias the model away from producing optimal response to the initial prompt, resulting in performance drop.

---

Published as a conference paper at ICLR 2024
Table 7: Results of multi-agent debate and self-consistency.
responses
GSM8K
Standard Prompting
76.7
Self-Consistency
82.5
Multi-Agent Debate (round 1)
83.2
Sclf-Consistency
85.3
Multi-Agent Debate (round 2)
83.0
Self-Consistency
88.2
Table 8: Results of Constrained Generation.
# calls
CommonGen-Hard
Standard Prompting*
1
44.0*
Self-Correct*
7
67.0*
Standard Prompting*
53.0
Self- #
7
61.1
Standard Prompting (ours)
81.8
Self- -off-Coreett*
7
75.1
Prompts and results from Madaan et al. (2023).
MULTI-AGENT DEBATE DOES NOT OUTPERFORM SELF-CONSISTENCY
Another potential approach for LLMs to self-correct their reasoning involves allowing the models to critique and debate through multiple model calls (Du et al. 2023 Liang et al. 2023 Chen et al.) 2023a). Du et al. (2023) implement a multi-agent debate method by leveraging multiple instances of single ChatGPT model and demonstrate significant improvements
reasoning tasks. We adopt their method to test performance on GSM8K. For an unbiased implementation, we use the exact prompt Du al. 2023) and replicate their experiment with the gpt-3. 3 5ttturooo0033 model, incorporating 3 agents and rounds of debate. The only distinction is that, to reduce result variance, test the complete test set of GSM8K, compared to their usage of 100 examples. For reference, we also report the results of self-consistency (Wang et al. (202)), which prompts models generate multiple responses and performs majority voting to select the final answer.
Table 7 presents the results. The results indicate that both multi-agent debate and self-consistency achieve significant improvements over standard prompting. However, when comparing multi-agent debate to self-consiie self-consistency, we observe that the performance of multi-agent is only slightly better than that of self- ceff-consiteenyy with the same number of agents (3 responses, the baseline also compared in Du et al. (2023)). Furthermore, for self-consistency with an equivalent number of responses, multi-agent debate significantly underperforms simple self-consistency using majority voting.
In fact, rather than labeling the multi-agent debate as form of "debate" or "critique", it is more appropriate to perceive
a means to achieve "consistency" across multiple model generations. Fundamentally, its concept mirrors that of self-consistency: the distinction lies in the voting mech- anism, whether voting is model-driven or purely based on counts. The observed improvement is evidently not attributed to "self-correction" but rather to "self-consistency" If we aim to argue that LLMs can self-correct reasoning through multi-agent debate, is preferable to exclude the effects of selection among multiple generations.
PROMPT DESIGN ISSUES SELF-CORRECTION EVALUATION
In Section 3 we observe that although self-correction decreases reasoning performance with all types of feedback prompts we have evaluated, performance varies with different feedback prompts. In this section, further emphasize the importance of proper prompt design in generating initial LLM responses to fairly measure the performance improvement achieved by self-correction. For example, if task requires that the model response should meet criteria that be easily specified

---

Published as a conference paper at ICLR 2024
in the initial instruction (e.g., the output should contain certain words, the generated code should be efficient, the sentiment should be positive, etc.), instead of including such requirements only in the feedback prompt, an appropriate comparison would be to directly and explicitly incorporate these requirements into the prompt for generating initial responses. Otherwise, when the instruction for generating initial predictions is not informative enough, even if the performance improves, it is unclear whether the improvement merely comes from more detailed instructions in the feedback prompt or from the self-correction step itself.
To illustrate such prompt design issues in the self-correction evaluation of some prior work, we take the Constrained Generation task in Madaan et al. (2023) as an example, where the task re- quires models to generate coherent sentences using all 20- 30 input concepts. The original prompt in Madaan et al. (023) (Figure 7 does not clearly specify that the LLM needs to include all concepts in the prompt: thus, they show that self-correction improves task performance by asking the model to identify missing concepts and then guiding it to incorporate these concepts through feedback.
Based on this observation, we add the following instruction "Write reasonable paragraph that includes *ALL* a the above concepts" to the prompt for initial response generation (refer to Figure for the full prompt). Following Madaan et al. (2023), we use concept coverage as the metric. We reference their results and replicate their experiments using gpt 3ppt-3.5-tuu 5.t.ttrro-06633 Table demonstrates that our prompt, denoted as Standard Prompting (ours), significantly outperforms the results after self-correction of Madaan et al.] (2023), and applying their self-correction prompt on top of model responses from our stronger version of the standard prompting again leads decrease in performance.
CONCLUSION AND DISCUSSION
Our work shows that current LLMs struggle to self-correct their reasoning without external feed- back. This implies that expecting these models to inherently recognize and rectify their reasoning mistakes overly optimistic so far. In light of these findings, it imperative for the community to approach the concept of self-correction with a discerning perspective, acknowledging its potential and recognizing its boundaries. By doing so, we can better equip the self-correction technique to address the limitations of LLMs and develop the next generation of LLMs with enhanced capabili- ties. the following, we provide insights into scenarios where self-correction shows the potential strengths and offer guidelines the experimental design of future self- self-correction techniques to ensure a fair comparison.
Leveraging external feedback for correction. In this work, demonstrate that current LLMs cannot improve their reasoning performance through intrinsic self-correction. Therefore, when valid external feedback is available, it is beneficial to leverage it properly to enhance model performance. For example, Chen et al.] (2023b) show that LLMs can significantly improve their code generation performance through self-debugging by including code execution results in the feedback prompt to fix issues in the predicted code. In particular, when the problem description clearly specifies the in- tended code execution behavior, e.g., with unit tests, the code executor serves as the perfect verifier to judge the correctness of predicted programs, while the eттоr messages also provide informative feedback that guides the LLMs to improve their responses. Gou al. (2023) demonstrate that LLMs can more effectively verify and correct their responses when interacting with various external tools such as search engines and calculators. Cobbe al.) (202T); Lightman et al. (2023); Wang et al.) (2023b) train a verifier or critique model on a high-quality dataset to verify refine LLM outputs, which can be used to provide feedback for correcting prediction errors. Besides automatically gen- erated external feedback, we also often provide feedback ourselves when interacting with LLMs, guiding them to produce the content we desire. Designing techniques that enable LLMs to interact with the external environment and learn from different kinds of available feedback is promising direction for future work.
Evaluating self-correction against baselines with comparable inference costs. By design, self- correction requires additional LLM calls, thereby increasing the costs for encoding and generating extra tokens. Section 4 demonstrates that the performance of asking the LLM to produce final response based on multiple previous responses, such as with the multi-agent debate approach, is in- ferior to that of self-consistency (Wang et al. 2022) with the same number of responses. Regarding
8

---

Published as a conference paper at ICLR 2024
this, encourage future work proposing new self-correction methods to always include in-depth inference cost analysis to substantiate claims of performance improvement. Moreover, strong base- lines that leverage multiple model responses, like self-consistenge should be used for comparison. An implication for future work is to develop models with a higher probability of decoding the opti- mal solution in their answer distributions, possibly through some alignment techniques. This would enable the model to generate better responses without necessitating multiple generations.
Putting equal efforts into prompt design. As discussed in Section 5 to gain better understanding of the improvements achieved by self-correction, is important to include a complete task descrip- tion the prompt for generating initial responses, rather than leaving part of the task description for the feedback prompt. Broadly speaking, equal effort should be invested in designing the prompts for initial response generation and for self-correction; otherwise, the results could be misleading.
7 LIMITATIONS AND BROADER IMPACT
Although we have conducted comprehensive evaluation spanning a variety of self-correction strate- gies, prompts, and benchmarks, our work focuses on evaluating reasoning of I.LMs. Thus, it is plausible that there exist self-correction strategies that could enhance LLM performance in other domains. For example, prior works have demonstrated the successful usage of self-correction that aligns model responses with specific preferences, such as altering the style of responses or enhanc- ing their safety (Bai et al. 2022 Ganguli et al. [023] Madaan et al., 2023). A key distinction arises in the capability of LLMs to accurately assess their responses relation the given tasks. For example. LLMs can properly evaluate whether response is inappropriate (Ganguli et al.) 2023).
but they may struggle to identify errors in their reasoning.
Furthermore, several prior works have already shown that LLM self-correction performance be- comes significantly weaker without access to external feedback (Gou et al. [2023] Zhou et al. (2023a) and can be casily biased by misleading feedback (Wang et al. (023a)), which is consistent with our findings in this work. However, we still identified prevailing ambiguity in the wider community. Some existing literature may inadvertently contribute this confusion, either by relegating crucial details about label usage to less prominent sections or by failing to clarify that their designed self- correction strategies actually incorporate external feedback. Regarding this, our paper serves call to action, urging researchers to approach this domain with a discerning and critical perspective. We also encourage future research to explore approaches that can genuinely enhance reasoning.
REPRODUCIBILITY STATEMENT
Our experiments utilize GPT-3.5 and GPT-4, which are accessible via the public API https: //plat www.efow....p..i.ooooooooaa.a... www.ai.en.........ooo... com/ docs/moded as well as Llama-2, an open-source model. To facilitate reproducibility, we detail the specific kernels used, e.g., 3. 5-turbo-066 or provide the access times for each experiment. We use prompts from previous works when possible. For our designed prompts, we include the exact prompts in Appendix A
ACKNOWLEDGEMENT
We would like to thank Chen Liang, William Cohen, Uri Alon, and other colleagues Google DeepMind for valuable discussion and feedback.
REFERENCES
Hussam Alkaissi and Samy McFarlane, Artificial hallucinations in chatgpt: implications scien- tific writing, Cureus, 15(2), 2023.
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint 33 2023.

---

Published as a conference paper at ICLR 2024
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harm- lessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022.
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. A multitask, multilingual, multimodal evaluation of chatgpt reasoning, hallucination, and interactivity. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguisties (Volume I: Long Papers), pp. 675-718, 2023.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom B Brown, Dawn Song, Ulfar Erlingsson, al. Extracting training data from large language models. In USENIX Security Symposium, volume 6, 2021.
Justin Chih-Yao Chen, Swarnadeep Saha, and Mohit Bansal. Reconcile: Round-table conference improves reasoning via consensus among diverse llms. arXiv preprint arXiv:2309. 13007, 2023,,
Xinyun Chen, Maxwell Lin, Nathanacl Schärli, and Denny Zhou. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05728, 2023b.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tswyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard. Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski. Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Er- ica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023. URL http: 7/jmlr. htt.lrrrrgpppppss////// v24/ 1144. html
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint 2xiv:2110.14166 2021.
Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improv- ing factuality and reasoning in language models through multiagent debate. arXiv preprint arXiv:2305. 14325, 2023.
Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas Liao, Kamilé Lukošiūté, Anna Chen, Anna Goldie, Azalia Mirhoseini, Catherine Olsson, Danny Hermandez, al. The capacity for moral self-correction in large language models. arXiv preprint arXiv:2302.07459 2023.
Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, al. Rarr: Researching and revising what language models say, using language models. In Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 16477-16508, 2023.
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. Critie: Large language models can self-correct with tool-interactive critiquing. arXiv preprint arXiv:2305.11733 11738 2023.
Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language models: A sur- vey. In Findings of the Association for Computational Linguistics: ACL 2023. Association for Computational Linguistics, 2023.
10

---

Published as a conference paper at ICLR 2024
Jie Huang, Hanyin Shao, and Kevin Chen Chen-Chuan Chang. Are large pre-trained language models leaking your personal information? In Findings of the Association for Computational Linguis- tics: EMNLP 2022. pp. 2038-2008000 2047, Abu Dhabi, United Arab Emirates, 2022. Association for Computational Linguistics.
Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Language models can solve computer tasks. Advances in Neural Information Processing Systems, 2023.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 0221199222113 2022.
Haoran Li. Dadi Guo, Wei Fan, Mingshi Xu, Jie Huang. Fanpu Meng, and Yangqiu Song. Multi- step jailbreaking privacy attacks chatgpt. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 4153- 2023.
Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang. Yujiu Yang, Zhaopeng Tu, and Shuming Shi. Encouraging divergent thinking in large language models through multi- agent debate. arXiv preprint arXiv:2305.19) /9/01//////... 2023.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20050 2023.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uni Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 2023.
Theo X Olausson, Jeevana Priya Inala, Chenglong Wang. Jianfeng Gao, and Armando Solar- Lezama. Demystifying gpt self-repair for code generation. arXiv preprint arXiv:2306.098%0 2023.
OpenAL. Gpt-4 technical report, 2023.
Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang. and William Yang Wang. Automatically correcting large language models: Surveying the landscape of diverse self- correction strategies. arXiv preprint arXiv:2308.033 2023.
Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beatriz Borges, Antoine Bosselut, Robert West, and Boi Faltings. Refiner: Reasoning feedback on intermediate representations. arXiv preprint arXiv:2304.01 00990044 2023.
Hanyin Shao, Jie Huang, Shen Zheng, and Kevin Chen-Chuan Chang. Quantifying association capabilities of large language models and its implications on privacy leakage. arXiv preprint arXiv:2305. 12707, 2023.
Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning, pp. 31210-30 31227. PMLR, 2023.
Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Infor- mation Processing Systems, 2023.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsensegaa A question answering challenge targeting commonsense knowledge. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, Volume 1 (Long and Short Papers), pp. 4149-4158, 2019.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint urX2v:2307.09288, 2023.

---

Published as a conference paper at ICLR 2024
Boshi Wang, Xiang Yue, and Huan Sun. Can chatgpt defend its belief in truth? evaluating Ilm reasoning via debate. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 11865-1188111 2023a.
Tianlu Wang, Ping Yu, Xiaoqing Ellen Tan, Sean O'Brien, Ramakanth Pasunuru, Jane Dwivedi- Yu, Olga Golovneva, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz. Shepherd: A critic for language model generation. arXiv preprint arXiv:2308. 04592, 2023b.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2022.
Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does Ilm safety training fail? arXiv preprint arXiv:2307.02488 2023.
Jason Wei, Xuezhi Wang. Dale Schuurmans. Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824 25224824-24837, 2022.
Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. Generating sequences by learning to self-correct. In The Eleventh International Conference on Learning Representations, 2023.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D) Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2018.
Shen Zheng, Jie Huang, and Kevin Chen-Chuan Chang. Why does chatgpt fall short in providing truthful answers? ArXiv preprint, abs/2304. 10513, 2023.
Aojun Zhou, Ke Wang. Zimu Lu, Weikang Shi, Sichun Luo. Zipeng Qin, Shaoqing Lu, Anya Jia, Linqi Song. Mingjie Zhan, et al. Solving challenging math word problems using gpt-4 code interpreter with code-based self-verification. arXiv preprini 2307022308.07921, 2023a.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuur- mans, Claire Cui, Olivier Bousquet, Quoc V Le, et al. Least-to-moo most prompting enables complex reasoning in large language models. The Eleventh International Conference on Learning Rep- resentations, 2023b.
Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307. 15043, 2023.
12

---

Published as a conference paper at ICLR 2024
A PROMPTS AND EXAMPLE OUTPUTS
Can you solve the following math problem? Christina is planning birthday party and needs 75 gift bags per invited guest, because 1/4 of attendees don't show up. invited friends. Gift bags $2 each. How much will she spend? Explain your reasoning. Your final answer should be single numerical number, in the form |boxed(answer), at the end of your response.
Christina invited 16 friends, then 1/4 of them won't show up, which means there will be 16 * 1/4 = 4 friends who won't attend the party. To determine how many gift bags to buy, we need to subtract the number of friends who won't show up from the total number of invited guests. The number of invited guests
16 4 12. Since Christina needs 0.75 gift bags per invited guest, we can multiply this quantity by the number of invited guests to find the total number of gift bags needed, which is 0.75 12 - gift bags. If each gift bag is $2, then Christina will
spend total of 2 9 $<<2+9=18>>18 on gift bags. Therefore, Christina will spend $18 on gift bags. Answer: \boxed[18). (Incorrect)
Review your previous answer and find problems with your answer.
Upon revicwing my previous answer, recognize a mistake. incorrectly stated that Christina needs 0.75 gift bags per invited guest. The given information states that 1/4 of the attendees won't show up, so the correct calculation should take this into account. To calculate the will multiply the namber invited guests
number of gift bags needed, by the ratic of attendees who will actually show up (758). The number of invited guests is 16, and if 1/4 of them won't show up, then the ratio of attendees who will show up is invited guests by the ratio of attendees who will show up gives us: 16 (3/4) 12 gift bags needed. Since each gift bag costs $2, the total $2 $<<12x2=245224.
1/4 - 3/4. Multiplying the number of
amount Christina will spend on gift bags is:
Based the problems you found, improve your answer. Please reiterate your answer, with your final answer single numerical number, in the form (boxed(answer).
Upon reviewing my previous answer, I recognize a mistake. I incorrectly stated that Christina needs 0.75 gift bags per invited guest. The given information states that 1/4 of the attendees won't show up, the correct calculation should take this into account. To calculate the number of gift bags needed, we will multiply the number of invited guests by the ratic of attendees who will actually show up (758). The number of invited guests is 16, and if 1/4 of them won't show up, then the ratio of attendees who will show up is invited guests by the ratio of attendees who will show up gives us: 16 (3/4) = 12 gift bags needed. Since each gift bag costs $2, the total amount Christina will spend on gift bags is: 12 $2 $<<12+2-24>>24. Therefore, Christina will spend $24 gift bags. Answer: (2x (Correct)
1/4 3/4, Multiplying the number of
Figure 3: Example on GSM8K where self-correction changes an incorrect answer to correct one.
Text in bold denotes user prompts.
13

---

Published as a conference paper at ICLR 2024
Can you solve the following math problem? Terry eats yogurts day. They are currently on sale at yogurts for $0.00. How much does he spend on yogurt over 30 days? Explain your reasoning. Your final answer should be single numerical number, in the form (boxed(answer), at the end of your response.
First, we need to calculate how many sets of 4 yogurts Terry buys in 30 days. Since Terry eats 2 yogurts day, he will need 2/4 0.5 sets of 4 yogurts per day, Next, multiply the number of sets by the price each set to calculate how much Terry spends per day. 8ach set of yogurts costs $5.00, so Terry spends 0.5 $5.00 yogurt per day, Finally, we multiply the amount Terry spends per day by the number of days to find the tetal amount he spends on yogurt over 30 days. Terry spends $2.50 per day for 30 days, so spends $2.50 30 (x<2.5x30=75.000 00te:75.00>7..0000 00 on yogurt over 30 days. So, Terry spends $1boxed on yogurt over 30 days. (Correct)
5x500.5452..5000 50>>2.50 on
Review your previous answer and find problems with your answer.
Upon reviewing my previous answer, realized that I made a mistake cal ealculating Terry's 's spending on yegurt over 30 days. incorrectly stated that Terry spends $2.50 per day for 30 days, resulting in a total of $75.00. However, since Terry eats yogurts day and the yogurts sold in sets of 4 for $5.00, Terry would actually need to buy 1 set of yogurts to last for 2 days. Therefcre, his spending on yogurt per set would be $5.00 for 2 days, or $5.00/2 $2.50 per day. To calculate the total amount spent
yogurt over 30 days, we would multiply the amount spent per day ($2.50) by the number of sets needed in 30 days (which is 30 divided by days per set). Therefore, the correct calculation would be $2.50 * (30/2) $2.50 15 $37.50. Apologies for the mistake. The correct answer is Terry spends 91bcxed[37.50] on yogurt over 30 days.
Based on the problems you found, improve your answer. Please reiterate your answer, with your final answer a single numerical number, in the form (boxed(answer).
Apologies for the error in my previcus response. Upon reviewing the problem, here the correct answer: Since Terry eats 2 yogurts day and the yogurts are sold in sets of 4 for $5.00, Terry needs to buy set of 4 yogurts to last for 2 days. Therefore, Terry's spending on yogurt per $5.00/2 = $2.50 per day. To calculate the total amount spent on yogurt over days, we multiply the amount spent per day ($2.50) by the number of sets needed Terry needs 30/2 = 15 sets of yegurts over 30 days. Multiplying $2.50 by 15, we find that Terry spends tctal of $2.50 x 15 = $37.50 on yogurt is Terry spends $ 37.500
set is $5.00 for 2 days,
days. In this case,
over 30 days. Therefore, the final
on yogurt over 30 days. (Incorrect)
Figure 4: Example on GSM8K where self-correction changes a correct answer to an incorrect one.
14

---

Published as a conference paper at ICLR 2024
Can you solve the following math problem? Toulouse has twice many sheep Charleston. Charleston many sheep do Toulouse, Charleston, and Seattle have together if Seattle has 20 sheep? Explain your reasoning. Your final answer should be a single numerical number, in the form |boxed(answer), at the end of your response.
times as many sheep as Seattle. How
Let's start by figuring out how many sheep Charleston has. We are given that Charleston has 4 times as many sheep as Seattle, so can calculate this by multiplying Seattle's number 80 sheep. Next, we are told that Tculouse has twice many sheep as Charleston. Therefore, we can find Toulouse's number of sheep by multiplying Charleston's number of sheep by 2: Toulouse 2 80 160 sheep. Finally, to find the total number of sheep, we add up the nunber of sheep for each city: Total Seattle Charleston Toulouse 20 160 260 sheep. Therefore, Teulcuse, Charleston, and Seattle have a total of 260 sheep. Answer: \boxed(260). (Correct)
sheep by Charleston
20
Review your previous answer and find problems with your
apologize for the mi mistake in my previous response. let's go through the problem again and find the correct answer. we are given that Charleston has 4 times as many sheep Seattle. If Seattle has 20 sheep, then told
20 80 sheep. Next, Toulouse that Toulouse has twice as many sheep as Charleston. Therefore, would have: Toulouse = 2 * 80 160 sheep. To find the total number of sheep, we add the number of sheep for each city: Total Seattle Charleston Toulouse 20 80 + 160 260 sheep. After reviewing ny previous answer, I have determined that there are no problems with ny updated response. The correct answer 260 sheep.
Charleston would have: Charlesten
Based the problems you found, improve your answer. Please reiterate your answer, with your final answer single numerical number, in the form (boxed(answer).
Apologies for the confusion. Let me provide an improved answer. We are given that Charleston has times many sheep Seattle, and Seattle has 20 sheep. So, Charleston would have the information, Toulouse has twice as many sheep as Charleston.
sheep. According to
Therefore, Toulouse would have 2 80 160 sheep. To determine the total number of sheep, seattle Charleston + Toulouse = 20 80 160 260 sheep. After revicwing the previous answer, the correct final answer \boxed(260). (Correct)
add up the number of sheep for each city: Total
Figure 5: Example on GSM8K where self-correction does not change the answer.
15

---

Published as a conference paper at ICLR 2024
Q: fencing thrust with sharp sword towards a person would result in what?
Answer Choices: (A) injury (B) small cuts (C) fever (D) competition (E) puncture wound.
Explain your reasoning. You must choose only one option from A to E. Your final answer should be single letter from A to E, in the form (answer), at the end your response.
A: A fencing thrust with a sharp swerd towards a persen would result in puncture wound. This is because a fencing thrust involves forcefully stabbing with sword, which narrow wound. Puncture wounds can be very dangerous as they can damage internal organs and cause bleeding. Therefore, the most appropriate answer is (E) puncture wound. (Correct)
penetrate the skin and create deep,
Review your previous
and find problems with your answer.
Upon reviewing my previous answer, I realize that there are some problem ems with it. While fencing thrust with sharp sword can potentially result in puncture wound, it is important match, the swords used are typically blunted or have safety tips, designed to minimize the risk of injury. Therefore, the likelihood of a puncture wound from a fencing thrust with sharp sword in a regulated fencing context is quite low. more appropriate and accurate answer would (D) competition,
note that in controlled fencing
fencing thrust with sharp sword is typically performed in the context of fencing match or competition, rather than with the intention to cause injury.
Based the problems you found, improve your answer. You must choose only one option from A to E. Please reiterate your answer, with your final answer a single letter from to E, in the form (answer).
Based the problems with my previ answer, apologize for the incorrect response. In controlled fencing match with sharp sword, fencing thrust is likely to result in (D) competition, rather than injury or puncture wound. Therefore, the correct final answer is (D) competition. (Incorrect)
Figure 6: Example CommonSenseQA where self-correction changes correct answer in- correct one.
16

---

Published as a conference paper at ICLR 2024
Concepts:
[ 'oottge', 'motion', 'ruin', 'tilt', 'window')
time lapse footage with tilt up motion of the sun streaking through window of ruin
Sentence:
###
Concepts:
['cause', 'hate', 'hut', 'local', 'love']
Sentence: beach huts the island have caused some controversy some locals love them others hate them
[8 more in-context examples]
###
Concepts:
['vest', 'snow', 'remove', 'silhouette', 'dunk', 'step', 'item', 'toddler', 'plaster', 'wall', 'piece', 'tool', 'circle', 'pull', 'wrap', 'hedphhon', 'deck', 'suit', 'press', 'mother', 'pit', 'cupcake', 'priest', 'rink', 'counte'', 'rest', 'piano', 'wine', 'serve', 'winter']
Sentence: toddler wearing vest and boots steps onto the icy deck, removes circle-shaped piece of plaster from the wall, and pulls out a tool from their winter suit to wrap their r head around headphone.
Figure 7: Example output for Constrained Generation using the prompt from Madaan et al. 2023 The constraint that the output needs to contain all the concepts is not explicitly mentioned in the prompt for initial response generation. However, in the feedback & refine prompts, the constraint is implied by "what concepts from the concept list are missing from the sentence?" (refer to Figures 25 and 26 of Madaan al. (2023) for the feedback & refine prompts).
Concepts:
('vest', 'snow', 'remove', 'silhouette', 'dunk', 'step', 'item', 'toddler', 'plaster', 'wall', 'piece', 'tool', 'circle', 'pull', 'wrap', 'headphone', 'deck', 'suit', 'press', 'mother', 'pit', "cupcake', 'priest', 'rink', 'counter', 'rest', 'piano', 'wine', 'serve', "winter']
Write a reasonable paragraph that includes *ALL* of the above concepts.
In the winter wonderland, mother dressed in warm vest watched her toddler play in the snow. As the child attempted remove the snow from the ground, their silhouette created an enchanting image against the white backdrop. Nearby, a group of friends gathered around & basketball hoop, taking turns to dunk the ball with skillful steps. One of then, holding an item resembling that had been damaged. Meanwhile, on the other side of the park, a cirele of chilaren pulled their hats tightly to wrap around their heads, protecting their ears from the cold wind as they listened to music through their colorful headphones. On the deck of nearby house, sleek suit pressed a button, causing hidden mechanism to reveal grand pianc. As the music filled the air, the host served glasses of wine to the guests, creating a warm and inviting atmosphere in the winter evening. In the distance, movements mirroring the smoothness of his words during sermon. At the counter of bakery, a customer crdered a cupcake and watched as the baker skillfully decorated it with a winter-themed design. Not far away, group of friends gathered around a pit, roasting marshrallows and enjoying the warmth of the fire. The scene painted picture of joy, friendship, and the beauty of winter.
teel, used it to plaster a piece of the wall
priest skated gracefully in an ice rink, his
Figure 8: Example output for Constrained Generation using prompt casually written by the authors.
17


Reviews:
[{"review_id": "TUaMWym9fv", "reviewer": "Reviewer_yEtd", "summary": "LLMs apparently have the ability to self-correct, as evidenced by previous publications. In the current article, which acts partly as a survey, the authors investigate how well LLMs can actually self-correct intrinsically and report nuanced, mixed results in terms of LLMs ability to perform such a feat.", "questions": "- I don't understand the second paragraph from section \"3.1.3 REFLECTION\". How does this guessing approach work? It seems that cannot be used for non-multiple-choice type answers, like in the case of the GSM8K dataset?          \nI cannot follow your footnote: \"For GSM8K, a similar random baseline might not exist, but the underlying rationale remains the same. Additionally, we can design a baseline, for example, by generating a random number each time. After a significant number of rounds, it may reach the correct answer, but such a kind of improvement is apparently not meaningful. A more direct justification is: If we already know the answer, why do we need to do this?\"      \n- \"Per the discussions in Section 3.1.3, since the idea that LLMs can self-correct their reasoning is not supported by the evidence so far, we turn our focus to the results in the intrinsic self-correction.\" I wasn't quite able to follow why section 3.1.3 shows this; Table 1 actually seems to show that self-reasoning is well-supported by evidence?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "The paper tackles a very important topic, has a good literature review section, and uses well-known and trusted datasets to investigate self-correction abilities. In particular, I appreciated the distinction between intrinsic self-correction and self-correction that leverages information from humans or training examples.", "weaknesses": "- Only a small set of questions (200) is used on GPT4, the remaining ones apply only to ChatGPT  \n- In terms of reasoning, there are much more challenging datasets out there\n- I found the presentation somewhat confusing since there wasn't a clear description of their methodology (e.g., were all self-correction prompts formulated as the examples in Figure 2, or did variations exist?).\n- Also, the wording is unfortunately often trendy rather than clear: From the conclusion: \"while LLMs represent a groundbreaking step forward in the realm of AI and language generation, their self-correction capabilities, particularly in reasoning, are still nascent\". \"Nascent\" can mean anything from the self-correction capabilities being \"not present\", \"weak\" \"promising\" etc. Please try to use more specific wording.\n- The paper is confusing to read, because there are somewhat contradictory statements: In the beginning the authors emphasize that they want to investigate *intrinsic* self-correction abilities. Then, on page 3, they state: \"With this in mind, we center our investigation on a pivotal query: Can large language models self-correct their reasoning?\" which seems to me to not imply intrinsic self-correction, but any type of self-correction. This unclarity seems to pervade the remainder to the paper.      \nE.g., in section 7 the authors state \" Some existing literature may inadvertently contribute to this confusion, either by\nrelegating crucial details about label usage to less prominent sections or by failing to clarify that their\ndesigned self-correction strategies actually incorporate external feedback. Our intention in this paper is to amplify these concerns and offer a comprehensive overview of the state of “self-correction” in LLMs\".      \nThis implies that the current paper was more like a survey, rather than establishing new results.    \nFurthermore, the title suggests that self-correction does not work, although the paper actually does not really argue this and in section 7 the authors remark \"The title, “Large Language Models Cannot Self-Correct Reasoning Yet”, is not an outright dismissal of self-correction techniques\". Generally, it would be good to use a title that is completely representative of the paper. (Perhaps add \"intrinsic\" to the title?)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b2CsLfx0dr", "reviewer": "Reviewer_No3p", "summary": "This work investigates prior reports that LLMs can self-correct their own responses when prompted to do so. A critical distinction is made between self-correction with and without external feedback (termed intrinsic self-correction). The results show that given the same self-correction prompt used in prior work on particular benchmark datasets, LLMs often do not succeed in self-correcting their responses without external feedback.", "questions": "Section 3.2 says “Intuitive Explanation. If the model is well-aligned and paired with a thoughtfully designed initial prompt, the initial response should already be optimal”  But why should this be intuitive or expected? Wouldn’t similar reasoning conclude that the value of chain-of-thought prompting is itself unintuitive?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Self-correction of today’s LLMs is a highly significant topic. The paper clearly points out the crucial distinction between self-correction with and without feedback, and sheds light on the latter case (intrinsic self-correction). The usage of oracle labels to terminate self-critique is also examined.\n\nThe paper’s organization and writing quality is uniformly high, making it a pleasure to read.", "weaknesses": "The most serious weakness of the paper is its misleading title, which baldly asserts a claim unsupported by the analysis and results. The words “cannot” and “yet” imply that even today’s most capable LLMs (GPT-4) obtain zero benefit from self-correction in nearly all cases. The abstract quickly tones down the claim by saying “our research indicates that LLMs struggle to self-correct their responses without external feedback”, but even that statement goes beyond what is actually demonstrated by the experiments. A more properly measured title for this work would be “Reexamining the Ability of Large Language Models to Self-Correct”. This is more like statements that appear later in the paper:  “we provide insights into the nuances of LLMs’ self-correction capabilities”, and “their self-correction capabilities, particularly in reasoning, are still nascent.”\n\nThe second major problem is the work’s heavy reliance on a single, loaded prompt:\n“Review your previous answer and find problems with your answer”\n\nPractitioners in the rapidly moving field of prompt engineering recognize this as a highly leading prompt, essentially telling the LLM that problems do exist in the previous answer. This typically causes the LLM to find problems that aren’t actually present. As the paper says at one point:  “careful consideration of prompt design is essential”.\n\nHere’s a longer list of important factors used routinely in prompt engineering that would be needed for a proper study of LLM self-correction:\n\n- Focus on GPT-4, since its capabilities are known to be significantly greater than those of GPT-3.5. This is reflected in Table 3. \n\n- Include diagrams like those in Figure 1 for GPT-4, not just for GPT-3.5.\n\n- Evaluate a set of reasonable, unbiased self-correction prompts. For instance: “Assume that this answer could be either correct or incorrect. Review the answer carefully and report any serious problems you find.”\n\n- Focus on the zero-temperature setting, since that’s far less prone to spurious hallucination than 1. \n\nIn discussing the option of trying other self-correction prompts (“Such a search essentially leverages feedback from humans or training examples.”), the paper conflates feedback received on a per-problem basis (as when using an oracle), with feedback received from the results of multiple self-correction prompts across entire datasets. The former does indeed go beyond the definition of intrinsic self-correction, but the latter is merely hard work. \n\nThe paper says that “Our main objective is to encourage a more critical examination of self-correction experiments.” But readers expect this paper to be a critical examination of that nature, not just a call for critical examination.\n\nThe paper also states that “in the reasoning tasks studied in this paper, we did not observe any improvement through self-correction.” That’s true enough, but as pointed out above, the study was not carried far enough to shed much light on the general question of how reliably LLMs can self-correct.\n\n**Post-rebuttal Comments**\n\nI commend the authors for performing the additional experiments reported in Appendix B.2, using GPT-4 with an unbiased feedback prompt and zero temperature. These results on these two datasets are interesting, so I have raised my assessment of the paper’s contribution from 2 to 3. \n\nI still view these limited results as insufficient to support the broad claim made by the paper’s title:  “Large Language Models Cannot Self-Correct Reasoning Yet”. The authors argue that the title’s claim is restricted to “reasoning” tasks, but this is not much of a restriction at all. \n\nRegarding other feedback prompts used in the experiments, the ones in Appendix B.1 only apply to GPT-3.5, which is widely known to not be good at self-critique. The authors imply that the feedback prompts from prior works are different, but Appendix A shows only the single, loaded prompt: “Review your previous answer and find problems with your answer”\n\nFor these reasons, I still view the work as fundamentally unsound, in the sense that the limited findings do not justify the headline-grabbing title of the paper.\n\n**Additional Post-rebuttal Comments**\n\n\n*we do not fully understand the argument that \"The authors argue that the title’s claim is restricted to “reasoning” tasks, but this is not much of a restriction at all.\"*\n\n\nLLMs are strong at memorization, but struggle with many kinds of reasoning. So while self-critique can be applied to both memorization and reasoning problems, application to reasoning is of greater interest and is being intensely studied. For this reason, restricting the consideration of self-critique to reasoning is not much of a restriction at all. \n\n\n\n*we will change the title to **\"Reexamining the Ability of Large Language Models to Self-Correct Reasoning\"** in the final version.*\n\n\nThis title would be in line with the experiments. So on the assumption that this will indeed be the title, I'm raising my rating from 3 to 6.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3vvEHgZfcp", "reviewer": "Reviewer_N1pE", "summary": "The paper studies *intrinsic* self-correctness in LLMs in the context of reasoning, where no external feedback is provided to the language model i.e., just simply asking the model to detect a mistake in its output and fix it. Through experiments over three reasoning tasks (GSM8K, Commonsense QA, and HotpotQA), the paper does the following. First, it argues that what is currently referred to in the literature as \"self-correctness\" where external feedback is provided (e.g., whether the final answer is correct) is not practical and we should focus on settings where we do not know the answer. Second, self-consistency is a strong baseline and similar approaches such as multi-agent debate methods should be considered an instance of voting rather than self-correction methods. Third, to assess whether LLMs actually have the capacity for self-correction, the authors emphasize the importance of designing a good initial (pre-hoc) prompt that performs well as opposed to using a suboptimal initial prompt, where providing additional information in the feedback prompt can be useful.  Overall, the paper argues that current LLMs fall short when prompted to self-correct their reasoning and when no additional information is provided in the feedback prompt.", "questions": "* Have you tried combining Multi-agent debating with self-consistency? \n* How much effort did you invest into finding the self-correct prompt you used \"Review your previous answer and find problems with your answer\"? Can't different variations of this same prompt lead to different results?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* The paper studies an important direction that is now taking over the LLM scene and brings a fresh perspective on how good SoTA LLMs are at detecting their own errors. \n* Focusing on *intrinsic self-correction* is much needed in the current \"sea\" of self-correction papers.\n* The experimental design is sound. I liked the random guessing baseline with Commonsense QA.  \n* I have to say I enjoyed reading the paper: the flow is natural, the writing is good, and most of the arguments are intuitive and make sense. \n\nOverall the community would certainly benefit from this paper gaining wider visibility.", "weaknesses": "* I find the explanation in section 3.2.1—why post-hoc prompting can lead the model to go from a correct to an incorrect answer—unsatisfying. We know that the feedback prompt is changing the model output somehow. The question is *why?* I suggest providing more intuition here.\n* The paper discusses the issue but does not provide any hint at a potential solution. I understand this is not the point of the paper but hinting at potential directions to improve intrinsic self-correctness could make the paper even more valuable. \n* The authors focus only on ChatGPT and GPT-4. I think the community could benefit from seeing that the results discussed generalize to other open-source LLMs such as LLaMA", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r40GN1Wttt", "reviewer": "Reviewer_oT3c", "summary": "Large Language Models (LLMs) have been increasingly capable. However, they still make many mistakes. Recent work has explored the idea of “self-correction” where LLMs refine their responses based on feedback to their previous outputs. This paper critically examines the role and efficacy of self-correction within LLMs. Central to the investigation is the definition of intrinsic self-correction, whereby an LLM attempts to correct its initial responses based solely on its inherent capabilities, with no external feedback. The paper finds that LLMs struggle to self-correct their responses for reasoning tasks without external feedback, and at times, their performance might even degrade post self-correction.", "questions": "The paper exposes an intriguing problem in prior work on self-correction of LLMs that shows, to the contrary that LLMs can not self-correct reasoning yet.\n\nHowever, it is still not clear how we should think about all the techniques to improve reasoning without external feedback. Does breakdown a problem into sub-problem and do step-wise verification paired with self-consistency considered self-correction? It would be very helpful to put all these techniques into perspective.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. Contrary to prior results, the paper finds the self-correct methods in prior research such as Kim et al. (2023); Shinn et al. (2023) make use of oracle labels to guide the self-correction process. \n\n2. For self-correction through multi-agent debate (Du et al., 2023; Liang et al., 2023) to improve reasoning where multiple instances of an LLM critique each other’s responses, the paper's results reveal that its efficacy is no better than self-consistency when considering an equivalent number of responses, highlighting the limitations of such an approach.", "weaknesses": "1. Intrinsic self-correction, defined as the model endeavors to rectify its initial responses based solely on its inherent capabilities without the crutch of external feedback, is not very clear to me. Does recall examples from parameter knowledge (see paper below) considered intrinsic self-correction? \nLarge Language Models as Analogical Reasoners\nhttps://arxiv.org/pdf/2310.01714\n\n2. It is not clear which prior papers have the various problems exposed. It would be very helpful to put them in a table. Furthermore, please provide details on where prior methods rely on oracle labels, specific problems on poorly constructed pre-prompts, specific benchmark results that are wrong. For example, I do not seem to locate which part Shinn et al. (2023) have the problems.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)