Large Language Models have Intrinsic Self-Correction Ability

Authors: Dancheng Liu, Amir Nassereldine, Chenhui Xu, Ziming Yang, Yuting Hu, Jiajie Li, Utkarsh Kumar, Changjae Lee, Ruiyang Qin, Yiyu Shi, Jinjun Xiong
Primary area: foundation or frontier models, including LLMs
Keywords: large language model, self correction, prompt engineering, foundation model

Abstract:
Large language models (LLMs) have attracted significant attention for their exceptional abilities in various natural language processing tasks, but they suffer from hallucinations that will cause performance degradation. One promising solution to improve the LLMs' performance is to ask LLMs to revise their answer after generation, a technique known as self-correction. Among the two types of self-correction, intrinsic self-correction is considered a promising direction because it does not utilize external knowledge. However, recent works doubt the validity of LLM's ability to conduct intrinsic self-correction. In this paper, we present a novel perspective on the intrinsic self-correction capabilities of LLMs through theoretical analyses and empirical experiments. In addition, we identify two critical factors for successful self-correction: zero temperature and fair prompts. Leveraging these factors, we demonstrate that intrinsic self-correction ability is exhibited across multiple existing LLMs. Our findings offer insights into the fundamental theories underlying the self-correction behavior of LLMs and remark on the importance of unbiased prompts and zero temperature settings in harnessing their full potential.

Decision: reject
Reviewer scores: 10, 3, 5, 3

Reviews and discussion:
<thread>
creation date:
1730080100599

thread_type:
official_review

<thread_object>
creator: reviewer_shev

<content>
summary:
- **Context: **The paper studies intrinsic self-correction which is LLMs ability to improve **their own** prior decisions (in contrast to **extrinsic** self-correction that uses external feedback of the environment). There are several prior works that cast doubt on the effectiveness of intrinsic self-correction. 

 - **This presented paper:** This work argues that LLMs are indeed capable of intrinsic self-correction capabilities. The paper makes this argument through theoretical analyses and empirical experiments. The paper points out two critical factors for successful self-correction: (1) 
zero temperature and (2) fair prompts.

strengths:
- The problem of self-improvement is of great interest and importance for the research community, particularly given recent debates about whether LLMs possess genuine self-correction abilities.
- The issue of temperature hasn't been studied enough in the context of self-improvement. The authors' attempt to systematically analyze this parameter's influence is valuable.
- The idea to break self-improvement into multiple stages like chain-of-thought and study each component separately (as shown in Section 2.2) is interesting and potentially useful for understanding the mechanism of self-correction.

weaknesses:
(1)  Theoretical Foundation Issues: The paper claims to provide theoretical proofs, but the analytical process lacks rigor. For example:
- Eq3 is just wrong, p(A|\tau) should be p(A, R | \tau). Basically, p(X, Y | Z) = p(X | Y, Z) x p(Y|Z)  (minor typo: \tau should be \tau_1). For the same reason, Eq 4 is also incorrect. 
 - Sec 4.1 opens by “the randomness in decision-making diminishes as the temperature decreases”; isn’t this just a trivial statement? In Equation 8, the authors take temperature to extremes to show that answers have more likelihood to flip (which is obvious even without all these equations). But more importantly, this finding has no meaningful connection to self-improvement. Even if it does, it does connect to self-improvement, it is somewhat unrealistic since practical applications rarely use such extreme temperatures.
 - The analysis in Section 5.1 makes an unreasonable assumption that biased prompts cause random answer flipping. This assumption isn't justified, and the subsequent theoretical analysis adds little insight to our understanding.

(2) Overstatement of Results: The authors frequently jump to conclusions without sufficient evidence:
 - In line 194, they cite Table 1 as evidence of "improvement in accuracy after SC", but the difference is merely 0.08% (75.92% to 76.00%).
 - While Table 2 shows statistical significance across multiple datasets, the actual improvements are typically within 1%. This hardly constitutes meaningful improvement.
- In line 311, they make a definitive claim that "GPT-3.5 follows Order 1 of Eq. (5), whereas GPT4 follows Order 2" without providing quantitative evidence for this classification.

(3) Limited Practical Relevance of Temperature Analysis:
 - Figure 2 shows that temperature effects in the practical range (0-0.7) are minimal.
 - The significant effects only appear at temperature values like 1.4, which are rarely used in practice for self-improvement tasks.

-------------

Presentation Issues:
 - Figure 1 would benefit from including key notation (τ1, R1, D, etc.) referenced throughout the paper.
 - Critical prompt examples are relegated to the appendix when they should be in the main text for better understanding of the methodology.
- In Fig1 caption, make it clear what makes your prompt “unbiased”. One might spend a long time reading the prompt and not understanding what they should pay attention to. 

> Feng et al. (2023) argues that CoT increases the effective depth of the circuit by letting the generated outputs repeatedly loop back to the input.

This phrasing is extremely misleading. The arguments of Feng et al. is that with longer generations, Transformers (for **SOME choice of weights**; not necessarily the pre-trained weights) have more expressive capacity to model more complex problems (circuits). However, this argument does not automatically mean that the existing pre-trained LLMs actually do this, which is what is implied by your phrasing. 

Missing references:

There is other works that casts doubt on the abilities of LLMs to self-improve that are missing in your work: 
- Self-[In]Correct: LLMs Struggle with Discriminating Self-Generated Responses, 2024
- The Google paper: LLMs cannot find reasoning errors, but can correct them given the error location, 2024
- ... (there are a few others that I might be missing)
Given that your work directly rebuts the above works, please include appropriate discission for each one (not just Huang et al.) 

Here are several other relevant works that are not cited in your work: 
- Crystal: Introspective reasoners reinforced with self-feedback, 2023
- Generating sequences by learning to self-correct, 2023
- The self-Instruct paper, 2023

Your work is also related to the growing literature on inference-scaling. As the prior work has shown, inference-scaling works. Naturally, one can extend these to data generation, as you do in your work.

questions:
N/A
</content>
</thread_object>

</thread>
<thread>
creation date:
1730605379970

thread_type:
official_review

<thread_object>
creator: reviewer_6zbp

<content>
summary:
The paper investigates the capacity of LLMs to perform self-correction without relying on external sources of information, namely “intrinsic self-correction.” The authors propose that this self-correction process is valuable for enhancing LLM performance and has similar principles as chain-of-thought and self-verification. The study identifies two key factors that influence self-correction: using an unbiased prompt to avoid influencing the model’s initial answer and setting a zero temperature to reduce randomness. Through theoretical analysis and empirical experiments, the paper demonstrates that intrinsic self-correction is achievable across multiple LLMs, providing a foundation for further research in leveraging this capability.

strengths:
1. **Insightful Analysis of Self-Correction Mechanisms**: The paper offers a novel perspective by comparing intrinsic self-correction to chain-of-thought and self-verification techniques. This theoretical framing provides a solid foundation for understanding the mechanisms that enable LLMs to self-correct.
2. **Practical Recommendations for Enhanced Model Performance**: By identifying unbiased prompts and zero temperature as key factors for effective self-correction, the authors present valuable insights that are directly applicable to real-world LLM deployment.

weaknesses:
1. **Limited Performance Improvement Even in Ideal Conditions**: The performance gain from self-correction is relatively small, even under the ideal settings proposed (unbiased prompts and zero temperature). As is shown in Table 1 & 2, under many circumstances, the performance gain is less than 2%. This marginal improvement raises questions about the practical impact of the self-correction process and the techniques provided in the paper.
2. **Lack of Significant Theory and Insight**: The paper creatively related the performance of self-correction to chain-of-thought prompting and temperature, but it fails to introduce solid theoretical backbones or experimental insights. 
3. **Limited Model Sizes and Architectures**: The paper only conducted experiments on two API-based models and two open-source models, and it fails to conduct ablation on model sizes (using the same series of models with different sizes) and other model architectures.   This limitation restricts the generalizability of the findings across a broader range of model types.

questions:
1. What are the novel contributions of this paper over previous works on intrinsic self-correction? Given that the paper investigates the topic related to prior work on self-correction (e.g., [Pan et al., 2023](https://arxiv.org/abs/2308.03188)), it would be helpful to clarify how this research extends beyond or differs from earlier findings.
2. Why is some certain experimental results not obviously show the advantage of self-correction? Is it from the method itself, or there are other failure modes?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730756498432

thread_type:
official_review

<thread_object>
creator: reviewer_3p3t

<content>
summary:
This paper investigates the sefl-correction ability of LLMs. In particular, they investigate two factors, i.e. temperature and the bias of prompt. The authors conduct experiments on 6 benchmarks using 4 LLMs. However, the paper is very difficult to read, with less rigour proof. The selection of models is not wide, resulting in less convincing empirical results. Also, I do not see a systematic and coherent motivation to investigate temperature and the bias of the prompt. It is somehow more like assembling these two factors in one paper.

**After the rebuttal**: the definitions of concepts in this paper are very unclear, with many left unexplained. Most proofs are not valid and rigorous. Many sections are difficult to understand. While I defer to some reviewer's appreciation, I do not believe this paper merits an ICLR paper standard.

strengths:
May be interesting to future research.

weaknesses:
1. The paper is very difficult to read. In particular, the equation part.  Proof 2.1 is very confusing and many explanations are not given, and I cannot understand it with 2 or 3 times reading. For example, in Line 869, "which we denote as correct(A ∈ Q) = λ > k1". What is λ? The portion of correctly answered questions/all questions? And I do not think the probability of that hallucination randomly changing the answer is equal: tokens are of different importance to a sentence (answer), thus, some hallucinated tokens may not change the final answer (this is also related to how the authors define hallucination, but I do not see any justification here). Also, what is h? The portion of hallucinated responses? I also do not understand what the authors mean by h∗(1−λ)/(k−1) answers will be changed from incorrect to correct. Also, why h∗λ> h∗(1−λ)/(k-1)? Line 283, "(a1/T −(1−α)1/T)" should be "α" not "a".
2. The selection of models is limited. More open sourced model should be included for a more comprehensive evaluation.

questions:
Please see above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731129809018

thread_type:
official_review

<thread_object>
creator: reviewer_mqrf

<content>
summary:
The paper claims to demonstrate the intrinsic self-correction capabilities of LLMs. Motivated from examples where bias in the prompts can interfere with the correction (in particular, by implying there’s a problem with the initial answer), they ask *if they can correct a wrong answer, why can’t an LM get it right the first time?*

They claim that they can’t due to inherent hallucinations but can perform intrinsic SC. They provide some theoretical analysis to demonstrate this, alongside empirical investigations into the impact of temperature, prompt structure, etc.

According to their theoretical analysis, using CoT during the initial answer generation produces a rationale alongside it, which can be attended over in the self-correction step to produce a candidate decision before reasoning for the decision has to be written. However, without the initial chain, generating this decision is effectively relying on another zero-shot correction in line with how the initial answer was (possibly erroneously) generated. In other words, systems where answers are generated by CoT are more robust to self-correction error introduction than ones that don’t use it, and proper self-correction on non-CoT answers essentially just brings performance back up to CoT levels.

This analysis leads to predictions such as hallucination rate increasing with higher temperature leading to worse performance of self-correction. This is confirmed. They also make some claims that feel more dubious to me, like “LLMs are naturally underperforming their intrinsic ability” which feels a little underdetermined to me.

**Edit**: I am happy with this paper overall, and will defer somewhat to the other reviewers on gripes about the theoretical arguements. That being said, I appreciate the authors' efforts to expand the scope of their experiments in the rebuttal.

strengths:
Solid theoretical argument that runs throughout the paper. Though this isn’t my core area it was quite easy to understand and most claims were plausible.

The experiments are well connected to the theoretical analysis. Neither part feels like an afterthought, which is a bit rare for LLM reasoning papers :)

The discovery of best practices for prompt sets can both be directly applied in engineering work and used to inspire future analysis.

weaknesses:
Few, mainly the more philosophical claims I complained about in my summary.

questions:
I wasn’t the most well-positioned to assess the novelty of this work. I took a look through Pan et al. 2024’s survey “Automatically Correcting Large Language Models” paper to help check this, might be useful to reference for other readers?
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper studies intrinsic self-correction in large language models (LLMs) – their ability to revise previous answers without external input. looking at how temperature and the bias of prompts affect their performance. The paper’s core claim is that with carefully chosen “fair” prompts and zero temperature at sampling, LLMs can correct some of their initial mistakes. The paper includes theoretical analysis and empirical experiments conducted across multiple benchmarks and LLMs. The main conclusion is that intrinsic self-correction can provide small but consistent performance gains.

After evaluating the reviews and the authors’ subsequent clarifications, I am inclined to side with Reviewers 3P3T and shEV, who both raise substantial issues regarding clarity, rigor, and limited practical significance. I agree with Reviewer shEV that temperature analysis does not provided a meaningful contribution and also that the reported gains in typical temperature range are minimal. 

While the authors make a good-faith effort to address questions, multiple reviewers remain unconvinced that the paper demonstrates the rigor and impact required for acceptance.