Embedding Self-Correction as an Inherent Ability in Large Language Models for Enhanced Mathematical Reasoning

Authors: Kuofeng Gao, Huanqia Cai, Qingyao Shuai, Dihong Gong, Zhifeng Li
Primary area: foundation or frontier models, including LLMs
Keywords: Large Language Models, Mathematical Reasoning, Self-correction

Abstract:
Accurate mathematical reasoning with Large Language Models (LLMs) is crucial in revolutionizing domains that heavily rely on such reasoning. However, LLMs often encounter difficulties in certain aspects of mathematical reasoning, leading to flawed reasoning and erroneous results. To mitigate these issues, we introduce a novel mechanism, the Chain of Self-Correction (CoSC), specifically designed to embed self-correction as an inherent ability in LLMs, enabling them to validate and rectify their own results. The CoSC mechanism operates through a sequence of self-correction stages. In each stage, the LLMs generate a program to address a given problem, execute this program using program-based tools to obtain an output, subsequently verify this output. Based on the verification, the LLMs either proceed to the next correction stage or finalize the answer. This iterative self-correction process allows the LLMs to refine its reasoning steps and improve the accuracy of its mathematical reasoning. To enable the CoSC mechanism at a low cost, we employ a two-phase finetuning approach. In the first phase, the LLMs are trained with a relatively small volume of seeding data generated from GPT-4, establishing an initial CoSC capability. In the second phase, the CoSC capability is further enhanced by training with a larger volume of self-generated data using the trained model in the first phase, without relying on the paid GPT-4. Our comprehensive experiments demonstrate that CoSC significantly improves performance on traditional mathematical datasets among existing open-source LLMs. Notably, our CoSC-Code-34B model achieved a 53.5\% score on MATH, the most challenging mathematical reasoning dataset in the public domain, surpassing the performance of well-established models such as ChatGPT, GPT-4, and even multi-modal LLMs like GPT-4V, Gemini-1.0 Pro, and Gemini-1.0 Ultra. It's important to note that, unlike these proprietary models, our CoSC performs inference in a zero-shot manner, without the need for demonstrations. The code and data for this work will be released once this paper is accepted.

Decision: reject
Reviewer scores: 6, 5, 5, 6

Reviews and discussion:
<thread>
creation date:
1729779520008

thread_type:
official_review

<thread_object>
creator: reviewer_grdy

<content>
summary:
This paper proposes a technique for improving mathematical reasoning with large language models which the authors call the  "Chain of Self-Correction (CoSC)". CoSC is designed to incorporate self-correction as an inherent ability in LLMs to iteratively refine their reasoning through multiple rounds of program generation, execution, and verification. The approach uses a two-phase fine-tuning process: first, training with a small dataset generated by GPT-4, and then self-enhancing with self-generated data, thereby reducing reliance on expensive models. On standard math reasoning datasets MATH and GSM8, the method outperforms a wide range of open source techniques and models that are tested in this work, and also performs better than some proprietary LLMs.

strengths:
1. Interesting and cost-efficient two-phase fine-tuning approach. Using a two-phase approach (starting with seeding data from GPT-4 and then using self-generated data) is an interesting and effective way to reduce reliance on expensive models. The authors also include an ablation study that shows the significant improvements with both initial GPT4 and self-enhanced finetuning. 
2. Evaluation is very broad and shows core value. The paper shows a very extensive evaluation with many proprietary and open source models as baselines, which is admirable. Their CoSC approach is also evaluated on three different sizes of LLMs up to 35B, where the approach consistently performs better than all other open source models on all sizes. Ablation studies also show the gains from multiple rounds of correction to show the value of the concept of chain of self correction, and further ablation studies are included in appendices.  
3. Presentation and details. Well-structured and well-written paper. The problem is motivated well, and the technique, implementation and evaluation is clearly explained with details, and prompts and more details are provided in extensive appendices.

weaknesses:
1. Novelty is not clear. In particular, I am not sure how exactly the CoSC technique is different from ToRA (Gou et al., 2023b). ToRA seems to also use the interleaved approach of program-generation, execution and rationales in multiple rounds (similar to the chain of correction you have here). One thing that seems different with CoSC is that you have explicit verification and conclusion steps and verification checks separately for question and output correctness. But those seem a bit like relatively smaller optimizations to guide the model to do more explicit reasoning. You also classify ToRA as under prompting approaches as opposed to fine-tuning in the related work discussion, but from reading their work it seems they are also fine tuning the models? (and in fact using GPT4 for initial data generation and also using two steps in training their models). But I am not completely sure of this - please clarify what exactly the significant differences in technique there are with ToRA. 

2. Evaluation results seem not to show very drastic gains. The overall improvement over the best open source model (ToRA) is 1.8% - 2.6% for all sizes of models, which seems pretty incremental. On the other hand, the differences are pretty big with the best  proprietary models (e.g. GPT4o has 76.6% on MATH vs CoSC's 53.5%  and Claude 3.5 Sonnet has 96.4% on GSM8K vs CoSC's 82.3%), which indicates a much bigger scale of potential (attainable) improvement that could have been made. Also, I am curious if you can show the core value of the technique with a prompting-based approach over any proprietery model? You are already using prompts to train the model with GPT-4 - what if you used those same prompts to generate CoSC type trajectories on top of the best performing proprietary models like GPT4o and Sonnet - will it improve upon their results further? That will also help establish the core value of the technique apart from the fine-tuning gains you see over open source models.    


Other comments:

- I'm not sure why it is stressed that CoSC can work with zero shot as opposed to proprietary models that require few shots - since it has been already fine tuned heavily on this task it is not surprising it does not need additional examples - whereas the proprietary models are very generic powerful LLMs so they need a few examples to orient them towards this specific form of reasoning tasks. So in a sense the heavy fine-tuning is already supposed to replace the few shot training right?

- This wording is pretty convoluted and confusing: "The results reveal that our CoSC-Code-34B can outperform all
the advanced proprietary LLMs, as well as most advanced proprietary multi-modal LLMs, such as GPT-4V, Gemini-1.0 Pro, and Gemini-1.0 Ultra". It can be misleading to say it outperforms "all the advanced proprietrary LLMs" as that covers ALL proprietary LLMs including the multimodal ones. It should be "all non-multi-modal proprietrary LLMs". (BTW, I am not sure why multi-modality distinction actually matters here?)

- Can you highlight in bold the best performing proprietary models in Table 2 (which I think is GPT4o and Claude 3.5 Sonnet). 


Typos:
 - "There are some recent studies (Chen et al., 2023b; Gou et al., 2023a; Lightman et al., 2023; Huang
et al., 2024a; Chen et al., 2024b) attempt to enable large language models to perform self-correction
by either prompting methods or or fine-tuning methods."
- "rewrited" should be "rewrote" or "rewritten" in multiple places
- "to obtain the final our CoSC model."
- "CoSC consisits of"

questions:
1. Please explain in detail what are the important differences in technique of your approach and ToRA. 
2. Can you compare a prompt-only version of your approach on the best proprietary models and show improvements over them?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730494054398

thread_type:
official_review

<thread_object>
creator: reviewer_kvmj

<content>
summary:
Authors are studying whether an LLM that performs poorly on mathematical reasoning tasks can achieve a much stronger performance through a combination of fine-tuning and structured sampling with code execution. They find that their method produces very strong performance compared to proprietary and OS models and they analyze how their different interventions contribute to this performance increase.

strengths:
The authors demonstrate that an initial seed model trained on GPT-4 generated data (distillation) can then be used to generate its own training data through self-correction (expert iteration), which
- reduces dependency on expensive API calls to GPT-4 after the initial seeding phase
- shows that a model can effectively act as its own teacher/critic through the Chain of Self-Correction mechanism
- demonstrates that relatively small amounts of high-quality seed data can go some distance towards bootstrap a more extensive self-improvement process
This methodology is powerful and the authors execute well on it.

weaknesses:
The authors train a small model on the output of a large model and find that it improves performance a lot (compare "model distillation" - https://arxiv.org/abs/2305.02301) and then train the resulting model on its own output and find that it improves performance further (compare born again networks https://arxiv.org/abs/1805.04770 or more generally "expert iteration"). The authors should mention these highly related fields of research and contextualize their research better, especially a big step of performance increase comes from the distillation step.

The comparison landscape is incomplete and potentially misleading: Notable omission of open-source models with strong mathematical performance (QWEN2, Orca-Math, ...) that match/surpass the achieved performance.

The authors should indicate exactly which GPT-4 version was used for training data generation (presumably gpt-4o-2024-08-06 or gpt-4o-mini-2024-07-18?). This is not only important for reproducibility, but also for understanding how much of the gap between the weak model and GPT-4 the models were able to close via distillation.

The phrase "for some complex solutions we can only get < 1" is slightly awkward

questions:
Do the experiments for Table 4 use a fewshot prompt for the non-CoSC models to tell them how to utilize multiple rounds of reasoning? How do the authors explain that the fraction of "more than one round of reasoning" very close to 0% for those models?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730677644767

thread_type:
official_review

<thread_object>
creator: reviewer_vqbn

<content>
summary:
The paper describes a method called Chain of Self-Correction (CoSC). The idea is to generate a large set of synthetic data that includes stages of self-correction in order to fine-tune a model to learn self-correction capabilities. Then, at inference time, they employ this verification step to enhance performance on mathematical reasoning.

strengths:
- The paper provides a framework for improving the self-correction abilities of models. They propose a framework to synthetically generate data for improving self correction.
- This synthetic data could be a useful resource for training models in the future. 
- The performance seems good, and the method beats prior work fine-tuning on synthetically generated data.

weaknesses:
- Compared to ToRA, the method only has about 3% improvement. However, while ToRA was only trained on 16k annotated samples, this method was trained on 37k, so the comparison is not apples to apples.
- There is no ablation to clarify what capabilities are gained at a more granular level. CoSC has a generation and a verification step, unlike previous methods. The paper does not analyze these separately. For example, what is the precision/recall of the verification step on the programs (how often does the verification step accidentally classify a correct program as wrong, or vice versa)?
- The main novelty of this paper is the verification/self-correction step. Without an extensive evaluation showing these capabilities have improved, it is hard to assess the effectiveness of the proposed method.

questions:
- Did you try any normalizing experiment between CoSC and ToRA where both are trained on the same number of samples?
- Compared to ToRA and looking at Table 5, it seems like the accuracy when using just one round of reasoning is lower than ToRA. Does that mean the data generated is worse than ToRA's when it comes to reasoning without any self-correction steps?
- Because the models were trained on code, do they have an improved sense of code understanding?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730677768821

thread_type:
official_review

<thread_object>
creator: reviewer_ediq

<content>
summary:
The paper presents a fine-tuning technique called Chain of Self-correction (CoSC) that embeds self-correction as an inherent ability of LLM. The method is specifically developed for LLM mathematics benchmarks - MATH and GSM8K. The paper proposes a specific format (program+output+verification+conclusion) for fine-tuning data. GPT4 is used first to get part of the fine-tuning data. This initial data is used to fine-tune a smaller model. After initial fine-tuning, the rest of the data is self-generated by a fine-tuned small model, and the model is further fine-tuned using the new data.

strengths:
- The paper is well-written and easy to follow. I appreciate Figure 1, which summarizes prompts used in some recent related works.

- Compared to the prior work ToRA, the main difference is that the model is made to verify and confirm the answer. If the model deems that the output is incorrect then the model is made to repeat the whole procedure in multiple rounds. This improves the model’s accuracy on the benchmark by around 2%. 

- Evaluation is quite comprehensive and considers many open-source and closed-source models

weaknesses:
I can’t understand some of the main contributions of CoSC without some further clarification:

From what I can get from the evaluation in the paper - the second-best method in the evaluation is ToRA which converts a mathematical problem into Python code, evaluates it, and then uses CoT to infer the final solution. And the CoSC model improves on the ToRA model by around 2%. 

How is ToRA prompted in this evaluation? If the ToRA models were not allowed to self-correct, is it possible to simply modify the prompt and use a similar prompt as CoSC (and possibly use some few-shot examples instead of fine-tuning) to allow it to self-correct? If not, then it would answer my main concern about whether collecting large amounts of data in two phases and fine-tuning in CoSC is actually necessary. 

The algorithm 1 is referenced in line 310. I think this place is quite inappropriate for referring to the main algorithm. Consider moving the algorithm a little bit earlier in section 3.2.1.

I don’t understand what Table 4 for the ToRA code means without much information about the evaluation setup. From Figure 1, it seems that ToRA does not perform any self-correction, so what does the 0.1% cases for ToRA in Round=2 mean? 

Line 142: “revolutionize the accuracy” - please remove revolutionize here

Line 215: what if the conclusion is never reached?

Grammar issues: Line 342, 344, 351, 363

Can you add more description to the appendix? Some parts of the appendix like Appendix B are quite unclear.

questions:
As pointed out in weakness, my main question is about the prompt used for ToRA in Table 2. 

Line 452: “Unlike these proprietary models, our CoSC performs the inference in a zero-shot manner without demonstration” - 
I don’t understand this. How do the proprietary models considered in the evaluation use demonstrations?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732514850048

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We would like to thank the AC for organizing the review of our paper. We found all the reviews and comments extremely helpful. We have addressed all the concerns raised by the reviewers and have revised the paper accordingly to properly address any questions raised. 

We would also like to thank all the reviewers for taking the time to review our paper and provide valuable feedback:
- We would like to thank Reviewer ediq for recognizing the clarity of our paper, the usefulness of Figure 1 in summarizing related prompts, and for highlighting the improvement in accuracy enabled by our CoSC mechanism. 
- We are grateful to Reviewer VQBN for appreciating our framework for enhancing self-correction abilities and for acknowledging the potential future value of this resource for training models. 
- We sincerely thank Reviewer KVMJ for commending our two-phase fine-tuning approach, which leverages self-correction to reduce reliance on expensive API calls, and for highlighting the strength of our methodology in enabling models to act as their own teacher. 
- We also thank Reviewer gRDy for recognizing the cost-efficiency and effectiveness of our two-phase fine-tuning approach, the breadth of our evaluation across various model sizes, and the clarity and structure of our paper. 



We address each reviewer’s comments individually below. We have worked hard to address your concerns and hope you find our responses informative. If you feel our comments have not sufficiently addressed your concerns, we would love to discuss them with you further. We have also uploaded a **Paper Revision** for your consideration.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732970516105

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
A Kind Reminder of Further Discussion

comment:
Dear Reviewers,

We would like to express our sincere gratitude for taking the time to review our paper. We greatly appreciate your insightful comments and are truly thankful for your efforts. We have worked diligently to address all of the concerns you raised, and we hope that our responses have sufficiently resolved any remaining issues. **If you have any further questions or require clarification, we would be more than happy to discuss them with you.** Your feedback is invaluable to us.

Sincerely,\
Authors of Submission 7154
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces Chain of Self-Correction (CoSC), a fine-tuning mechanism enabling LLMs to iteratively self-correct by generating programs, executing them, verifying outputs, and refining or finalizing answers. It uses a two-stage data synthesis approach: GPT-4 generates initial data to train a seed LLM, which then generates additional data to fine-tune itself. Experiments show that CoSC outperforms existing prompting and fine-tuning methods on MATH and GSM8K across proprietary and open-source LLMs. 

Most reviewers and the AC generally acknowledge that the proposed method is well-motivated, with comprehensive experiments to support its claims. However, concerns are raised regarding the novelty of the approach. The method integrates several well-established ideas, such as tool use, verification, and iterative reasoning, without introducing significant conceptual innovation. Similarly, the data synthesis strategy combines well-known techniques like data distillation and expert iteration. While the improved performance on math datasets is notable, the gains over prior work, such as ToRA, appear marginal.

Overall, while the paper presents a promising approach and demonstrates incremental improvements, the lack of substantial novelty and limited performance gains lead me to consider this a borderline submission.