Embedding Self-Correction as an Inherent Ability in Large Language Models for Enhanced Mathematical Reasoning

## Abstract

Abstract Accurate mathematical reasoning with Large Language Models (LLMs) is crucial in revolutionizing domains that heavily rely on such reasoning. However, LLMs often encounter difficulties in certain aspects of mathematical reasoning, leading to flawed reasoning and erroneous results. To mitigate these issues, we introduce a novel mechanism, the Chain of Self-Correction (CoSC), specifically designed to embed self-correction as an inherent ability in LLMs, enabling them to validate and rectify their own results. The CoSC mechanism operates through a sequence of self-correction stages. In each stage, the LLMs generate a program to address a given problem, execute this program using program-based tools to obtain an output, subsequently verify this output. Based on the verification, the LLMs either proceed to the next correction stage or finalize the answer. This iterative self-correction process allows the LLMs to refine their reasoning steps and improve the accuracy of their mathematical reasoning. To enable the CoSC mechanism at a low cost, we employ a two-phase finetuning approach. In the first phase, the LLMs are trained with a relatively small volume of seeding data generated from GPT-4, establishing an initial CoSC capability. In the second phase, the CoSC capability is further enhanced by training with a larger volume of self-generated data using the trained model in the first phase, without relying on the paid GPT-4. Our comprehensive experiments demonstrate that CoSC significantly improves performance on traditional mathematical datasets among existing open-source LLMs. Notably, our CoSC-Code-34B model achieved a 53.5% score on MATH, the most challenging mathematical reasoning dataset in the public domain, surpassing the performance of well-established models such as ChatGPT, GPT-4, and even multi-modal LLMs like GPT-4V, Gemini-1.0 Pro, and Gemini-1.0 Ultra. It’s important to note that, unlike these proprietary models, our CoSC performs inference in a zero-shot manner, without the need for demonstrations. To the best of our knowledge, this is the first work in the literature to embed self-correction as an inherent capability in large language models and achieve significant performance improvements. To facilitate open-source research on improving the reasoning capabilities of LLMs, the code and data for this work will be released upon the acceptance of this paper.

## 1 Introduction

Large Language Models (LLMs), such as GPT-4 (OpenAI, ), have recently demonstrated state-of-the-art performance across a variety of natural language processing (NLP) tasks, including natural language generation and understanding (Chowdhery et al., ; Team et al., ; Anil et al., ; Penedo et al., ). Despite their success, LLMs often struggle with mathematical reasoning tasks due to their lack of explicit logical reasoning and judgment, which are crucial for solving such problems. Moreover, there is a fundamental gap between natural language and the language of mathematical formulas, which further complicates these tasks. As a result, accurate mathematical reasoning remains an essential yet challenging capability for LLMs to develop, in order to further advance various domains. Consequently, it is still an open challenge to tackle mathematical problems for existing open-source LLMs, such as LLaMA-2 (Touvron et al., ).

To improve the mathematical reasoning abilities, numerous approaches have been investigated in previous research, including prompting (Wei et al., ; Chen et al., ; Wang et al., ), pretraining (Azerbayev et al., ; Fu et al., ; Shao et al., ), and finetuning (Luo et al., ; Yu et al., ; Yue et al., ; Gou et al., ; Liu et al., ). In particular, finetuning has become a favored technique among them, which updates open-sourced LLMs based on previously generated high-quality question-response pair datasets. Compared to open-source LLMs, finetuning has demonstrated significant improvement, but there is still potential room for further enhancement.

Most current methods (Yu et al., ; Gou et al., ) generate finetuning datasets by prompting GPT-4 to rephrase mathematical questions from various perspectives or incorporate chain-of-thoughts (CoT) analysis (Wei et al., ) and program-of-thoughts (PoT) code (Chen et al., ) to diversify mathematical responses. As a result, LLMs trained on these datasets can comprehend different questions
and learn to use code to solve mathematical problems. However, precise multi-round reasoning capabilities remain challenging for them.
Once potential errors occur throughout reasoning stages, it can lead to incorrect results. Consequently, it is essential to incorporate a self-correction mechanism into mathematical responses, which can enable LLMs to learn to correct themselves in multiple rounds.
In addition, while supervised finetuning datasets can be developed with GPT-4’s assistance, it still requires human experiments with different prompts and the cost of using the interface. This highlights the need for effective finetuning with unlabeled datasets.

Figure: Figure 1: Comparison of four reasoning frameworks for solving an example mathematical question. (a) Chain of Thoughts (CoT) (Wei et al., ). (b) Program of Thoughts (PoT) (Chen et al., ). (c) ToRA (Gou et al., ) that incorporates CoT, PoT, and the utilization of tools. (d) Our proposed CoSC consisits of a sequence of multiple self-correction stages (two stages are shown in this example). Each stage has four sub-stages: (p1) LLMs generate program w.r.t. the question; (o1) execute the program to obtain program output; (v1) perform two-step verification for consistency of the question with both the generated program and the program output; (c1) conclude a refined answer or continue the next subsequent self-correction stage depending on the verification result. The final answer is extracted from the last conclusion sub-stage with regular expression matching.
Refer to caption: x1.png

To address the aforementioned challenges, our study introduces the Chain of Self-Correction (CoSC), a novel mechanism designed to embed self-correction as an inherent capability in LLMs, enabling them to validate and rectify their own results. The CoSC mechanism operates through a sequence of self-correction stages, where LLMs generate a program to solve a given problem, execute the program using program-based tools to obtain an output, and subsequently verify this output. Depending on the verification, the LLMs either advance to a subsequent stage of self-correction or conclude with the refined solution. An example of our CoSC reasoning trajectory with multiple
self-correction stages (two stages are shown in this example) is shown in Fig. 1. This iterative self-correction process allows the LLMs to refine their reasoning steps and improve the accuracy of its mathematical reasoning.

To implement the CoSC mechanism at a low cost, we adopt a two-phase finetuning approach. In the first phase, termed the CoSC foundational learning, LLMs are trained with a relatvely small volume of seeding data generated from GPT-4, equipping them with a baseline proficiency in the CoSC methodology. In particular, we prompt GPT-4 with training questions from MATH (Hendrycks et al., ) and GSM8K (Cobbe et al., ) datasets to generate mathematical reasoning trajectories that adhere to the CoSC protocol. Specifically, each generated trajectory consists of program-of-thoughts code, program output, a two-step verification process that ensures the alignment of the question with both the generated program and the resulting output, and a conclusion to determine whether the trajectory should be refined or if the final answer can be provided.

Utilizing GPT-4 can be expensive, especially when dealing with large volumes of training data. Alternatively, we propose a cost-free method to further boost performance through self-enhancement in the second phase. This second phase, referred to as CoSC self-enhancement, builds upon the foundational learning by further adapting the LLMs obtained from the first phase with self-generated trajectories. These trajectories are produced by the models trained in the foundational phase, allowing for the generation of a substantial volume of data without additional GPT-4 intervention. In both phases, we retain only the trajectories whose answers match the ground-truth labels of the corresponding questions.

In summary, our study makes the following contributions:

- •
We propose the Chain of Self-Correction (CoSC) mechanism, which effectively embeds self-correction as an inherent ability in Large Language Models (LLMs). Once LLMs learn this ability during training, they can self-correct in a zero-shot setting during inference without the need for external feedback or few-shot demonstrations. With the inherent self-correction ability, even an originally weak LLM is able to achieve excellent performance in mathematical reasoning, as strongly supported by our experimental results. This unique contribution distinguishes our work from related works discussed in Section 2.2 of this field.
- •
To implement the CoSC mechanism at a low cost, we introduce a two-phase finetuning approach. The first phase involves CoSC foundational learning, where we use a relatively small volume of seeding data generated by GPT-4. In the second phase, CoSC self-enhancement occurs using a larger volume of self-generated data with the model obtained from the first phase, without relying on paid GPT-4.
- •
Our comprehensive experimental evaluations demonstrate that the CoSC mechanism provides a new benchmark for performance on established mathematical datasets when compared to existing open-source LLMs. Notably, our CoSC-Code-34B model achieves superior performance over both closed-source LLMs and some of multi-modal LLMs, particularly on the challenging MATH dataset. These findings underscore the potential of our CoSC to revolutionize the accuracy and reliability of LLMs in complex problem-solving domains.
- •
The proposed CoSC mechanism, by embedding self-correction as an inherent capability in LLMs, enables them to think before responding to a question, creating an internal chain of self-correction to progressively verify and rectify their original answers. It is more akin to the slow thinking process of humans, which is particularly helpful in solving difficult mathematical reasoning problems. This approach can provide valuable insights for future research and contribute to the ongoing advancement of LLMs across various domains.

## 2 Related Work

### 2.1 LLMs for Mathematical Reasoning

Mathematical reasoning (Liu et al., ; Wang et al., ; Huang et al., ; Toshniwal et al., ; Chen et al., ; Zhang et al., ) is a challenging reasoning task for LLMs, which requires the ability to understand mathematical concepts, computation and multi-round reasoning. Existing mathematical reasoning approaches can be broadly classified into three categories: (1) Prompting methods (Wei et al., ; Chen et al., ; Wang et al., ) focus on extracting the inherent mathematical reasoning skills of LLMs by utilizing well-crafted prompting strategies during inference. Notably, they leverage the existing knowledge in LLMs without the need for parameter updates. (2) Pretraining methods (Azerbayev et al., ; Fu et al., ) pre-train LLMs on large-scale corpora containing mathematical problems and related content with language modeling objectives. The goal is to train a base foundation language model as a platform for mathematical domain. (3) Finetuning methods (Luo et al., ; Yu et al., ; Yue et al., ; Gou et al., ; Liu et al., ; Gao et al., ) refine the mathematical reasoning capabilities of LLMs by offering more targeted training. Central to this approach is the generation of high-quality question-response pair datasets, which needs the assistance of complexity-based chain-of-thoughts prompting or tools-based augmentation. In this paper, we propose a Chain of Self-Correction (CoSC) along this direction that incorporates an iterative self-correction process into datasets generation.

### 2.2 Existing methods related to Self-Correction in LLMs

Mathematical reasoning poses a significant challenge due to its demand for precise multi-round logical reasoning to solve problems. The potential for errors increases with each reasoning step, making it crucial for LLMs to have the ability to self-correct in order to produce accurate results. There are some recent studies (Chen et al., ; Gou et al., ; Lightman et al., ; Huang et al., ; Chen et al., ) attempt to enable large language models to perform self-correction by either prompting methods or or fine-tuning methods. For prompting methods, they can correct their responses when interacting with external tools (Gou et al., ; Chen et al., ), such as search engines and calculators, or designing complex prompts (Chen et al., ). Notably, prompting methods can be orthogonally combined with finetuning methods. For finetuning methods, previous works (Yu et al., ; An et al., ) only model single-round correction during the training stage, and perform verification in a straightforward manner during the inference stage.

### 2.3 Difference between the proposed method and existing self-correction techniques

The existing self-correction research can be broadly classified into two categories: prompt-based methods (Gou et al., ; Chen et al., ; ) and SFT-based methods (Yu et al., ; An et al., ). Our approach falls under the SFT-based category and fundamentally differs from prompt-based methods. Specifically, prompt-based methods leverage the intrinsic capabilities of large language models (LLMs) for mathematical reasoning. In contrast, our method embeds the chain of self-correction, a strong reasoning mechanism, as an inherent capability in the LLMs through parameter fine-tuning. Furthermore, compared to existing SFT-based methods, our approach models mathematical reasoning as a multi-round procedure during the training stage. In addition, these SFT-based works (Yu et al., ; An et al., ) perform the verification process in a straightforward manner, whereas our work conducts verification in a step-by-step manner by generating intermediate verification steps. Similar to the essence of the CoT approach, our model first verifies whether the generated code aligns with the problem description and then verifies whether the obtained result is consistent with the problem description. This step-by-step verification strategy significantly improves the verification process in self-correction. The excellent performance of our method on the challenging mathematical problem tasks, such as MATH and GSM8k, clearly demonstrates its effectiveness compared to the existing SFT-based methods (Yu et al., ; An et al., ).

## 3 Method

### 3.1 Overview

Figure: Figure 2: The training of Chain of Self-Correction (CoSC) consists of two phases. The first phase, (a) CoSC Foundational Learning, trains LLMs with seeding data generated from proprietary models, equipping them with a baseline proficiency in the CoSC methodology. In particular, we prompt GPT-4 with training questions from MATH (Hendrycks et al., ) and GSM8K (Cobbe et al., ) datasets to generate mathematical reasoning trajectories that adhere to the CoSC protocol. The second phase, (b) CoSC Self Enhancement, further adapts the seed model obtained from the previous phase with self-generated trajectories. These trajectories are produced by the seed model trained in the foundational phase, thereby enabling the generation of a substantial volume of data without the need for additional GPT-4 intervention. In both phases, we only retain trajectories whose answers match the ground-truth label.
Refer to caption: x2.png

We propose a Chain of Self-Correction (CoSC) to address mathematical questions in a self-correction manner.
Given a mathematical question denoted as $q$, the CoSC mechanism initiates the process by generating a program $p$ that comprises a Python function. The program $p$ is then executed using a Python interpreter to produce an output $o$. However, the program $p$ may contain bugs or incorrect intermediate code due to potential misinterpretations of the question $q$, leading to erroneous outputs. Unlike existing methods that either base their reasoning on incorrect intermediate content or halt reasoning upon encountering an error, the CoSC mechanism introduces a self-correction rationale during the reasoning process. After the generation of program $p$ and its output $o$, the CoSC model analyzes them and generates a verification $v$. This verification includes suggestions for improvements to ensure the consistency of the program $p$ and its output $o$ with the mathematical question $q$, respectively. Following this, the CoSC mechanism draws a conclusion $c$ based on these suggestions, which is used to either refine the program $p$ or generate the final answer. This process is repeated until a conclusive answer is reached, which can be summarized as $answer=povcpovc\cdots povc$. By employing this reasoning method, we can annotate public mathematical datasets and use the annotated multi-round, self-correction data to fine-tune LLMs.

### 3.2 Training

To enable the CoSC mechanism, we propose a two-phase finetuning method. (1) CoSC foundational learning with seeding data generated from proprietary models. (2) CoSC self enhancement with self-generated data using the seed model obtained in the first training phase. A summary for training our CoSC is shown in Fig. 2.

#### 3.2.1 CoSC Foundational Learning

Existing mathematical reasoning datasets, such as chain-of-thoughts (CoT) (Wei et al., ) and program-of-thoughts (PoT) (Chen et al., ), primarily contain single-round annotations without multi-round, self-correction solutions for the reasoning process. This makes it challenging to fine-tune models to inherently possess self-correction capabilities. To address this issue, we utilize GPT-4 ($\mathcal{G}$) and a few-shot approach to annotate questions from publicly available mathematical datasets, including MATH (Hendrycks et al., ) and GSM8K (Cobbe et al., ), to generate seeding data with our Chain of Self-Correction (CoSC) mechanism. Table 1 compares our seeding data of CoSC with recently proposed mathematical reasoning datasets.
Detailed instructions and example cases of our CoSC can be found in Appendix A and Appendix B.

Given a mathematical question $q$, the few-shot prompt used to call GPT-4 is defined as $\wp$. The corresponding trajectory, denoted as $\tau$, is generated through the following process.
Firstly, we feed both the few-shot prompt $\wp$ and the question $q$ into GPT-4, which then generates a Python function code $p$ to solve the question $q$. It can be represented as:

$$ $\wp\oplus q\oplus\tau_{i-1}\to p_{i},$ (1) $$

where the symbol $\oplus$ represents concatenation and $\tau_{i}$ indicates the trajectory in the $i$ interaction round.
After obtaining the code $p_{i}$, we execute it using a Python interpreter to acquire the runtime result $o_{i}$.
Then, we enclose the result $o$ within “‘‘‘output” and feed it, along with the previous few-shot prompt $\wp$, question $q$, the previous trajectory $\tau_{i}$, and generated code $p_{i}$ to the GPT-4. This produces a verification $v_{i}$ that analyzes the consistency between the question $q$ and the program code $p_{i}$, as well as between the question $q$ and the program output $o_{i}$.
Additionally, it produces a conclusion $c_{i}$ on whether to proceed with the next round of reasoning to perform adjustment, denoted as:

$$ $\wp\oplus q\oplus\tau_{i-1}\oplus p_{i}\oplus o_{i}\to v_{i}\oplus c_{i}.$ (2) $$

Depending on the conclusion $c_{i}$, it decides whether to terminate the reasoning process or proceed to the next round. If there is no inconsistency found during the verification step, it generates a natural language conclusion $c_{i}$ and encloses the answer to the problem with the special symbol “boxed{}”. Otherwise, it should further revise the generated code $p_{i}$ in a new reasoning round by updating the trajectory $\tau_{i}$, denoted as:

$$ $\tau_{i}=\tau_{i-1}\oplus p_{i}\oplus o_{i}\oplus v_{i}\oplus c_{i}.$ (3) $$

**Table 1: The comparison to the previous mathematical reasoning datasets. Note that our CoSC only employs questions from the original training set of MATH and GSM8K datasets. Particularly, we provide the number of annotation data as #Annotation in each mathematical reasoning method. Additionally, Correction signifies whether the self-correction is to be utilized.**
| Methods | #Annotation | Tool | Correction | LLM Used | Source |
| --- | --- | --- | --- | --- | --- |
| RFT (Yuan et al., ) | $>$100k | ✗ | ✗ | LLaMA-2 | GSM8K |
| Open-Platypus (Lee et al., ) | 25k | ✗ | ✗ | GPT-4 | 11 datasets with MATH |
| WizardMath (Luo et al., ) | $>$96k | ✗ | ✗ | ChatGPT | MATH & GSM8K |
| Lila (Mishra et al., ) | 134k | ✓ | ✗ | - | 20 datasets with MATH & GSM8K |
| MathInstruct (Yue et al., ) | 260k | ✓ | ✗ | GPT-4 | 14 datasets with MATH & GSM8K |
| ToRA (Gou et al., ) | 16k | ✓ | ✗ | GPT-4 | MATH & GSM8K |
| CoSC (Ours) | 37k | ✓ | ✓ | GPT-4 | MATH & GSM8K |

Based on the above steps, CoSC iteratively generates PoT code, program output, verification, and conclusion. At final, the reasoning process is stopped by checking whether the generated result contains an answer enclosed in “boxed{}”.

To ensure that the self-correction can concisely and clearly analyze the code and its runtime results, we design the self-correction in a step-by-step format as follows:

- •
The first step involves verifying whether the generated program code $p$ is consistent with the question $q$, such as checking the variables and their relationships.
- •
The second step involves verifying whether the code runtime results $o$ meet the requirements of the question $q$, such as checking the reasonableness of numerical values.

Finally, for summary, CoSC will generate a conclusion $c$ to determine whether we should start the next round of reasoning or provide the final answer.

With CoSC, we use GPT-4 to annotate the MATH and GSM8K training datasets. To explore diverse data, we apply nucleus sampling (Holtzman et al., ) during GPT-4 annotation. Each question is sampled in 3 times. However, for some complex questions, we can only get less than 1 correct solutions. For these problems, we apply 10 more samplings and retained up to 4 correct data. Finally, we filtered out incorrect answers and constructed 37k pieces of data using GPT-4. The algorithm of the generation with our CoSC for each question from the training set is shown in Algorithm 1.

Based on the 37k data constructed by GPT-4 as the seeding data, we apply CoSC foundational learning to train models. For a given question $q$, the response trajectory $\tau$ generated by GPT-4, denoted as $\tau=povcpovc\cdots povc$, the minimized negative log-likelihood loss used for training the model can be represented as:

$$ $arg\min_{\theta}\sum_{q,\tau}\sum_{i=1}^{n_{q}-1}-\text{log}\ \mathbb{P}_{ \theta}(p_{i+1}o_{i+1}v_{i+1}c_{i+1}|q,p_{i}\cdots o_{i}v_{i}c_{i}),$ (4) $$

where $n_{q}$ is the iteration rounds of the question $q$ in our CoSC for the mathematical reasoning.

Figure: Algorithm 1 Inference Reasoning with our CoSC

#### 3.2.2 CoSC Self Enhancement

After the completion of the CoSC foundational learning, the seed model gains the ability to self-correct during inference and perform multi-round reasoning. Subsequently, we employ the seed model, after CoSC foundational learning, to apply dense solution sampling and dense question sampling, which enables the generation of more self-generated data with self-correction mechanisms.

Dense solution sampling. For mathematical questions, there are usually multiple solutions, but use GPT-4 for annotating them is expensive.
Therefore, we use the seed model after CoSC foundational learning to resample the questions in the datasets multiple times further improve the model generalization ability in answering questions.
Specifically, we use the CodeLLaMA-34B model which has the best performance after CoSC foundational learning to perform dense nucleus-sampling on 16k training data questions.
Each question in the MATH and GSM8K training datasets is sampled 64 times.
We filter out the correct answers based on whether they match the ground-truth.

Dense question sampling. Similarly, there are multiple ways to ask a mathematical question. To improve the model generalization ability of question understanding, we use the data from MetaMath (Yu et al., ) to generalize the questions.
Each question in MetaMATH is rewrited by simply rephrasing it, self-verifying the conditions in the question, adding if-then questions to reverse the question conditions, etc.
Specifically, we use the CodeLLaMA-34B model with CoSC foundational learning to perform nucleus-sampling on the questions, sampling each question once, and filtering out data points based on whether they match the ground-truth.

In the end, we obtain a total of 339k data points, including 37k seeding data generated from GPT-4 and 302k generated from the CodeLLaMA-34B model with CoSC foundational learning. Then, we adopt them to train models from scratch to obtain the final our CoSC model.

### 3.3 Implementation Details

By using 339k data points, including 37k seeding data and 302k self-generated data, we fine-tune the base models of LLaMA-2 (Touvron et al., ) and CodeLLaMA (Roziere et al., ) to obtain our CoSC and CoSC-Code, respectively. They are with different parameter sizes, such as 7B, 13B, and 34B. All models use full-scale fine-tuning. We use the AdamW optimizer with a learning rate of 2e-5 for all models, with a batch size set to 128, training for 1 epoch. To enable training, we use DeepSpeed ZeRO stage 3 (Rajbhandari et al., ) and Flash-Attention 2 (Dao, ) to optimize the model’s memory usage.
During inference, we set a maximum of 3 calls to the Python interpreter and a maximum token length of 2048.
Our experiments train the models in 7B size with 8 NVIDIA A800 80GB GPUs and train the models in 13B and 34B with 16 NVIDIA A800 80GB GPUs.

## 4 Experiments

### 4.1 Evaluation Setup

Datasets.
We evaluated models on the most widely used mathematical problem tasks, MATH (Hendrycks et al., ) and GSM8K (Cobbe et al., ).
The MATH dataset encompasses a total of 12,500 problems, partitioned into 7,500 for training and 5,000 for testing.
The GSM8K dataset contains 8,500 problems, with 7,500 for training and 1,000 for testing.
These datasets collectively encompass a broad range of mathematical questions, from basic arithmetic to competition level. More details of datasets are provided in Appendix C.

**Table 2: Accuracy results (%) on MATH and GSM8K datasets. Vanilla models are tested with CoT. ZS indicates the zero-shot inference without demonstrations. PAL refers to the Program-Aided Language model prompting (Gao et al., ). The best results in each section are in bold and the second-best results are underlined. Our CoSC can outperform open-sourced LLMs, proprietary LLMs and some of multi-modal LLMs, on the more challenging MATH dataset.**
| Model | Base | Size | ZS | MATH | GSM8K | AVG |
| --- | --- | --- | --- | --- | --- | --- |
| Proprietary Models |  |  |  |  |  |  |
| GPT-4o | - | - | ✗ | 76.6 | - | - |
| GPT-4V | - | - | ✗ | 52.9 | 92.0 | 72.5 |
| GPT-4 (PAL) | - | - | ✗ | 51.8 | 94.2 | 73.0 |
| GPT-4 | - | - | ✗ | 42.5 | 92.0 | 67.3 |
| ChatGPT (PAL) | - | - | ✗ | 38.7 | 78.6 | 58.7 |
| ChatGPT | - | - | ✗ | 35.5 | 80.8 | 58.2 |
| Claude-3.5 Sonnet | - | - | ✗ | 71.1 | 96.4 | 83.8 |
| Claude-3 Opus | - | - | ✗ | 60.1 | 95.0 | 77.6 |
| Claude-2 | - | - | ✗ | 32.5 | 85.2 | 58.9 |
| Gemini-1.5 Pro | - | - | ✗ | 67.7 | 90.8 | 79.3 |
| Gemini-1.5 Flash | - | - | ✗ | 54.9 | 86.2 | 70.6 |
| Gemini-1.0 Ultra | - | - | ✗ | 53.2 | 88.9 | 71.1 |
| Gemini-1.0 Pro | - | - | ✗ | 32.6 | 77.9 | 55.3 |
| PaLM-2 | - | 540B | ✗ | 34.3 | 80.7 | 57.5 |
| Open-Source Models |  |  |  |  |  |  |
| LLaMA-2 | LLaMA-2 | 7B | ✗ | 4.1 | 13.3 | 8.7 |
| LLaMA-2 SFT | LLaMA-2 | 7B | ✓ | 7.2 | 41.3 | 24.3 |
| LLaMA-2 RFT | LLaMA-2 | 7B | ✓ | - | 51.2 | - |
| CodeLLaMA (PAL) | CodeLLaMA | 7B | ✗ | 16.6 | 34.0 | 25.3 |
| Platypus-2 (Lee et al., ) | LLaMA-2 | 7B | ✗ | 5.4 | 14.4 | 9.9 |
| WizardMath (Luo et al., ) | LLaMA-2 | 7B | ✓ | 10.7 | 54.9 | 32.8 |
| MetaMath (Yu et al., ) | LLaMA-2 | 7B | ✓ | 19.8 | 66.5 | 43.2 |
| ToRA (Gou et al., ) | LLaMA-2 | 7B | ✓ | 40.1 | 68.8 | 54.5 |
| CoSC (Ours) | LLaMA-2 | 7B | ✓ | 42.7 | 70.5 | 56.6 |
| ToRA-Code (Gou et al., ) | CodeLLaMA | 7B | ✓ | 44.6 | 72.6 | 58.6 |
| CoSC-Code (Ours) | CodeLLaMA | 7B | ✓ | 47.6 | 74.7 | 61.2 (+2.6) |
| LLaMA-2 | LLaMA-2 | 13B | ✗ | 6.3 | 24.3 | 15.3 |
| LLaMA-2 SFT | LLaMA-2 | 13B | ✓ | 9.2 | 51.1 | 30.2 |
| LLaMA-2 RFT | LLaMA-2 | 13B | ✓ | - | 55.3 | - |
| CodeLLaMA (PAL) | CodeLLaMA | 13B | ✗ | 19.9 | 39.9 | 29.9 |
| Platypus-2 (Lee et al., ) | LLaMA-2 | 13B | ✗ | 7.1 | 23.7 | 15.4 |
| WizardMath (Luo et al., ) | LLaMA-2 | 13B | ✓ | 14.0 | 63.9 | 39.0 |
| MetaMath (Yu et al., ) | LLaMA-2 | 13B | ✓ | 22.4 | 72.3 | 47.4 |
| ToRA (Gou et al., ) | LLaMA-2 | 13B | ✓ | 43.0 | 72.7 | 57.9 |
| CoSC (Ours) | LLaMA-2 | 13B | ✓ | 45.3 | 73.9 | 59.6 |
| ToRA-Code (Gou et al., ) | CodeLLaMA | 13B | ✓ | 48.1 | 75.8 | 62.0 |
| CoSC-Code (Ours) | CodeLLaMA | 13B | ✓ | 50.3 | 77.3 | 63.8 (+1.8) |
| LLaMA-1 RFT | LLaMA-1 | 34B | ✓ | - | 57.9 | - |
| CodeLLaMA (PAL) | CodeLLaMA | 34B | ✗ | 23.9 | 53.3 | 38.6 |
| ToRA-Code (Gou et al., ) | CodeLLaMA | 34B | ✓ | 50.8 | 80.7 | 65.8 |
| CoSC-Code (Ours) | CodeLLaMA | 34B | ✓ | 53.5 | 82.3 | 67.9 (+2.1) |

Baselines. For proprietary models, we present results from an array of SoTA LLMs, such as OpenAI’s ChatGPT (gpt-3.5-turbo), OpenAI’s GPT-4, OpenAI’s GPT-4V, OpenAI’s GPT-4o, Google’s PaLM-2, Google’s Gemini-1.0 Pro, Google’s Gemini-1.0 Ultra, Google’s Gemini-1.5 Flash, Google’s Gemini-1.5 Pro, Anthropic’s Claude-2, Anthropic’s Claude-3 Opus, and Anthropic’s Claude-3.5 Sonnet. By default, we report CoT prompting results, and include PAL (Program-Aided Language model) prompting (Gao et al., ) results for selected models. For open-source models, base models comprise LLaMA-2 (Touvron et al., ) and CodeLLaMA (Roziere et al., ) with CoT and PAL prompting (Gao et al., ). Supervised Fine-Tuning (SFT) employs CoT rationales from the original MATH and GSM8K dataset (15k samples) for fine-tuning. Rejection sampling Fine-Tuning (RFT) (Yuan et al., ) leverages multiple models to generate diverse reasoning paths for fine-tuning. WizardMath (Luo et al., ) augments data using ChatGPT, and conducts SFT and RLHF. Platypus-2 (Lee et al., ) is fine-tuned with Open-Platypus reasoning datasets. ToRA (Gou et al., ) uses GPT-4 to generate tool-used trajectories and finetunes on it.

Metric.
We report accuracies of predicted answers. For numerical values, we perform rounding, while for expressions, we employ the Python library sympy for parsing.

### 4.2 Main Results

In our experiments, we observe several findings that demonstrate the effectiveness of our proposed CoSC model on mathematical datasets. We show the experimental results in Table 2.
First and foremost, our proposed CoSC consistently outperforms previous state-of-the-art open-source LLMs across all scales. Specifically, our CoSC-Code can achieve an average improvement of 2.6%, 1.8%, and 2.1%, on 7B, 13B, and 34B size, respectively.

Moreover, to further emphasize the superiority of our CoSC, we conduct a comprehensive comparison against multiple proprietary models. The results reveal that our CoSC-Code-34B can outperform all the advanced proprietary LLMs, such as ChatGPT and GPT-4, as well as most advanced proprietary multi-modal LLMs, such as GPT-4V, Gemini-1.0 Pro, and Gemini-1.0 Ultra on MATH dataset.
It is important to note that, unlike these proprietary models, our CoSC performs the inference in a zero-shot manner without demonstrations.

### 4.3 Ablation Study

In our ablation study, we conduct experiments on MATH dataset with the largest number of 5,000 test samples with broad spectrum of subjects and difficulty levels. For the consideration of computational efficiency, we choose the smaller models of CodeLLaMA with size 7B and 13B as the base models.

#### 4.3.1 Effect of Each Training Phase

Our CoSC mechanism comprises two training phases: (1) CoSC foundational learning and (2) CoSC self-enhancement. In order to assess the individual contributions of each training phase, we evaluate multiple combinations of the proposed training phases, the results of which are shown in Table 5.

The experiments conducted on CodeLLaMA serve as the baseline for our study. Initially, we employ GPT-4 to generate 37k seeding data for CoSC foundational learning, which equips LLMs with initial self-correction capabilities. As a result, we observe a notable improvement in performance on the MATH dataset, with accuracy increasing from 16.6% to 42.3% for the 7B size and from 19.9% to 47.0% for the 13B size.

To further enhance the self-correction performance, we utilize the seed model obtained in the first phase to generate additional data without relying on GPT-4. This approach leads to further
improvements in performance on the MATH dataset from 42.3% to 47.6% and 47.0% to 50.3%, for 7B and 13B respectively.

It is exciting to see that the originally weak LLMs, such as the CodeLLaMA 7B base model and the 13B base model, can significantly improve in mathematical reasoning by using the proposed CoSC method, as shown in Table 5. This demonstrates the effectiveness of the CoSC method, which is able to embed self-correction as an inherent capability in LLMs, leading to significant improvements in mathematical reasoning performance.

#### 4.3.2 Effect of Multi-Round Reasoning in the Proposed CoSC mechanism

Our CoSC mechanism integrates a series of self-correction stages to progressively verify and refine output of LLMs. In particular, the conclusion step in our CoSC relies on clues from the verification step to determine whether to proceed to the next round of reasoning or directly provide the final answer. Such iterative multi-round mechanism enables LLMs to self-correct their outputs and improve accuracy. This experiment quantitatively investigates effect of this multi-round mechanism. As shown in Table 5, our CoSC can effectively generate more rounds of mathematical reasoning during inference, confirming efficacy of our CoSC mechanism in enhancing the reasoning process.

In addition, we conduct a comparison between single-round reasoning and multi-round reasoning using our CoSC mechanism on the test samples from the MATH dataset. The single-round results are obtained by extracting the answer solely from the output of the first round, without any self-correction enabled. To illustrate the impact of multi-round reasoning, we present the comparative results in Table 5. The results clearly demonstrate the effectiveness of multi-round reasoning in rectifying errors and improving overall accuracy. With the 7B model, we observe an improvement in accuracy from 40.2% to 47.6%, while with the 13B model, accuracy increases from 42.4% to 50.3%. These findings highlight the significant benefits of employing multi-round reasoning within our CoSC mechanism.
More ablation studies are shown in Appendix D.

**Table 3: Accuracy results (%) on the MATH dataset for the CoSC foundational learning and CoSC self enhancement. The results show the superiority and necessity of both training phases.**
| Chain of Self-Correction (CoSC) | 7B | 13B |  |
| --- | --- | --- | --- |
| Foundational Learning | Self Enhancement |  |  |
| ✗ | ✗ | 16.6 | 19.9 |
| ✓ | ✗ | 42.3 | 47.0 |
| ✓ | ✓ | 47.6 | 50.3 |

## 5 Conclusion

In conclusion, our Chain of Self-Correction (CoSC) mechanism equips Large Language Models (LLMs) with the ability to autonomously validate and refine their outputs. This mechanism facilitates a sequence of self-correction stages that progressively refine the reasoning process, leading to enhanced accuracy in mathematical reasoning. Our two-phase finetuning approach, involving initial training with proprietary model-generated seeding data and subsequent training with self-generated data, effectively enables the implementation of CoSC at a low cost. Through extensive experiments, we have demonstrated the remarkable performance improvement that CoSC brings to traditional mathematical datasets when compared to existing open-source LLMs, even the advanced close-sourced LLMs, such as ChatGPT, GPT-4 and multi-modal LLMs like GPT-4V, Gemini-1.0 Pro, and Gemini-1.0 Ultra.
We believe that our CoSC mechanism can provide valuable insights for future research and contribute to the ongoing advancement of LLMs across various domains.

## Appendix A Prompt for our CoSC

We present specific instructions and example few-shot prompts of our CoSC for querying GPT-4 to generate the seeding data.

### A.1 Prompt for our CoSC on MATH

The prompt for CoSC on MATH is as follows:

### A.2 Prompt for our CoSC on GSM8K

The prompt for CoSC on GSM8K is as follows:

## Appendix B Case of our CoSC on MATH and GSM8K Testing Sets

### B.1 Case on MATH testing set

The case for CoSC on MATH testing set is as follows:

### B.2 Case on GSM8K testing set

The case for CoSC on GSM8K testing set is as follows:

## Appendix C Datasets Details

MATH (Hendrycks et al., ). This dataset consists of competition level mathematics problems. It encompasses a total of 12,500 problems, partitioned into 7,500 for training and 5,000 for testing. Each problem is accompanied by a step-by-step solution and concludes with a distinct final answer, which is formatted for straightforward verification of the model generated solutions. Notably, the MATH dataset spans a broad spectrum of subjects and difficulty levels, including seven categories: Prealgebra, Algebra, Number Theory, Counting and Probability, Geometry, Intermediate Algebra, and Precalculus.

GSM8K (Cobbe et al., ). Comprising a diverse collection of grade school mathematical word problems, GSM8K is recognized for its high quality. While it is generally considered less challenging than the MATH dataset, it similarly provides step-level solutions with basic arithmetic operations (addition, subtraction, multiplication, division). The GSM8K dataset contains 8,500 problems, with 7,500 for training and 1,000 for testing.

## Appendix D More Ablation Studies

### D.1 Accuracy of the Verification Module and Error Reduction of the Correction Module

Our CoSC framework comprises two main components: verification and correction. The verification component identifies potential erroneous reasoning, while the correction component generates improved reasoning to address the issues identified in the verification step. To evaluate the effectiveness of these components, we conducted ablation studies. We provide a detailed analysis of the benefits derived from both modules, reporting the accuracy of the verification module and the error reduction achieved by the correction module. The accuracy of the verification module reflects how precisely it identifies errors, whereas the error reduction of the correction module measures the rate at which errors are corrected from one round to the next. Specifically, it examines the proportion of solutions deemed incorrect in the i-th round that are successfully corrected in the (i + 1)-th round.

The results of CoSC-Code in 7B and 13B on MATH dataset are shown in Table 7. It can be observed that the verification of our CoSC-Code can iteratively refine its outputs with high accuracy, about 70%. Furthermore, for more difficult questions that require self-correction across multiple rounds, our CoSC-Code is still capable of successfully reducing errors by over 25%. This confirms the effectiveness of both the verification module and the correction module in our CoSC method.

### D.2 Context Lengths for Different Questions

We explore the effect of the context lengths when addressing different questions over multiple rounds of self-corrections. We have calculated the percentage distribution of context lengths for CoSC-Code in 7B and 13B on MATH dataset, as shown in Table 7. These statistics indicate that all context lengths fall within the 4K range, which is well-suited for modern LLMs.

**Table 6: The accuracy of the verification module and the error reduction rate achieved by the correction module of CoSC-Code in 7B and 13B on MATH dataset.**
| Models | accuracy of the verification module | error reduction of the correction module |
| --- | --- | --- |
| CoSC-Code-7B | 68.5% | 25.74% |
| CoSC-Code-13B | 70.4% | 26.25% |

### D.3 Alleviated Issues by our CoSC

Our CoSC primarily targets resolving two types of issues related to (a) code errors, such as NotImplementedError, and (b) inconsistency errors between outputs and the given questions. We have quantified the reduction in errors for the 7B models on MATH dataset, as demonstrated in the results below. It can be observed that these two types of errors are effectively minimized by our CoSC method. This reduction highlights the effectiveness of our CoSC approach in enhancing the accuracy and reliability of LLMs in mathematical reasoning tasks.

(a) Statistics for error reduction related to code errors: reduced from 931 to 167.

(b) Statistics for error reduction related to inconsistency errors: reduced from 310 to 1.

## Appendix E Limitations

In this paper, we propose a novel mechanism known as the Chain of Self-Correction (CoSC) designed to enhance the reasoning capabilities of Large Language Models (LLMs). While our research primarily concentrates on mathematical reasoning, we posit that the CoSC mechanism could be effectively utilized across a wider spectrum of applications to rectify errors produced by LLMs. However, due to constraints related to the length of this paper, a comprehensive exploration of this generalization will be reserved for future study.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: As pointed out in weakness, my main question is about the prompt used for ToRA in Table 2. 

Line 452: “Unlike these proprietary models, our CoSC performs the inference in a zero-shot manner without demonstration” - 
I don’t understand this. How do the proprietary models considered in the evaluation use demonstrations?
soundness: 3
strengths: - The paper is well-written and easy to follow. I appreciate Figure 1, which summarizes prompts used in some recent related works.

- Compared to the prior work ToRA, the main difference is that the model is made to verify and confirm the answer. If the model deems that the output is incorrect then the model is made to repeat the whole procedure in multiple rounds. This improves the model’s accuracy on the benchmark by around 2%. 

- Evaluation is quite comprehensive and considers many open-source and closed-source models
summary: The paper presents a fine-tuning technique called Chain of Self-correction (CoSC) that embeds self-correction as an inherent ability of LLM. The method is specifically developed for LLM mathematics benchmarks - MATH and GSM8K. The paper proposes a specific format (program+output+verification+conclusion) for fine-tuning data. GPT4 is used first to get part of the fine-tuning data. This initial data is used to fine-tune a smaller model. After initial fine-tuning, the rest of the data is self-generated by a fine-tuned small model, and the model is further fine-tuned using the new data.
weaknesses: I can’t understand some of the main contributions of CoSC without some further clarification:

From what I can get from the evaluation in the paper - the second-best method in the evaluation is ToRA which converts a mathematical problem into Python code, evaluates it, and then uses CoT to infer the final solution. And the CoSC model improves on the ToRA model by around 2%. 

How is ToRA prompted in this evaluation? If the ToRA models were not allowed to self-correct, is it possible to simply modify the prompt and use a similar prompt as CoSC (and possibly use some few-shot examples instead of fine-tuning) to allow it to self-correct? If not, then it would answer my main concern about whether collecting large amounts of data in two phases and fine-tuning in CoSC is actually necessary. 

The algorithm 1 is referenced in line 310. I think this place is quite inappropriate for referring to the main algorithm. Consider moving the algorithm a little bit earlier in section 3.2.1.

I don’t understand what Table 4 for the ToRA code means without much information about the evaluation setup. From Figure 1, it seems that ToRA does not perform any self-correction, so what does the 0.1% cases for ToRA in Round=2 mean? 

Line 142: “revolutionize the accuracy” - please remove revolutionize here

Line 215: what if the conclusion is never reached?

Grammar issues: Line 342, 344, 351, 363

Can you add more description to the appendix? Some parts of the appendix like Appendix B are quite unclear.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: - Did you try any normalizing experiment between CoSC and ToRA where both are trained on the same number of samples?
- Compared to ToRA and looking at Table 5, it seems like the accuracy when using just one round of reasoning is lower than ToRA. Does that mean the data generated is worse than ToRA's when it comes to reasoning without any self-correction steps?
- Because the models were trained on code, do they have an improved sense of code understanding?
soundness: 2
strengths: - The paper provides a framework for improving the self-correction abilities of models. They propose a framework to synthetically generate data for improving self correction.
- This synthetic data could be a useful resource for training models in the future. 
- The performance seems good, and the method beats prior work fine-tuning on synthetically generated data.
summary: The paper describes a method called Chain of Self-Correction (CoSC). The idea is to generate a large set of synthetic data that includes stages of self-correction in order to fine-tune a model to learn self-correction capabilities. Then, at inference time, they employ this verification step to enhance performance on mathematical reasoning.
weaknesses: - Compared to ToRA, the method only has about 3% improvement. However, while ToRA was only trained on 16k annotated samples, this method was trained on 37k, so the comparison is not apples to apples.
- There is no ablation to clarify what capabilities are gained at a more granular level. CoSC has a generation and a verification step, unlike previous methods. The paper does not analyze these separately. For example, what is the precision/recall of the verification step on the programs (how often does the verification step accidentally classify a correct program as wrong, or vice versa)?
- The main novelty of this paper is the verification/self-correction step. Without an extensive evaluation showing these capabilities have improved, it is hard to assess the effectiveness of the proposed method.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: Do the experiments for Table 4 use a fewshot prompt for the non-CoSC models to tell them how to utilize multiple rounds of reasoning? How do the authors explain that the fraction of "more than one round of reasoning" very close to 0% for those models?
soundness: 2
strengths: The authors demonstrate that an initial seed model trained on GPT-4 generated data (distillation) can then be used to generate its own training data through self-correction (expert iteration), which
- reduces dependency on expensive API calls to GPT-4 after the initial seeding phase
- shows that a model can effectively act as its own teacher/critic through the Chain of Self-Correction mechanism
- demonstrates that relatively small amounts of high-quality seed data can go some distance towards bootstrap a more extensive self-improvement process
This methodology is powerful and the authors execute well on it.
summary: Authors are studying whether an LLM that performs poorly on mathematical reasoning tasks can achieve a much stronger performance through a combination of fine-tuning and structured sampling with code execution. They find that their method produces very strong performance compared to proprietary and OS models and they analyze how their different interventions contribute to this performance increase.
weaknesses: The authors train a small model on the output of a large model and find that it improves performance a lot (compare "model distillation" - https://arxiv.org/abs/2305.02301) and then train the resulting model on its own output and find that it improves performance further (compare born again networks https://arxiv.org/abs/1805.04770 or more generally "expert iteration"). The authors should mention these highly related fields of research and contextualize their research better, especially a big step of performance increase comes from the distillation step.

The comparison landscape is incomplete and potentially misleading: Notable omission of open-source models with strong mathematical performance (QWEN2, Orca-Math, ...) that match/surpass the achieved performance.

The authors should indicate exactly which GPT-4 version was used for training data generation (presumably gpt-4o-2024-08-06 or gpt-4o-mini-2024-07-18?). This is not only important for reproducibility, but also for understanding how much of the gap between the weak model and GPT-4 the models were able to close via distillation.

The phrase "for some complex solutions we can only get < 1" is slightly awkward

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. Please explain in detail what are the important differences in technique of your approach and ToRA. 
2. Can you compare a prompt-only version of your approach on the best proprietary models and show improvements over them?
soundness: 3
strengths: 1. Interesting and cost-efficient two-phase fine-tuning approach. Using a two-phase approach (starting with seeding data from GPT-4 and then using self-generated data) is an interesting and effective way to reduce reliance on expensive models. The authors also include an ablation study that shows the significant improvements with both initial GPT4 and self-enhanced finetuning. 
2. Evaluation is very broad and shows core value. The paper shows a very extensive evaluation with many proprietary and open source models as baselines, which is admirable. Their CoSC approach is also evaluated on three different sizes of LLMs up to 35B, where the approach consistently performs better than all other open source models on all sizes. Ablation studies also show the gains from multiple rounds of correction to show the value of the concept of chain of self correction, and further ablation studies are included in appendices.  
3. Presentation and details. Well-structured and well-written paper. The problem is motivated well, and the technique, implementation and evaluation is clearly explained with details, and prompts and more details are provided in extensive appendices.
summary: This paper proposes a technique for improving mathematical reasoning with large language models which the authors call the  "Chain of Self-Correction (CoSC)". CoSC is designed to incorporate self-correction as an inherent ability in LLMs to iteratively refine their reasoning through multiple rounds of program generation, execution, and verification. The approach uses a two-phase fine-tuning process: first, training with a small dataset generated by GPT-4, and then self-enhancing with self-generated data, thereby reducing reliance on expensive models. On standard math reasoning datasets MATH and GSM8, the method outperforms a wide range of open source techniques and models that are tested in this work, and also performs better than some proprietary LLMs.
weaknesses: 1. Novelty is not clear. In particular, I am not sure how exactly the CoSC technique is different from ToRA (Gou et al., 2023b). ToRA seems to also use the interleaved approach of program-generation, execution and rationales in multiple rounds (similar to the chain of correction you have here). One thing that seems different with CoSC is that you have explicit verification and conclusion steps and verification checks separately for question and output correctness. But those seem a bit like relatively smaller optimizations to guide the model to do more explicit reasoning. You also classify ToRA as under prompting approaches as opposed to fine-tuning in the related work discussion, but from reading their work it seems they are also fine tuning the models? (and in fact using GPT4 for initial data generation and also using two steps in training their models). But I am not completely sure of this - please clarify what exactly the significant differences in technique there are with ToRA. 

2. Evaluation results seem not to show very drastic gains. The overall improvement over the best open source model (ToRA) is 1.8% - 2.6% for all sizes of models, which seems pretty incremental. On the other hand, the differences are pretty big with the best  proprietary models (e.g. GPT4o has 76.6% on MATH vs CoSC's 53.5%  and Claude 3.5 Sonnet has 96.4% on GSM8K vs CoSC's 82.3%), which indicates a much bigger scale of potential (attainable) improvement that could have been made. Also, I am curious if you can show the core value of the technique with a prompting-based approach over any proprietery model? You are already using prompts to train the model with GPT-4 - what if you used those same prompts to generate CoSC type trajectories on top of the best performing proprietary models like GPT4o and Sonnet - will it improve upon their results further? That will also help establish the core value of the technique apart from the fine-tuning gains you see over open source models.    


Other comments:

- I'm not sure why it is stressed that CoSC can work with zero shot as opposed to proprietary models that require few shots - since it has been already fine tuned heavily on this task it is not surprising it does not need additional examples - whereas the proprietary models are very generic powerful LLMs so they need a few examples to orient them towards this specific form of reasoning tasks. So in a sense the heavy fine-tuning is already supposed to replace the few shot training right?

- This wording is pretty convoluted and confusing: "The results reveal that our CoSC-Code-34B can outperform all
the advanced proprietary LLMs, as well as most advanced proprietary multi-modal LLMs, such as GPT-4V, Gemini-1.0 Pro, and Gemini-1.0 Ultra". It can be misleading to say it outperforms "all the advanced proprietrary LLMs" as that covers ALL proprietary LLMs including the multimodal ones. It should be "all non-multi-modal proprietrary LLMs". (BTW, I am not sure why multi-modality distinction actually matters here?)

- Can you highlight in bold the best performing proprietary models in Table 2 (which I think is GPT4o and Claude 3.5 Sonnet). 


Typos:
 - "There are some recent studies (Chen et al., 2023b; Gou et al., 2023a; Lightman et al., 2023; Huang
et al., 2024a; Chen et al., 2024b) attempt to enable large language models to perform self-correction
by either prompting methods or or fine-tuning methods."
- "rewrited" should be "rewrote" or "rewritten" in multiple places
- "to obtain the final our CoSC model."
- "CoSC consisits of"

Meta-review: This paper introduces Chain of Self-Correction (CoSC), a fine-tuning mechanism enabling LLMs to iteratively self-correct by generating programs, executing them, verifying outputs, and refining or finalizing answers. It uses a two-stage data synthesis approach: GPT-4 generates initial data to train a seed LLM, which then generates additional data to fine-tune itself. Experiments show that CoSC outperforms existing prompting and fine-tuning methods on MATH and GSM8K across proprietary and open-source LLMs. 

Most reviewers and the AC generally acknowledge that the proposed method is well-motivated, with comprehensive experiments to support its claims. However, concerns are raised regarding the novelty of the approach. The method integrates several well-established ideas, such as tool use, verification, and iterative reasoning, without introducing significant conceptual innovation. Similarly, the data synthesis strategy combines well-known techniques like data distillation and expert iteration. While the improved performance on math datasets is notable, the gains over prior work, such as ToRA, appear marginal.

Overall, while the paper presents a promising approach and demonstrates incremental improvements, the lack of substantial novelty and limited performance gains lead me to consider this a borderline submission.

Decision: Reject