SIaM: Self-Improving Code-Assisted Mathematical Reasoning of Large Language Models

## Abstract

Abstract There is a growing trend of teaching large language models (LLMs) to solve mathematical problems through coding. Existing studies primarily focus on prompting powerful, closed-source models to generate seed training data followed by in-domain data augmentation, equipping LLMs with considerable capabilities for code-aided mathematical reasoning. However, continually training these models on augmented data derived from a few datasets such as GSM8K may impair their generalization abilities and restrict their effectiveness to a narrow range of question types. Conversely, the potential of improving such LLMs by leveraging large-scale, expert-written, diverse math question-answer pairs remains unexplored. To utilize these resources and tackle unique challenges such as code response assessment, we propose a novel paradigm that uses a code-based critic model to guide steps including question-code data construction, quality control, and complementary evaluation. We also explore different alignment algorithms with self-generated instruction/preference data to foster continuous improvement. Experiments across both in-domain (up to + 5.7 % percent 5.7 +5.7\% + 5.7 % ) and out-of-domain ( + 4.4 % percent 4.4 +4.4\% + 4.4 % ) benchmarks in English and Chinese demonstrate the effectiveness of the proposed paradigm. Models and code will be released at https://github.com/tencent-ailab/siam .

## 1 Introduction

Though large language models (LLMs) have demonstrated strong performance on mathematical benchmarks, they still struggle with achieving accurate computation and reasoning, especially in out-of-domain scenarios. For example, even powerful closed-source LLMs achieve only $2\%$ accuracy on 5-digit multiplication (Chen et al., ) with step-by-step reasoning (or Chain-of-Thought, CoT) (Wei et al., ). To alleviate the computational burden on LLMs, particularly those of smaller sizes, there is a growing trend of utilizing code and code interpreters to enhance precise computation and reasoning of LLMs in solving mathematical problems (Chen et al., ; Gao et al., ; Zhou et al., ). An effective method involves prompting closed-source LLMs to generate code-based solutions for given questions. However, previous studies demonstrated that closed-source models often struggle with real-world high school and college-level math exams (Liu et al., ). Solving advanced problems through coding demands not only mathematical expertise but also interdisciplinary knowledge and skills, including programming and natural language, making it a more formidable challenge even for closed-source LLMs. Previous code-assisted studies primarily focus on using closed-source LLMs such as GPT-4 to label a few small-scale, representative datasets such as GSM8K (Cobbe et al., ) and MATH (Hendrycks et al., ), verifying the correctness of the solutions via pattern-based answer matching, and training models on the verified data for further in-domain data augmentation through sampling, code execution, and answer validation (Wang et al., ; Liu et al., ; Gou et al., ; Lu et al., ). However, continually learning from these datasets or their augmented versions, regardless of the use of code, is evidently less effective for improving the generalization of LLMs due to the limited diversity.

Figure: Figure 1: Overview of our self-improving code-assisted paradigm using large-scale web QA data.
Refer to caption: x1.png

On the other hand, large-scale, expert-written, mathematical question-answer (QA) pairs from educational web resources remain under-studied to improve code-assisted math reasoning abilities of LLMs. These resources span educational levels from primary school to college and include various question types and answer formats, such as multiple-choice, application, proof, and cloze. To use these resources to self-improve code-assisted LLMs, instead of further extensively distilling closed-source models, one natural solution is to use a fine-tuned model to generate code samples for each problem and use the valid data to (iteratively) improve this LLM, similar to self-improved CoT reasoners (Zelikman et al., ; Yuan et al., ; Xu et al., ; Hosseini et al., ) over data with reference answers. However, the key challenge is to determine whether the self-generated code responses align with reference answers in diverse formats. Fortunately, with the aid of an external code interpreter, we are less concerned about potential step-wise computation errors that may occur in CoT reasoning. We assume a code solution is more likely to be correct if its execution result matches the reference answers, thus shifting the focus from the step-by-step comparison to comparing the reference answers with the code execution results. Based on our analysis (Section 3.1), we observe that most cases primarily require format conversion between plain text and code syntax (e.g., “(x-5)(x^2-4x+7)” vs. “(x-5)*(x**2-4*x+7)” and “(1, -2, 2, -3)” vs. “{A:1, B:-2, C:2, D:-3}”) and relatively simple numerical calculations, which do not require advanced logical reasoning abilities or in-depth language-specific knowledge (Section 3.5).

These observations and task simplification motivate us to design a critic model to evaluate the correctness of the code execution result against the reference answer by predicting Yes or No (see examples in Table 1). As illustrated in Figure 1, this critic model is used to guide multiple steps during self-improvement. We first train a model with seed question-code data following previous code-assisted studies and consider it as the initial policy model. In each iteration, we use the current policy model to generate code samples for new questions and keep the highest-scoring valid code responses rated by the critic model for supervised fine-tuning (SFT) in the subsequent iteration. To foster continuous improvement, we also explore different preference learning algorithms such as DPO (Rafailov et al., ) and ORPO (Hong et al., ) with self-generated preference data, where the preference labels are provided by the critic model.

We perform experiments on various model families, such as Llama3-8B (AI@Meta, ) and DeepSeek-Coder-7B (Guo et al., ), and Qwen2-7B (Yang et al., ). Experimental results across both in-domain (up to $+5.7\%$) and out-of-domain (OOD) ($+4.4\%$) benchmarks in English and Chinese show the effectiveness of self-improving LLMs using our proposed paradigm with large-scale web QA pairs. The resulting 7-8B models can outperform state-of-the-art 70B code-assisted math LLMs (Gou et al., ) by $11.9\%$ in OOD scenarios. Notably, we observe a strong correlation between the traditional heuristic-based evaluation method and the critic model (Section 3.5), with the latter reducing the additional human effort needed to design rules for new mathematical benchmarks. Additionally, introducing SFT loss into the DPO training is surprisingly effective in controlling the code response length. To summarize the contributions of this work:

- •
To the best of our knowledge, this is the first attempt to leverage large-scale web QA pairs to improve the code-assisted mathematical reasoning abilities of LLMs.
- •
To better leverage these large-scale, diverse web QA pairs, we propose a novel iterative self-improving paradigm that employs a new critic model to guide various steps such as data construction and filtering. This critic model can also serve as a complementary evaluation scorer, reducing the reliance on heuristic design for new evaluation tasks.
- •
Extensive experiments on both English and Chinese tasks demonstrate the effectiveness of our paradigm, and our comprehensive analysis of the key factors in achieving continuous improvement at different stages may shed light on future studies.

**Table 1: Example instances of the code-based critic task ((question, reference answer) pairs come from the web QA data, and Chinese texts are translated into English).**
| System Prompt | Your goal is to evaluate whether the candidate answer provided by the model for a math problem matches the reference answer. Here are the steps to complete the task: |
| --- | --- |
|  | – First, carefully read the given math problem. |
|  | – Next, review the reference answer for the math problem. |
|  | – Then, examine the candidate answer provided by the model, which may include a program and the result of running that program. |
|  | – Finally, summarize whether the candidate answer matches the reference answer or can be made to match through simple calculations/conversions. |
|  | – The response format should be Yes or No. |
| Instruction | ### Question\n\n Given f（1-2x）=3x+1，find f（-3）= __ ． |
|  | ### Reference Answer\n \n 7 \n \n |
|  | ### Candidate Answer\n \n <code>from sympy import symbols, Eq, solve\n# Define the variable\n x = symbols(’x’)\n# Establish the equation based on the problem conditions\n equation = Eq(1 - 2*x, -3)\n# Solve the equation\n solution = solve(equation, x, dict=True)\n solution_value = solution[0][x]\n # Calculate f(-3)\n f_neg_3 = 3*solution_value + 1\n f_neg_3</code><solution>7</solution>\n\n |
|  | ### Assessment |
| Output | Yes |
| Instruction | ### Question\n\n Given the sequence {a<sub>n</sub>} with the sum of the first n terms S<sub>n</sub>=2n-3, what is a<sub>3</sub>? A: 3B: 2C: 1D: 0 |
|  | ### Reference Answer\n \n B \n \n |
|  | ### Candidate Answer\n \n <code># Define a function to calculate the nth term\n def calculate_nth_term(n):\n # Based on the derived formula\n an = 2 ** n - 2 ** (n - 1)\n return an\n # Calculate the value of the 3rd term \n a3 = calculate_nth_term(3) # a3 </code><solution>4</solution>\n\n |
|  | ### Assessment |
| Output | No |

## 2 Method

### 2.1 Training an Initial Model

One key factor for an iterative paradigm is to have a reasonably strong initial model. To train such a model, we first use high-quality seed data to fine-tune an LLM, resulting in model $M_{\text{seed}}$. We use $M_{\text{seed}}$ to generate code samples and keep up to four predictions per question wherein the execution result of the code matches the reference answer and combines the seed data and the self-distilled data to train $M_{0}$, which is further used as the initial model for later stages. We will introduce more details about the seed data construction in the experiment section.

### 2.2 Building a Multi-Use Code-Based Critic Model

To improve LLMs with large-scale, diverse-format math QA data without code annotations, several challenges arise in data utilization, filtering, and evaluation. First, previous studies primarily use pattern-based methods to compare predictions and ground truth answers during validation and evaluation. This works well for GSM-style datasets, where answers are single numbers and well-formatted (e.g., *“72”* in *“…72 clips altogether in April and May.\n #### 72”*). However, pattern-based methods face inherent challenges in handling diverse answer types and formats and bridging the gap between natural language and programming language. For example, with the MATH dataset, comparing CoT predictions with reference answers in LaTeX-like format already requires human-written patterns and answer conversion (Yue et al., ). This complexity is compounded when predictions are presented in code syntax, even when the task is simplified to compare the reference answer with the code execution result.

To address the above challenges, we propose building a code-based critic model optimized by the following objective:

$$ $L(r_{\phi})=-\log r_{\phi}(y\mid q,a,c,e),$ (1) $$

where $q$ denotes a question, $a$ is the reference answer to $q$, $c$ represents the code response to $q$, and $e$ is the execution result of code $c$. To simplify the task, we let $y$ be either “Yes” or “No”. Examples are shown in Table 1. We leave other formulations, such as training a scalar critic model (Ouyang et al., ), to future work.

### 2.3 Code Data Generation

As mentioned previously, our goal is to leverage web math QA data to continuously self-improve the code-assisted mathematical reasoning ability of LLMs. For well-formatted, web-collected data such as APE (Zhao et al., ) and CM (Qin et al., ), where most answers are one or two numerical values (see examples in Table 15), it is efficient and effective to compare the reference answer and the execution result of the code using scripts released by previous studies (Section 3.2). For real-world math data involving multiple types of questions, such as multiple-choice, multiple-question, fill-in-the-blank, application, and proof, using a critic model introduced in the previous section is more flexible and saves the intensive effort of writing task-specific patterns, which is time-consuming and may suffer from relatively low recall. Note that for all questions, we only use their reference answers to verify the correctness of code execution results instead of directly training on these answers, and we only use benchmarks’ training sets.

In the $k+1$-th iteration, for each new question, we use the current policy model $\pi_{\theta_{k}}$ to generate five code samples and execute them to obtain the results. For questions in the diverse-format web data, the critic model is then used to predict Yes or No for each response ($a_{i}$, $c_{ij}$, $e_{ij}$) given $q_{i}$. We use the probability of Yes or No as the confidence value for the critic model’s judgment. A higher probability score indicates a greater confidence in the code response, either agreeing with or disagreeing with the reference answer.

### 2.4 Self-Improvement with Unseen Data

One natural choice is to perform supervised fine-tuning (SFT) on $\pi_{\theta_{k}}$ using $D_{\text{SFT}}$:

$$ $\begin{split}L_{\text{SFT}}(\pi_{\theta_{k+1}})=-\log\pi_{\theta_{k+1}}(c\mid q )\end{split}$ (2) $$

$$ $D_{\text{SFT}}=\{(q_{i},c_{ij})\mid r_{\phi}(y=\textsc{Yes}\mid q_{i},a_{i},c_ {ij},e_{ij})\}$ (3) $$

As critiques may contain errors, we explore using the probability of each judgement as a confidence score to filter out noise. Besides, we introduce extra constraints: for each question, we only retain the highest-scoring positive instance $t_{ij}=\{q_{i},a_{i},c_{ij},e_{ij}\}$, similar to rejection sampling (Bai et al., ), where $t_{ij}\in T_{i}$ of the same question $q_{i}$. To encourage models to learn from more challenging problems, if all instances in $T_{i}$ are labeled as Yes, we discard this question and its corresponding generated code from consideration.

$$ $\displaystyle D_{\text{SFT, H}}=\{(q_{i},c_{ij})\mid$ $\displaystyle\ r_{\phi}(y=\textsc{Yes}\mid t_{ij}),$ (4) $\displaystyle\ p_{r_{\phi}}(y=\textsc{Yes}\mid t_{ij})>\lambda_{1},\$ $\displaystyle\ t_{ij}=\arg\max_{t_{ij}\in T_{i}}p_{r_{\phi}}(y=\textsc{Yes} \mid t_{ij}),\$ $\displaystyle\ \sum_{j=1}^{|T_{i}|}\mathbf{1}\{r_{\phi}(y=\text{No}\mid t_{ij} )\}\geq\lambda_{2}\}$ $$

where $\lambda_{1}$, $\lambda_{2}$ represent thresholds for filtering and difficulty control.

In addition to supervised fine-tuning a policy model on self-generated SFT data ($D_{\text{SFT, H}}$ or $D_{\text{SFT}}$), we further leverage negative instances by optimizing the policy model on preference data using algorithms such as DPO (Rafailov et al., ) and ORPO (Hong et al., ). Compared to SFT, these preference learning algorithms additionally decrease the probability of losing responses. We mainly focus on DPO and leave other options for future studies, and we jointly train the policy with the SFT objective to alleviate overfitting to the preference data and ensure a stable update (Hong et al., ). See more discussions on the impact of the SFT objective, especially its role in controlling the response length, in Section 3.4.

$$ $\centering L_{\text{DPO}}(\pi_{\theta_{k+1}})=-\log\sigma\beta\log\frac{\pi_{ \theta_{k+1}}(y_{w}\mid x)}{\pi_{\theta_{k}}(y_{w}\mid x)}-\beta\log\frac{\pi_ {\theta_{k+1}}(y_{l}\mid x)}{\pi_{\theta_{k}}(y_{l}\mid x)}-\lambda\cdot\log \pi_{\theta_{k+1}}(y_{w}\mid x)\@add@centering$ (5) $$

We can easily leverage our critic model to build preference ($c_{w}$, $c_{l}$) pairs, where $c_{w}$ represents the winning code and $c_{l}$ represents the losing code. For each question, we use the highest-scoring Yes response and the highest-scoring No response to form a preference pair, aiming to maximize the difference between them. See preference data examples in Section A.6.

$$ $\displaystyle D_{\text{DPO}}=\{(q_{i},c_{ij},c_{ik})\mid$ $\displaystyle\ r_{\phi}(y=\textsc{Yes}\mid t_{ij}),$ (6) $\displaystyle\ r_{\phi}(y=\textsc{No}\mid t_{ik}),$ $\displaystyle\ t_{ij}=\arg\max_{t_{ij}\in T_{i}}p_{r_{\phi}}(y=\textsc{Yes} \mid t_{ij}),\$ $\displaystyle\ t_{ik}=\arg\max_{t_{ik}\in T_{i}}p_{r_{\phi}}(y=\textsc{No}\mid t _{ik})\}$ $$

## 3 Experiments

### 3.1 Data

We summarize the statistics of data used for self-improvement in Table 2. Due to limited space, see statistics of evaluation benchmarks in Table 14 and more examples in Appendix.

**Table 2: Statistics of training data used in our three-stage paradigm ($D_{1}$ and $D_{2,\text{in-house}}$ are Chinese resources; $D_{2,\text{WebInstruct}}$ is English-dominant).**
| Data/Subset | QA Source | Size |  |
| --- | --- | --- | --- |
| $D_{0}$ | zh | web | 76K |
| en | GSM8K, MATH | 44K |  |
| $D_{1}$ | APE, CM | 211K |  |
| $D_{2,\text{in-house}}$ | SFT | educational websites | 893K |
| SFT(H) | 273K |  |  |
| DPO | 465K |  |  |
| $D_{2,\text{WebInstruct}}$ | DPO | pre-training corpora | 447K |

Seed Data $D_{0}$: To generate the seed data for English, following previous work, we use GPT-4-0613 to generate Python code in an iterative fashion: we repeatedly sample the remaining questions that do not have correct code (i.e., the code execution results match the reference answer of the questions) for up to three iterations. We use questions from the training sets of GSM8K (7.5K) and MATH (7.5K) as the seed questions for imitation learning. For datasets such as GSM8K in which the answers are mostly single numbers, it is easier to compare answer and code execution results. After two iterations, we can annotate $98.5\%$ of questions in GSM8K. For datasets such as MATH wherein the answers are diverse in formats, we simply keep the code that can be successfully executed without errors. For seed questions for Chinese, we randomly sample 20K from (1.13M in total) collected or purchased from educational web resources (Section A.6) and follow the same procedure using GPT-4-0613 for code generation to construct the Chinese subset of $D_{0}$.

Value-Style $D_{1}$:
We utilize the initial policy $M_{0}$ to generate code samples to questions in training sets of two open-source word math problem datasets APE (200.5K) (Zhao et al., ) and CM (13.6K) (Qin et al., ), both collected from educational websites covering elementary and middle-school levels. Since all the answers are one or two numerical values, for efficiency, we use heuristics with Python to compare the code execution results with reference answers for validation. We keep up to four valid code samples for each question.

Diverse-Format Data $D_{2}$ and Critic Data: To increase the diversity of our training data, we further consider large-scale mathematical QA pairs (excluding those used for seed data) mentioned previously. For each question, we retain only one positive code and one negative code (if any exists) judged by the critic. To evaluate the generalization and robustness of our paradigm, we also use a recently released large-scale QA dataset named WebInstruct (Yue et al., ) to construct a similar-scale $D_{2}$, containing 447K preference pairs (see examples in Section A.6). Compared to our in-house web QA data, WebInstruct is mostly in English and is extracted from the pre-training corpora. Therefore, the answers are not guaranteed to be written by educational experts as our Chinese web data.

To better understand this web data and the critic task, we analyze the reference answers for $50$ instances. Only $14\%$ of them are single numerical values, while $50\%$ involve format conversion (e.g., syntax or structure) when the answers are expressions, equations, coordinates, sets, etc. Another difference between real-world data and well-formatted benchmarks is the inconsistency in the format of reference answers. Specifically, half of them are mixed with CoT-style (Wei et al., ) explanations and/or irrelevant contents such as tags and URLs. This makes it difficult to parse short-form answers for easier matching with a few patterns, as done for clean benchmarks (e.g., answer indicators “###” for GSM8K and “box” for MATH). For multiple-choice or multi-part questions ($8\%$ in total), we additionally require the question context for mapping option labels and their contents, as well as question decomposition. These observations reflect the diversity of question types in our web QA data. See more examples and statistics in the Appendix.

To build the training data for the critic model, we use $M_{0}$ to generate code samples for randomly sampled questions from $D_{2}$ and execute these code samples. We then prompt GPT-4-0613 with the input (question, code, code result, reference answer) following the template in Table 1. After filtering, we retain 16.8K training instances, of which $48.6\%$ of are judged as Yes.

### 3.2 Implementation

We use LllamaFactory (Zheng et al., ) for efficient fine-tuning built upon DeepSpeed (ZeRO-3). Our experiments are conducted using 8XA100 40GB GPUs. We train LLMs with BF16 mixed-precision. The training for the self-improving paradigm takes approximately 96 hours. With 80 workers in multi-processing mode on a CPU machine, we can execute about 9,003 code samples per minute. Each model at each stage is trained for two epochs with a learning rate of 1e-5 for SFT and 1e-6 for preference learning. We set the SFT loss coefficient ($\lambda$ in Equation 7) to $1.0$. The maximum sequence length is set to $1024$, and the batch size is set to $64$. We set $\lambda_{1}$ to $0.8$ and $\lambda_{2}$ to $3$.

We experiment with various LLMs to select backbone models such as CodeLlama-7B-Python (Roziere et al., ), Llama3${}_{\text{instruct}}$ (AI@Meta, ), CodeQwen1.5-7B-Chat (Team, ), QWEN2(Yang et al., ), and Deepseek-Coder-7B-instruct-v1.5 (Guo et al., ), which demonstrate strong coding capabilities on code-related benchmarks. Due to limited computational resources, we use their 7-8B versions with their default templates and leave the model scaling up for future work. We primarily follow the evaluation scripts from previous studies (Liang et al., ) for Chinese benchmarks and FastEval(^1^11github.com/FastEval/FastEval/.) for English benchmarks GSM8K and MATH, which often use Python for numerical comparison. We also make adjustments to these scripts, as our predicted answers are in code syntax. CodeLlama-7B-Python is used as the backbone model to train the code-based critic model for three epochs with the maximum sequence length $4096$.

### 3.3 The Performance of the Initial Policy and Self-Improved LLMs

As shown in Table 3, we experiment with three backbone models for self-improvement — DeepSeek${}_{\text{code}}$, Llama3${}_{\text{instruct}}$, QWEN2Math${}_{\text{instruct}}$ — that show superior average performance across math datasets in both Chinese (APE, CM, and CMATH (Wei et al., )) and English (GSM8K and MATH) than other investigated models when trained with seed data (see complete results of initial policy models based on eight LLMs in Table 9). Therefore, we consider them as initial policy models (i.e., M0) for self-improvement. After two additional iterations on the unseen data D1, and D2 constructed with the help of our code-based critic model, the resulting models (i.e., M2) consistently outperform M0 by a large margin on Chinese benchmarks.

We observe that self-improving the initial policy model with Chinese-only data, D1 and D2, does not hurt the accuracy of M2 on English tasks. In fact, it may be beneficial (e.g., $+1.5\%$ on both MATH and GSM8K datasets using DeepSeek${}_{\text{code}}$). Conversely, adding English seed data ($36.7\%$ of D0) consistently improves M0’s average performance on Chinese benchmarks (D0 vs. $D_{0,\text{zh}}$ in Table 4). To some extent, we may interpret code as a universal language for solving mathematical problems across different languages. The language-specific parts are mainly in the code comments, which are relatively indirect for problem-solving via code execution. Thus, our paradigm may reduce the burden of preparing large-scale, language-specific math data for each language. We observe similar trends on DeepSeek${}_{\text{code}}$ and QWEN2Math${}_{\text{instruct}}$, see detailed results in Table 10 for space constraints.

We list several general-purpose/math-specified multi-lingual/English LLMs for reference. Note that direct comparisons are challenging due to differences in architectures, pre-training corpora, alignment algorithms, model size, the use of tools, and labeled data. For example, code-assisted methods ToRA, MathCoder, and MathGenieLM are trained on 69K, 80K, and 170K English-only data, respectively, augmented based on GSM8K and MATH. In contrast, our experiments use 44K English seed data and explore the use of large-scale Chinese QA pairs. Moreover, the evaluation scripts, originally designed for plain-text answers instead of code outputs, may cause an underestimation of our methods’ performance on datasets such as MATH, where answers involve more expressions and structures beyond numerical values. This also highlights the need for a more flexible evaluation method.

**Table 3: Accuracy across the development sets of math datasets. All Chinese datasets are OOD for M0. CMATH is OOD for M2 as CM and CMATH are later used for distant supervision.**
|  |  |  | Chinese Tasks | English Tasks |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Model | Size (B) | Tool | CM | APE | CMATH | GSM8K | MATH |
| GPT-4-1106-Preview | – | $\times$ | – | 84.2 | 89.3 | 93.6 | 53.6 |
| Qwen-Chat (Bai et al., ) | 72 | $\times$ | – | 77.1 | 88.1 | 76.4 | 31.8 |
| ChatGLM-Math (Xu et al., ) | 32 | $\times$ | – | 89.4 | 85.6 | 82.6 | 40.6 |
| Skywork-Math (Yang et al., ) | 13 | $\times$ | – | 74.4 | 77.3 | 72.3 | 17.0 |
| Math-InternLM2 (Team, ) | 20 | $\times$ | – | 75.2 | 78.5 | 82.6 | 37.7 |
| MetaMath (Yu et al., ) | 70 | $\times$ | – | – | – | 82.3 | 26.6 |
| MathCoder (Wang et al., ) | 34 | $\checkmark$ | – | – | – | 81.7 | 45.2 |
| ToRA (Gou et al., ) | 70 | $\checkmark$ | – | – | – | 84.3 | 49.7 |
|  | 7 | $\checkmark$ | – | – | – | 72.6 | 44.6 |
| MathGenieLM (Lu et al., ) | 70 | $\checkmark$ | – | – | – | 88.4 | 51.2 |
| MinT (Liang et al., ) | 7 | $\checkmark$ | 77.6 | 76.0 | – | 40.8 | – |
| Initial Model Baselines (M0) |  |  |  |  |  |  |  |
| QWEN2Math${}_{\text{instruct}}$ | 7 | $\checkmark$ | 84.9 | 83.4 | 87.3 | 79.5 | 48.0 |
| DeepSeek${}_{\text{code}}$ | 7 | $\checkmark$ | 82.7 | 81.2 | 87.0 | 77.4 | 44.4 |
| Llama3${}_{\text{instruct}}$ | 8 | $\checkmark$ | 83.3 | 83.2 | 87.2 | 76.8 | 41.8 |
| Self-Improvement with Chinese Diverse-Format Web Data (M2) |  |  |  |  |  |  |  |
| SIaM(QWEN2Math${}_{\text{instruct}}$) | 7 | $\checkmark$ | 90.1 (+5.2) | 88.1 (+4.7) | 93.2 (+5.9) | 81.5 (+2.0) | 50.0 (+2.0) |
| SIaM(DeepSeek${}_{\text{code}}$) | 7 | $\checkmark$ | 87.3 (+4.6) | 85.9 (+4.7) | 91.2 (+4.2) | 78.9 (+1.5) | 45.9 (+1.5) |
| SIaM(Llama3${}_{\text{instruct}}$) | 8 | $\checkmark$ | 89.0 (+5.7) | 86.8 (+3.6) | 90.8 (+3.6) | 80.5 (+3.7) | 41.9 (+0.1) |

**Table 4: Impacts of different stages and data selection on the development sets of datasets.**
| Model | Stages | Data | CM | APE | CMATH | GSM8K | MATH | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Llama3${}_{\text{instruct}}$ | SFT | $D_{0,\text{en}}$ | – | – | – | 75.1 | 37.2 | – |
|  | SFT | $D_{0,\text{zh}}$ | 82.5 | 83.3 | 85.5 | – | – | – |
|  | SFT | $D_{0}$ | 83.3 | 83.2 | 87.2 | 76.8 | 41.8 | 74.4 |
|  | SFT | $D_{0}$ + $D_{1}$ | 87.6 | 85.0 | 89.0 | 76.6 | 41.8 | 76.0 |
|  | SFT $\rightarrow$ DPO | $D_{0}$ + $D_{1}$; $D_{2,\text{WebInstruct}}$ | 87.5 | 86.1 | 88.7 | 80.2 | 42.1 | 76.9 |
|  | SFT $\rightarrow$ DPO | $D_{0}$ + $D_{1}$; $D_{2,\text{in-house}}$ | 89.0 | 86.8 | 90.8 | 80.5 | 41.9 | 77.8 |

### 3.4 The Comparison of Different Choices of Data and Alignment Methods

Diversity: Based on the experimental results, given D0 and D1, we observe that two-stage SFT (first on D0 for two epochs and then on D1 for two epochs) under-performs one-stage SFT (over the concatenation of D0 and D1 for two epochs) (B vs. C in Table 5). However, incorporating D2 using either strategy achieves similar performance (E vs. F in Table 5). One possible reason may be that the questions in D1 are from two web-collected value-style benchmarks, resulting in less diversity compared with D2, which has a broader range of question types (Section 3.1). Ensuring the diversity of data in each stage may help the model generalize better across various types of math questions, similar to the observations seen when training general-purpose LLMs (e.g., (Shen et al., )).

Denoised SFT Data: As mentioned previously, we use the code-based critic model to construct SFT data. Since the process will inevitably introduce false positive data, we further consider several constraints for filtering (Equation 4 in Section 2.4). Experimental results show that we can achieve similar average accuracy using either D${}_{2,\text{SFT,H}}$ or the D${}_{2,\text{SFT}}$ (D vs. E in Table 5). However, D${}_{2,\text{SFT,H}}$ is only $30.6$% of the latter’s size, indicating the usefulness of the filtering.

DPO or SFT: Based on a reasonably good model M1 (trained with D0 and D1, such as C in Table 5), we can either self-improve it via SFT or DPO as described in Section 2.4. We compare using the positive (question, code) pairs in the DPO data for another round of SFT, which results in a $1.8\%$ drop in accuracy on downstream tasks (G vs. I in Table 5). Since we do not impose strict constraints on the positive data in DPO, D${}_{2,\text{DPO, positive}}$ is $1.7$ times the size of D${}_{2,\text{SFT,H}}$. Still, using the filtered SFT data D${}_{2,\text{SFT,H}}$ achieves slightly better performance (F vs. G), showing the effectiveness of filtering.

**Table 5: The self-improving average accuracy of Llama3${}_{\text{instruct}}$ on the development sets of different datasets with various training strategies and data. Refer to Section A.5 for the accuracy on each task.**
| ID | Alignment | Data | Average Accuracy |
| --- | --- | --- | --- |
| A | SFT | $D_{0}$ | 74.4 |
| B | SFT $\rightarrow$ SFT | $D_{0}$ ; $D_{1}$ | 75.4 |
| C | SFT | $D_{0}$ + $D_{1}$ | 76.0 |
| D | SFT | $D_{0}$ + $D_{1}$ + $D_{2,\text{SFT}}$ | 76.1 |
| E | SFT | $D_{0}$ + $D_{1}$ + $D_{2,\text{SFT,H}}$ | 76.1 |
| F | SFT $\rightarrow$ SFT | $D_{0}$ + $D_{1}$; $D_{2,\text{SFT,H}}$ | 76.2 |
| G | SFT $\rightarrow$ SFT | $D_{0}$ + $D_{1}$; $D_{2,\text{DPO, positive}}$ | 76.0 |
| H | SFT $\rightarrow$ ORPO | $D_{0}$ + $D_{1}$; $D_{2,\text{DPO}}$ | 77.0 |
| I | SFT $\rightarrow$ DPO | $D_{0}$ + $D_{1}$; $D_{2,\text{DPO}}$ | 77.8 |

**Table 6: The impact of the weight of the SFT loss in DPO training on the average accuracy and average response length in words on GSM8K and CMATH (L0: response length of reference policy).**
| $\lambda$ | GSM8K | CMATH |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| ACC | $\mathrm{L}$ | $\frac{\mathrm{L}}{\mathrm{L_{0}}}$ | ACC | $\mathrm{L}$ | $\frac{\mathrm{L}}{\mathrm{L_{0}}}$ |  |
| reference model |  |  |  |  |  |  |
| - | 76.6 | 323 | 1.0 | 89.0 | 136 | 1.0 |
| 0.0 | 73.4 | 1834 | 5.7 | 57.5 | 3160 | 23.2 |
| 0.5 | 78.8 | 532 | 1.6 | 90.7 | 201 | 1.5 |
| 1.0 | 80.5 | 352 | 1.1 | 90.8 | 136 | 1.0 |
| 1.5 | 79.0 | 328 | 1.0 | 90.7 | 135 | 1.0 |
| 2.0 | 79.8 | 326 | 1.0 | 90.7 | 134 | 1.0 |

DPO with SFT: Our experiments indicate that DPO training is relatively insensitive to the weight ($\lambda$ in Equation 7) of the SFT loss. We tested with $\lambda=1.0$ and $\lambda=2.0$, both of which resulted in similarly good performance ($77.8$%). However, as shown in Table 6, removing the SFT loss (i.e., $\lambda=0$) from DPO training leads to a dramatic increase in response length, especially for Chinese tasks such as CMATH, and yields worse results than the reference policy model (C in Table 5). This observation aligns with discussions on length exploitation issue of the original DPO loss (Park et al., ). One possible reason for the length control achieved by adding the SFT loss could be that the positive responses used for the SFT loss are generated by the reference policy model. By setting a larger weight to SFT, we control the deviation from the reference policy, which alleviates a substantial increase in response length. We also experiment with using ORPO (Hong et al., ), which removes the need for a reference model and jointly trains with the SFT loss. However, this method is not as effective as jointly training DPO and SFT in our experiments (H vs. I in Table 5).

**Table 7: Correlation of two evaluation methods: heuristics-based EM and the critic model. ACC represents the accuracy of our best-performing M2 on downstream tasks rated by the two methods.**
| Dataset | ACC${}_{\text{EM}}$ | ACC${}_{\text{critic}}$ | Correlation${}_{\text{Kendall}}$ |
| --- | --- | --- | --- |
| CM | 89.0 | 84.6 | 0.66 |
| APE | 86.8 | 86.5 | 0.76 |
| CMATH | 90.8 | 91.8 | 0.77 |
| GSM8K | 80.5 | 80.6 | 0.97 |
| MATH | 41.9 | 48.2 | 0.79 |
| average | 77.8 | 78.3 | 0.79 |

Other Diverse-Format Resources: We also experiment with constructing similar-scale preference data using the diverse-format D2 based on WebInstruct (Section 3.1). However, the resulting improvement in average accuracy is less substantial compared to that achieved with the Chinese diverse-format D2 (+$0.9\%$ vs. +$1.8\%$ on Llama3${}_{\text{instruct}}$ in Table 4; +$0.6\%$ vs. +$2.5\%$ on QWEN2Math${}_{\text{instruct}}$ in Table 10). One possible reason for this difference could be that the QA pairs extracted from pre-training corpora, despite being of similar scale, provide weaker supervision compared to those sourced from educational websites, where answers are typically written by experts. Although we have filtered out QA pairs where reference answers contain no numbers, we observe that some questions still do not require any calculations, such as *“How is the interquartile range (IQR) connected to percentiles?”* or related to other subjects such as *“What is the most prevalent state of matter in the universe …?”*. As a result, these QA pairs may be less effective in improving performance on benchmarks that primarily require numerical computation. Nevertheless, these results demonstrate the robustness of our paradigm.

### 3.5 Using the Critic Model as An Evaluator

We have shown the effectiveness of using the critic model to construct SFT and preference data. All scores are computed by comparing predictions with ground truth answers, using heuristics-based exact match (EM) following previous studies for fair comparisons. To explore the potential of using the critic model as a complementary evaluator, we examine the correlation between the two evaluation methods on the previously used benchmarks. We use the original ground truth answers (final-step answers if answers are COT-style) (e.g., “3750”, “[12, 18]”, and “\\frac{1}{2}”) in these benchmarks. Since all scores are either 0 0 (No) or $1$ (Yes), we report the Kendall’s $\tau$ between the two methods. As shown in Table 7, there is a very strong correlation ($0.79$) (compared to the very-strong-cutoff value $0.71$ and strong-cutoff value $0.49$ (Schober et al., )) between the scores computed by the two evaluators. The strong associations in English tasks are surprising, given that the critic model is trained on Chinese-only data. This may be due to (i) the backbone model being a well-instructed model focused on English, and (ii) comparing answers to mathematical questions relying less on language-specific knowledge.

### 3.6 The Performance of Self-Improved LLMs on More Out-of-Domain Tasks

**Table 8: OOD accuracy on MathBench ($\star$: scored by the critic model; $\dagger$: based on the numbers reported by (Liu et al., ); A: application; T: theoretical).**
| model | Tool | Subset-A | Subset-T | ACC${}_{\text{average}}$ |
| --- | --- | --- | --- | --- |
| GPT-4-0125-Preview | $\times$ | 58.8^† | 78.4^† | 68.6^† |
| GLM4 | $\times$ | 51.3^† | 73.1^† | 62.2^† |
| Qwen-Chat-72B | $\times$ | 49.7^† | 77.2^† | 63.5^† |
| Math-InternLM2-20B | $\times$ | 41.9^† | 64.3^† | 53.1^† |
| Llama3${}_{\text{instruct}}$-8B | $\times$ | 36.7^† | 52.1^† | 44.4^† |
| MathCoder-7B | $\checkmark$ | 32.6^⋆ | 27.4^⋆ | 30.0^⋆ |
| MathCoder-34B | $\checkmark$ | 50.1^⋆ | 49.3^⋆ | 49.7^⋆ |
| ToRA-7B | $\checkmark$ | 31.0^⋆ | 28.4^⋆ | 29.7^⋆ |
| ToRA-70B | $\checkmark$ | 54.3^⋆ | 54.4^⋆ | 54.3^⋆ |
| Language-specific system prompt: |  |  |  |  |
| SIaM(Llama3${}_{\text{instruct}}$)0-8B | $\checkmark$ | 62.5^⋆ | 57.9^⋆ | 60.2^⋆ |
| SIaM(Llama3${}_{\text{instruct}}$)2-8B | $\checkmark$ | 66.7^⋆ | 62.6^⋆ | 64.6^⋆ |
| Chinese-only system prompt: |  |  |  |  |
| SIaM(Llama3${}_{\text{instruct}}$)0-8B | $\checkmark$ | 64.0^⋆ | 64.4^⋆ | 64.2^⋆ |
| SIaM(Llama3${}_{\text{instruct}}$)2-8B | $\checkmark$ | 69.5^⋆ | 65.8^⋆ | 67.6^⋆ |

Considering the above results in Section 3.5, we are now more confident in using the critic model to evaluate models’ performance on additional out-of-domain benchmarks, without the need to write extensive heuristics for different tasks. Besides CMATH, we evaluate the out-of-domain performance of our models using MathBench (Liu et al., ), a newly released benchmark supporting evaluation in both Chinese and English. The open-ended or multiple-choice questions in MathBench span various educational stages, from primary school to college levels. We report scores on its two subsets: MathBench-A, which evaluates practical problem-solving skills, and MathBench-T, which assesses theoretical understanding.

As shown in Table 8, the self-improved models demonstrate substantial gains on both subsets, with an accuracy improvement of $4.4\%$. On both subsets, the self-improved model consistently outperforms the initial one across all educational levels and subjects with notable improvements particularly in middle school tasks and English theoretical tasks. See sub-category performance in Tables 11 and 12. Note that we provide the scores of other models for reference, as they are judged by a different scorer. We compare our method with ToRA, one of the SOTA code-aided math LLMs, rated by the same critic model. Though trained on English-only data, ToRA exhibits a surprising performance on Chinese tasks. Nevertheless, our 8B model also outperforms ToRA 70B on the English subset of MathBench by $14.5\%$ and $9.2\%$, respectively, on A and T (Table 13). See more discussions and selection of system prompts in Section A.3. Compared to practical application questions, it seems that using CoT, LLMs are much better at handling theoretical knowledge questions. In contrast, solving all questions via coding shows balanced and reasonable performance. This shows the advantage of using tools to aid in computation, but also indicates the limitations of relying solely on code to address questions that may not require actual computation. It remains an open question whether, and how, code can be used to assist advanced theoretical reasoning–a topic beyond the scope of this paper (Liu et al., ).

## 4 Related Work

For automatic math evaluation on well-formatted benchmarks, previous studies mostly use heuristics and external tools (e.g., the Python eval() function) to compare answers and predictions (Fourrier et al., ; Gao et al., ), which works quite well for single numerical value answers, as seen in datasets such as GSM8K (Cobbe et al., ), ASDiv (Miao et al., ), and SVAMP (Patel et al., ). However, since answers from web resources are diverse in formats and language-code syntactic difference, using carefully designed task-specific heuristics becomes infeasible for comparing answers and code execution results. For datasets that are beyond value-style answers such as MATH (Hendrycks et al., ), closed source LLMs are also used for evaluation such as OpenAI-Evals, which however is not cost-effective for rating large-scale code samples.

Several approaches (Li et al., ; Yu et al., ; Lu et al., ; Yuan et al., ; Hu et al., ) use the LLM itself or use separate critic model (Ouyang et al., ; Xu et al., ) for scoring or filtering natural-language responses. We focus on tool-assisted assessment of code responses to math questions. Compared to the previously mentioned self-improving studies that use a single LLM to provide feedback on its own generations, we interpret “self” in contrast to distilling knowledge from larger closed-source or open-source LLMs for continuous improvements.

Though recent studies shown that in-domain CoT-based or code-based data augmentation and alignment will lead LLMs to achieve strong performance on in-domain math datasets (Luo et al., ; Yu et al., ; An et al., ; Li et al., ), we leave data augmentation on web data (CoT or code) for future work. We only use GPT-4 to annotate seed data and the critic data, and using closed-source LLMs to annotate large-scale web questions is beyond the scope of this paper.

## 5 Conclusions and Future Work

We introduce a novel paradigm for improving LLMs, which employs a code-based critic model to guide stages such as the creation and filtering of question-code data as well as complementary evaluation. We also investigate various alignment algorithms using self-generated instruction/preference data for further improvement. Results show the effectiveness of self-improving LLMs with this proposed paradigm. Future research includes studying post-training on code-only data to enhance the computational capabilities of LLMs and improvement of the critic model.

## Appendix A Appendices

### A.1 Backbone Comparisons for Initial Model Selection

Although QWEN2 also demonstrates strong performance, we use its math-specific variant to ensure the diversity of selected backbone models. For the same reason, and given the marginal performance difference between Llama3${}_{\text{instruct}}$ and Llama3${}_{\text{base}}$ when both are fine-tuned on $D_{0}$, we only Llama3${}_{\text{instruct}}$ for our experiments.

**Table 9: Accuracy across the development sets of math datasets of initial policy models based on different backbone models.**
|  |  |  | Chinese Tasks | English Tasks | Average |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | Size (B) | Tool | CM | APE | CMATH | GSM8K | MATH |  |
| CodeLlama | 7 | $\checkmark$ | 77.7 | 78.0 | 84.5 | 69.7 | 37.6 | 69.5 |
| QWEN${}_{\text{code}}$ | 7 | $\checkmark$ | 81.9 | 81.5 | 86.0 | 71.9 | 41.4 | 72.6 |
| Llama3.1${}_{\text{instruct}}$ | 8 | $\checkmark$ | 82.4 | 82.1 | 86.2 | 76.5 | 41.1 | 73.6 |
| Llama3${}_{\text{base}}$ | 8 | $\checkmark$ | 83.9 | 82.6 | 86.8 | 76.8 | 41.9 | 74.4 |
| Llama3${}_{\text{instruct}}$ | 8 | $\checkmark$ | 83.3 | 83.2 | 87.2 | 76.8 | 41.8 | 74.4 |
| DeepSeek${}_{\text{code}}$ | 7 | $\checkmark$ | 82.7 | 81.2 | 87.0 | 77.4 | 44.4 | 74.5 |
| QWEN2 | 7 | $\checkmark$ | 83.9 | 82.8 | 87.3 | 77.7 | 44.4 | 75.2 |
| QWEN2Math${}_{\text{instruct}}$ | 7 | $\checkmark$ | 84.9 | 83.4 | 87.3 | 79.5 | 48.0 | 76.6 |

### A.2 Impacts of Stages and Data Selection

**Table 10: Impacts of the stages and data selection on the development sets of datasets. The best performance for each model family is highlighted in bold.**
| Model | Stages | Data | CM | APE | CMATH | GSM8K | MATH | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| DeepSeek${}_{\text{code}}$ | SFT | $D_{0,\text{en}}$ | – | – | – | 74.6 | 43.8 | – |
|  | SFT | $D_{0,\text{zh}}$ | 81.0 | 82.4 | 86.8 | – | – | – |
|  | SFT | $D_{0}$ | 82.7 | 81.2 | 87.0 | 77.4 | 44.4 | 74.5 |
|  | SFT | $D_{0}$ + $D_{1}$ | 87.0 | 84.3 | 88.0 | 77.6 | 44.6 | 76.3 |
|  | SFT $\rightarrow$ DPO | $D_{0}$ + $D_{1}$; $D_{2,\text{WebInstruct}}$ | 87.0 | 84.4 | 88.2 | 78.2 | 44.4 | 76.5 |
|  | SFT $\rightarrow$ DPO | $D_{0}$ + $D_{1}$; $D_{2,\text{in-house}}$ | 87.3 | 85.9 | 91.2 | 78.9 | 45.9 | 77.8 |
| QWEN2Math${}_{\text{instruct}}$ | SFT | $D_{0,\text{en}}$ | – | – | – | 78.5 | 47.7 | – |
|  | SFT | $D_{0,\text{zh}}$ | 83.9 | 83.8 | 87.0 | – | – | – |
|  | SFT | $D_{0}$ | 84.9 | 83.4 | 87.3 | 79.5 | 48.0 | 76.6 |
|  | SFT | $D_{0}$ + $D_{1}$ | 87.8 | 85.9 | 88.3 | 79.2 | 49.5 | 78.1 |
|  | SFT $\rightarrow$ DPO | $D_{0}$ + $D_{1}$; $D_{2,\text{WebInstruct}}$ | 87.8 | 86.0 | 88.5 | 82.4 | 48.7 | 78.7 |
|  | SFT $\rightarrow$ DPO | $D_{0}$ + $D_{1}$; $D_{2,\text{in-house}}$ | 90.1 | 88.1 | 93.2 | 81.5 | 50.0 | 80.6 |

Ablation studies of the stages and data selection on the development sets of datasets.

### A.3 Sub-Type Performance on MathBench

The data presented in the tables clearly shows the advantage of SIaM(Llama3${}_{\text{instruct}}$)2 over SIaM(Llama3instruct)0 across various educational levels and subjects. For both the MathBench-A and MathBench-T datasets, SIaM(Llama3instruct)2 consistently outperforms SIaM(Llama3instruct)0. In the MathBench-A dataset, improvements are seen in all levels from Primary to College, with notable jumps in Middle and High school levels ($6.7\%$ and $7.0\%$ improvement, respectively, in Table 11). Similarly, the MathBench-T dataset shows improvement across all levels, particularly in the Middle school and English categories, which demonstrate $8.1\%$ and $10.5\%$ increases, respectively. These results indicate that SIaM(Llama3instruct)2 provides enhanced accuracy in out-of-distribution scenarios, making it a more reliable choice for varied educational levels.

In the seed data $D_{0}$, we use a language-specific system prompt for each English instance: *“Please write a python code to solve the following questions”*. For the Chinese subset of $D_{0}$ and all instances in $D_{1}$ and $D_{2}$ — which are exclusively Chinese data — we use a consistent Chinese system prompt “请用python代码解决以下问题” (*“Please write a python code to solve the following questions”*). When evaluating our self-improved model on MathBench, we observe that it performs better when the Chinese system prompt is applied to solve English questions (Table 12). This may be due to the fact that our training data primarily consists of Chinese data with Chinese system prompts.

We compare with SOTA code-assisted models trained on augmented MATH and GSM8K datasets ToRA and MathCoder. Before detailed comparisons, we first review the background of the use of code for mathematical reasoning. Code can be used either directly (Chen et al., ; Gao et al., ) (code-only) or interactively (Wang et al., ) during problem-solving. The latter approaches such as ToRA and MathCoder jointly solve problems using CoT explanation and code. One advantage of these interactive methods over code-only methods is that the final step of their solution is usually written in CoT, allowing the easy use of existing scripts designed for CoT-style benchmarks for evaluation. However, this does not allow for robust comparisons for unseen diverse-format comparisons. In addition, the role of using tools multiple times to address a single math problem is unclear based on the performance difference of interactive methods (Table 3). For example, ToRA needs $1.02$ tool interaction rounds per question while MathCoder requires $2.05$ for MATH and GSM8K. This work focuses on the direct usage of code as a case study to avoid multi-step inference and leave the interactive setting for future studies.

For ToRA 7B(^2^22https://huggingface.co/llm-agents/tora-code-7b-v1.0.) and 70B(^3^33https://huggingface.co/llm-agents/tora-70b-v1.0.) models, we use their official inference scripts.(^4^44https://github.com/microsoft/ToRA/tree/main.) On MathBench, ToRA needs an average of $1.00$ and $1.01$ tool interaction rounds per question. It seems its final CoT reasoning primarily focuses on adjusting formatting answers to fully leverage existing CoT evaluation scripts. We use ToRA’s generated code and execution result, keeping the rest of the inputs for the critic model the same. We also experiment with replacing the execution results with the CoT outputs, but this does not result in significant changes. Our self-improved 8B model outperforms one SOTA code-assisted model, ToRA-70B, across all subcategories on this OOD dataset (Table 13).

For MathCoder, we evaluate its best-performing 34B model(^5^55https://huggingface.co/MathLLMs/MathCoder-CL-34B.) and 7B model(^6^66https://huggingface.co/MathLLMs/MathCoder-CL-7B.), which needs $1.53$ and $2.13$ tool interaction rounds per question, respectively. We also use their released inference scripts(^7^77https://github.com/mathllm/MathCoder.) and follow the data format.

**Table 11: Fine-grained OOD accuracy on the MathBench dataset scored by the critic model using language-specific system prompts.**
|  | MathBench-A | MathBench-T |  |  |
| --- | --- | --- | --- | --- |
| Level | SIaM(Llama3${}_{\text{instruct}}$)0 | SIaM(Llama3${}_{\text{instruct}}$)2 | SIaM(Llama3${}_{\text{instruct}}$)0 | SIaM(Llama3${}_{\text{instruct}}$)2 |
| Arith | 98.0 | 99.0 | – | – |
| Primary | 75.7 | 80.7 | 66.6 | 67.5 |
| Middle | 56.3 | 63.0 | 60.1 | 68.2 |
| High | 50.3 | 57.3 | 59.1 | 60.6 |
| College | 32.0 | 33.3 | 50.2 | 57.9 |
| Chinese | 56.8 | 63.5 | 62.7 | 63.6 |
| English | 66.2 | 68.8 | 50.6 | 61.1 |

**Table 12: Fine-grained OOD accuracy on the MathBench dataset scored by the critic model using a Chinese-only system prompt.**
|  | MathBench-A | MathBench-T |  |  |
| --- | --- | --- | --- | --- |
| Level | SIaM(Llama3${}_{\text{instruct}}$)0 | SIaM(Llama3${}_{\text{instruct}}$)2 | SIaM(Llama3${}_{\text{instruct}}$)0 | SIaM(Llama3${}_{\text{instruct}}$)2 |
| Arith | 97.3 | 98.3 | – | – |
| Primary | 71.0 | 79.0 | 71.6 | 71.3 |
| Middle | 60.0 | 69.0 | 70.3 | 71.5 |
| High | 54.0 | 59.7 | 61.8 | 62.3 |
| College | 37.7 | 41.3 | 59.2 | 62.6 |
| Chinese | 57.3 | 63.7 | 63.8 | 63.4 |
| English | 68.4 | 73.3 | 65.4 | 69.5 |

**Table 13: Fine-grained OOD accuracy of ToRA (70B and 7B) and MathCoder (34B and 7B) on the MathBench dataset scored by the critic model (T: ToRA; M: MathCoder).**
|  | MathBench-A | MathBench-T |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Level | T-7B | T-70B | M-7B | M-34B | T-7B | T-70B | M-7B | M-34B |
| Arith | 39.3 | 82.7 | 40.7 | 66.3 | – | – | – | – |
| Primary | 40.3 | 77.7 | 43.3 | 70.0 | 30.9 | 53.9 | 26.2 | 47.0 |
| Middle | 24.3 | 39.7 | 28.7 | 45.3 | 31.0 | 57.6 | 25.0 | 46.8 |
| High | 30.0 | 39.7 | 29.7 | 39.0 | 28.0 | 51.9 | 28.2 | 47.8 |
| College | 21.0 | 31.7 | 20.7 | 30.0 | 25.5 | 55.1 | 29.0 | 54.3 |
| Chinese | 28.2 | 47.5 | 23.0 | 41.5 | 26.5 | 50.5 | 25.4 | 43.8 |
| English | 32.9 | 58.8 | 39.0 | 55.9 | 31.2 | 60.3 | 30.4 | 57.5 |

### A.4 Data Statistics

**Table 14: Statistics of evaluation benchmarks. Note that in our experiments, we do not use any rationale in these datasets as we focus on solving problems via coding. We only use the questions and short-form answers from the training set of MATH and GSM8K for constructing the seed data, and we use the questions and short-form answer from the training set of APE and CM for constructing the data for self-improvement.**
| Dataset | Language | Answer Type | Level | Training | Validation |
| --- | --- | --- | --- | --- | --- |
| APE (Zhao et al., ) | zh | numerical value | elementary | 200,488 | 5,000 |
| CM (Qin et al., ) | zh | numerical value(s) | grades 6—12 | 13,628 | 1,703 |
| CMATH (Wei et al., ) | zh | numerical value | elementary | – | 600 |
| MathBench (Liu et al., ) | en, zh | mixed | from primary to college | – | 3,709 |
| MATH (Hendrycks et al., ) | en | mixed | college | 7,500 | 5,000 |
| GSM8K (Cobbe et al., ) | en | numerical value | elementary | 7,473 | 1,319 |

**Table 15: An example instance of the APE dataset (Zhao et al., ) (we translate the question into English; $\star$: we do not use this rationale in our paradigm).**
| Question: | Given: Apples cost 6 yuan for 4 kilograms, and oranges cost 11 yuan for 5 kilograms. Uncle Wang buys 16 kilograms of apples and 20 kilograms of oranges. How much should he pay in total? |
| --- | --- |
| Answer: | 68 |
| Rationale^⋆: | x=6/4*16+11/5*20 |

### A.5 Other Alignment Algorithms

**Table 16: The self-improving performance in different stages on the development sets of different datasets. The best open-sourced performance for each backbone model is highlighted in bold.**
| Model | Alignment | Data | CM | APE | CMATH | GSM8K | MATH | ACC${}_{\text{average}}$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| DeepSeek${}_{\text{code}}$ | SFT | $D_{0}$ + $D_{1}$ | 87.0 | 84.3 | 88.0 | 77.6 | 44.6 | 76.3 |
|  | SFT $\rightarrow$ ORPO | $D_{0}$ + $D_{1}$; $D_{2}$ | 87.7 | 85.5 | 91.2 | 76.5 | 44.5 | 77.1 |
|  | SFT $\rightarrow$ DPO | $D_{0}$ + $D_{1}$; $D_{2}$ | 87.3 | 85.9 | 91.2 | 78.9 | 45.9 | 77.8 |
| Llama3${}_{\text{instruct}}$ | SFT | $D_{0}$ + $D_{1}$ | 87.6 | 85.0 | 89.0 | 76.6 | 41.8 | 76.0 |
|  | SFT $\rightarrow$ ORPO | $D_{0}$ + $D_{1}$; $D_{2}$ | 88.0 | 86.4 | 91.8 | 76.4 | 42.1 | 77.0 |
|  | SFT $\rightarrow$ DPO | $D_{0}$ + $D_{1}$; $D_{2}$ | 89.0 | 86.8 | 90.8 | 80.5 | 41.9 | 77.8 |

As shown in Table 16, DPO demonstrates superior performance compared to ORPO, both with the SFT loss. We leave the exploration of more length-regularized alignment algorithms and the role of the reference policy model in preference optimization to future studies.

$$ $\begin{split}L_{\text{ORPO}}(\pi_{\theta_{k+1}})=&-\lambda\cdot\log\sigma\left (\log\frac{\pi_{\theta_{k+1}}(y_{w}\mid x)}{1-\pi_{\theta_{k+1}}(y_{w}\mid x)} \right.\\ &\left.-\log\frac{\pi_{\theta_{k+1}}(y_{l}\mid x)}{1-\pi_{\theta_{k+1}}(y_{l} \mid x)}\right)\\ &-\log\pi_{\theta_{k+1}}(y_{w}\mid x)\end{split}$ (7) $$

### A.6 Self-Generated Preference Data

Existing preference datasets (Cui et al., ; Weyssow et al., ) do not include many these kinds of inter-disciplinary annotations. We use the current policy to generate preference data. Examples are provided in Table A.6 and Table A.6.

**Table 17: Examples of self-generated preference pairs based on in-house web QA data (I). Texts have been translated into English.**
| question | The perimeter of an isosceles triangle is 20, and the ratio of the lengths of the two equal sides is 1:2. What is the length of the base? A.10 B.4 C.4 or 10 D.5 or 8 |
| --- | --- |
| reference answer | B |
| winning code (w) | [⬇](data:text/plain;base64,ZnJvbSBzeW1weSBpbXBvcnQgc3ltYm9scywgRXEsIHNvbHZlCgojIERlZmluZSB0aGUgdW5rbm93bgphLCBiID0gc3ltYm9scygnYSBiJykKCiMgRXN0YWJsaXNoIHRoZSBlcXVhdGlvbgojIFRoZSBoeXBvdGVudXNlIGlzIHR3aWNlIHRoZSBiYXNlCmVxdWF0aW9uMSA9IEVxKGEsIDIqYikKCiMgVGhlIHBlcmltZXRlciBpcyAyMAplcXVhdGlvbjIgPSBFcSgyKmEgKyBiLCAyMCkKCiMgU29sdmUgdGhlIGVxdWF0aW9uCnNvbHV0aW9uID0gc29sdmUoKGVxdWF0aW9uMSwgZXF1YXRpb24yKSwgKGEsIGIpLCBkaWN0PVRydWUpCnNvbHV0aW9u)<br>from sympy import symbols, Eq, solve<br># Define the unknown<br>a, b = symbols(’a b’)<br># Establish the equation<br># The hypotenuse is twice the base<br>equation1 = Eq(a, 2*b)<br># The perimeter is 20<br>equation2 = Eq(2*a + b, 20)<br># Solve the equation<br>solution = solve((equation1, equation2), (a, b), dict=True)<br>solution |
| code result of w | [a: 8, b: 4] |
| losing code (l) | [⬇](data:text/plain;base64,ZnJvbSBzeW1weSBpbXBvcnQgc3ltYm9scywgRXEsIHNvbHZlCgojIERlZmluZSB0aGUgdW5rbm93biB2YXJpYWJsZQphID0gc3ltYm9scygnYScpCgojIEVzdGFibGlzaCB0aGUgZXF1YXRpb24KZXF1YXRpb24gPSBFcSgyICogYSArIDIgKiAoMiAqIGEpLCAyMCkKCiMgU29sdmUgdGhlIGVxdWF0aW9uCnNvbHV0aW9uID0gc29sdmUoZXF1YXRpb24sIGRpY3Q9VHJ1ZSkKc29sdXRpb24=)<br>from sympy import symbols, Eq, solve<br># Define the unknown variable<br>a = symbols(’a’)<br># Establish the equation<br>equation = Eq(2 * a + 2 * (2 * a), 20)<br># Solve the equation<br>solution = solve(equation, dict=True)<br>solution |
| code result of l | [a: 10/3] |

## Limitations

Language Diversity of Resources: in this paper, we focus on large-scale question-answer pairs from educational websites for Chinese, and accordingly, our critic model used for guiding self-improvement is trained on Chinese data. While considering resources in other languages such as English could enhance the the generalizability of LLMs, it would require extensive human efforts for data collection and cleaning, which is beyond the scope of this work. On the other hand, since the backbone LLMs are pre-trained and aligned on multi-lingual data and our seed data includes English instruction data, the initial policy already exhibits reasonable performance on in-domain (Section 3.3) and out-domain benchmarks (Section 3.6). Self-improving this initial policy model on Chinese data may even improve its performance on English tasks. Finally, experiments show that the critic model is as effective at rating English responses as rating Chinese ones (Section 3.5).

Copyright of Resources: The large-scale question-answer pairs (excluding APE and CM) are either collected or purchased from educational websites. We will not release the full-scale resources. Instead, we will provide question-answer samples, along with all scripts, seed data in English, initial and self-improved policy models, self-generated preference data, and the critic model to facilitate future studies.

LLM Scalings: Due to limited computational resources, our experiments focus on 7-8B LLMs. Generally, improving the math reasoning abilities of relatively small LLMs requires a large amount of training data and knowledge distillation (Li et al., ; Shao et al., ), which may not be necessary for larger LLMs.

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: - You may want to fix your typos Table '??' in A.4 in line 850.

- In Table 3, the scores for the initial model QWEN2Math_instruct on the English benchmarks are GSM8K: 79.5 and MATH: 48.0, which significantly differ from the official blog scores of GSM8K: 89.9 and MATH: 75.1. Could you please clarify the discrepancy between your reported scores and the official blog scores? Additionally, could you please give an explanation for why tools are indicated for QWEN2Math in the table, given its post-training with COT data?
soundness: 2
strengths: - The authors propose a method to continuously self-improve LLMs by utilizing well-formatted math data from the web, focusing on the efficient comparison of reference answers and code execution results.

- A critic model is introduced to evaluate the performance of LLMs. This model is designed to handle diverse answer formats and improve the assessment of code responses, thereby enhancing the overall evaluation process.
summary: The paper introduces a novel approach to enhancing large language models (LLMs) for solving mathematical problems via coding. The authors propose using large-scale, expert-written, diverse math question-answer pairs. Their framework employs a code-based critic model to guide question-code dataset construction, quality control, and evaluation. They also explore alignment algorithms using self-generated instruction data to foster self-improvement.
weaknesses: - The self-improvement pipeline proposed in this paper is not particularly novel; it combines rejection sampling fine-tuning and DPO, mirroring approaches in [1] and [2], but with specific adaptations for code data. All these methods generate multiple model responses, selecting high-quality responses for SFT training data and constructing good-bad response pairs for DPO training data. While this paper generalizes these techniques to large-scale web QA pairs and code data for mathematical reasoning, it does not present any particularly new conclusions.

- The proposed method appears to be data-inefficient, utilizing over 1,000K samples yet producing suboptimal results. For instance, accuracy rates of 80.5% and 41.9% are reported on the MATH and GSM8K datasets, respectively. In contrast, other works, such as Dart-Math (which samples additional code responses from Deepseek-MATH-RL for difficult questions) [3], also using Llama3 as a base model, achieve accuracies of 81.1% and 46.6% with only 590K query-response pairs. The discrepancy in performance may be attributed to differences in response style, quality, and other factors.
 
[1] ChatGLM-Math: Improving Math Problem-Solving in Large Language Models with a Self-Critique Pipeline.

[2] ReFT: Reasoning with Reinforced Fine-Tuning.

[3] DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solving.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: N/A
soundness: 2
strengths: Experiments are conducted both on English and Chinese
summary: The main purpose this paper is that previous methods for improving models to solve mathematical problems with code have largely relied on more powerful models to generate data and only used a limited number of datasets, such as GSM8k and MATH. This paper proposes using a critical model to compare the output of code execution with the golden answers. The amount of data used in this paper is at least an order of magnitude greater than that used in datasets like GSM8k and MATH.
weaknesses: 1. The paper lacks clarity overall, with the abstract and introduction failing to clearly explain the specific problem the authors are attempting to address.

2. Although the authors propose using a critic model to enhance the model's capabilities, they do not adequately justify the necessity of this approach. The primary purpose of incorporating code execution is to improve the model's ability to solve problems. However, Table 3 appears to lack a critical baseline—namely, the results of fine-tuning the model directly with the large amount of collected data. This omission leaves the purpose of this approach somewhat unclear.

3. The proposed method in the paper lacks innovation, appearing more as an experimental report rather than introducing any genuinely novel approach.

Grammars:
1. In Line 177 lacks definition of $j$

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. The generalization of this paper largely relies on data from other domains, such as APE310K and CM, which also require golden labels. I believe a fairer comparison would involve training traditional methods like RFT or V-STAR on these datasets as baselines before comparing them to the proposed SIaM.

2. Regarding the advantages of RFT within the TIR context, it is unclear what specific improvements are achieved over traditional RFT. What is the rationale for utilizing TIR, and what unique benefits arise from the integration of these two methods? Additionally, Tool-Augmented approaches may focus more on achieving correct results rather than detailing the reasoning process. Is there a significant difference compared to directly using Chain-of-Thought (COT) techniques? I suggest conducting experiments to validate this.

3. For more complex reasoning tasks, such as MATH, any observed performance improvements appear marginal. The authors need to provide more comprehensive explanations rather than attributing these variations solely to evaluation scripts. Are there instances where code implementation might not be feasible? How might these situations limit the accuracy and scalability of the proposed methods?
soundness: 3
strengths: 1. The paper effectively employs an iterative refinement technique, utilizing a substantial number of QA pairs to enhance the mathematical reasoning capabilities of large language models.

2. The main experiments conducted are solid, with validation across multiple datasets and models that support the reliability of the findings.

3. The proposed method exhibits strong scalability, indicating its potential for broader applications in the field.
summary: The paper introduces a code-assisted self-improving method, which utilizes a critic model to verify the correctness of the implementation result of the code generated by the policy model, iteratively optimizing the policy model for large-scale web QA. The study evaluates a wide range of models and conducts extensive experiments on both in-domain and out-of-domain datasets, demonstrating the effectiveness of SIaM and its generalization capabilities.
weaknesses: 1. Despite presenting a practical framework for model training, the paper's novelty is somewhat limited. The methodology closely resembles existing approaches such as RFT, STAR, and V-STAR, with the primary distinction being its application to code-augmented COT.

2. Additionally, The motivation behind the study is not sufficiently compelling. The results seem somewhat predictable; adding TIR to traditional COT is generally expected to enhance performance. It would be beneficial for the authors to clearly explain the specific advantages of their code-augmented COT over traditional COTs with RFT. Consider additional experiments that could highlight these differences.

3. Given that the critic model is trained using GPT-4, the training labels are not guaranteed to be correct, which may affect the performance of the critic model. The authors should further clarify how their approach compares to exact match metrics to substantiate the reliability of the critic model's evaluations.