WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct

## Abstract

Abstract Large language models (LLMs), such as GPT-4, have shown remarkable performance in natural language processing (NLP) tasks, including challenging mathematical reasoning. However, most existing open-source models are only pre-trained on large-scale internet data and without math-related optimization. In this paper, we present WizardMath , which enhances the mathematical reasoning abilities of Llama-2, by applying our proposed Reinforcement Learning from Evol-Instruct Feedback ( RLEIF ) method to the domain of math. Through extensive experiments on two mathematical reasoning benchmarks, namely GSM8k and MATH, we reveal the extraordinary capabilities of our model. WizardMath surpasses all other open-source LLMs by a substantial margin. Furthermore, our model even outperforms ChatGPT-3.5, Claude Instant-1, PaLM-2 and Minerva on GSM8k, simultaneously surpasses Text-davinci-002, PaLM-1 and GPT-3 on MATH. More details and model weights are public at https://github.com/nlpxucan/WizardLM 1 1 1 We are working with our legal team to review and publicly release the code and data in accordance with our policy. and https://huggingface.co/WizardLM .

## 1 Introduction

Recently, Large-scale language models (LLMs) have garnered significant attention and become the go-to approach for numerous natural language processing (NLP) tasks, including open domain conversation [, , , ], coding [, , , , , , , , ] and math [, , , , , ]. A conspicuous example is ChatGPT, developed by OpenAI. This model uses extensive pre-training on large-scale internet data and further fine-tuning with specific instruction data and methods. As a result, it achieves state-of-the-art zero-shot performance on various benchmarks. Subsequently, Anthropic, Google, and Meta also launched their competitive products one after another. Notably, Meta’s series of Llama [, ] models have sparked an open-source revolution and quickly narrowed the gap with those closed-source LLMs. This trend also gradually stimulates the releases of MPT[6](#footnote6), Falcon [], StarCoder [], Alpaca [], Vicuna [], and WizardLM [], etc. However, these open models still struggles with the scenarios which require complex multi-step quantitative reasoning, such as solving mathematical and science challenges [, , , , , , , , , , ].

Figure: Figure 1: A diagram illustrating the three steps of our *Reinforcement Learning from Evol-Instruct Feedback* (*RLEIF*): (1) supervised fine-tuning (SFT), (2) Instruction Reward Model (IRM) training and Process-supervised Reward Model (PRM) training, and (3) Active Evol-Instruct and reinforcement learning via proximal policy optimization (PPO).
Refer to caption: /html/2308.09583/assets/x1.png

Chain-of-thought (CoT) [] proposes to design better prompts to generate step-by-step solutions, which can lead to improved performance. Self-Consistency [] also achieves remarkable performance on many reasoning benchmarks, which generates several possible answers from the model and selects the correct one based on majority vote []. In recent, [] finds that process supervision with reinforcement learning significantly outperforms outcome supervision for solving challenging MATH problems.

Inspired by *Evol-Instruct* and Process-supervised Reinforcement Learning, this work aims to enhance the mathematical reasoning abilities of the SOTA open-source LLM, Llama-2 []. As shown in the Figure [1](#S1.F1), we propose a new method named *Reinforcement Learning from Evol-Instruct Feedback* (*RLEIF*), which could firstly generate diverse math instructions data by math-specific *Evol-Instruct*, then we train an instruction reward model (IRM) and a process-supervised reward model (PRM) [, , , , , , ], the former indicates the quality of the evolved instruction and the later receives feedback for each step in the solution. The brand-new *Evol-Instruct* method includes two downward evolution and upward evolution progress to produce the grade school math and challenging math respectively. Initially, we re-generate, filter and finetune the original math instruction data from GSM8k [] and MATH []. Immediately, we train the Llama-2 models to obtain the reward models and our *WizardMath*.

We perform experiments on two mathematical reasoning benchmarks, namely GSM8k [] and MATH [], the results demonstrate that our *WizardMath* outperforms all other open-source LLMs, achieving state-of-the-art performance. Specifically, *WizardMath* observe a substantial improvement in pass@1 with an increase of +24.8 (81.6. vs. 56.8) on GSM8k, and +9.2 (22.7 vs. 13.5) on MATH. Notably, our model even also significantly surpasses OpenAI’s ChatGPT-3.5[3](#footnote3), Anthropic’s Claude Instant-1 [], and Google’s PaLM-2 [] in terms of pass@1 on GSM8k.

The main contributions of this work are as following:

- •
We introduce *WizardMath* model, which enhances the mathematical reasoning abilities for open-source pretrained large language model Llama-2 [].
- •
We propose a new method, *Reinforcement Learning from Evol-Instruct Feedback* (*RLEIF*), alongside *Evol-Instruct* and Reinforcement Learning, for improving LLM reasoning performance.
- •
*WizardMath* surpasses all other open-source LLMs by a substantial margin in terms of mathematical reasoning, including Llama-2 70B [], Llama-1 65B [], Falcon-40B [], MPT-30B[6](#footnote6), Baichuan-13B Chat[7](#footnote7) and ChatGLM2 12B [] on both GSM8k [] and MATH [].
- •
*WizardMath* significantly outperforms various main closed-source LLMs, such as ChatGPT[3](#footnote3), GPT-3.5, Claude Instant [], PaLM-2 [], PaLM-1 [] and Minerva[] on GSM8k.

## 2 Method

In this section, we elaborate on the details of our *WizardMath*. Following WizardLM and PRMs[], we propose *Reinforcement Learning from Evol-Instruct Feedback* (*RLEIF*), which integrates the *Evol-Instruct* and reinforced process supervision method to evolve GSM8k and MATH, and fine-tune the pre-trained Llama-2 with the evolved data and reward models.

As shown in the Figure  [1](#S1.F1), our methods apply three steps:

- 1.
Supervised fine-tuning.
- 2.
Training instruction reward model, and process-supervised reward model.
- 3.
Active Evol-Instruct, and PPO training.

### 2.1 Supervised fine-tuning

Following InstructGPT[], we also firstly fine tune the base with supervised instruction-response pairs, which contains:

- 1.
To make the parsing of each step easier, we few-shot re-generate 15k answers for GSM8k and MATH with an Alpha version of WizardLM 70B model to produce solutions in a step-by-step format, then find out those with a correct answer, and use this data to finetune base Llama model.
- 2.
To enhance the model’s ability to adhere to the neural and diverse instructions, we also sample 1.5k open-domain conversations from WizardLM’s training data, then merge it with above math corpus as the final SFT training data.

### 2.2 Evol-Instruct principles for math

Motivated by the Evol-Instruct [] method proposed by WiazrdLM and its effective application on WizardCoder [], this work attempts to make math instructions with various complexities and diversity to enhance the pre-trained LLMs. Specifically, we adapt Evol-Instruct to a new paradigm including two evolution lines:

- 1.
Downward evolution: It enhances instructions by making the questions easier. For example i): revising high difficulty questions to lower difficulty, or ii) producing a new and easier question with another different topic.
- 2.
Upward evolution: Derived from original Evol-Instruct method, it deepens and generates new and harder questions by i) adding more constraints, ii) concretizing, iii) increasing reasoning.

### 2.3 Reinforcement Learning from Evol-Instruct Feedback ( RLEIF )

Inspired by InstructGPT[] and PRMs[], we train two reward models to predict the quality of the instructions and the correctness of each step in the answer respectively:

- 1.
Instruction Reward Model (IRM): This model aims to judge the quality of the evolved instructions on three aspects: i) Definition, ii) Precision, and iii) Integrity. To produce the ranking list training data of IRM, for each instruction, we firstly use ChatGPT and Wizard-E (^2^22 Wizard-E named Wizard-Evol-Generator, which is an Alpha version fine-tuned Llama model specifically used to execute *Evol-Instruct* without APIs.) to generate 2~4 evolved instructions respectively. Then we leverage Wizard-E to rank the quality of those 4~8 instructions.
- 2.
Process-supervised Reward Model (PRM): As there is no powerful open-source math reasoning LLMs before this work, there is no simple way to support highly precise process supervision without professional human-labelers and close-source ChatGPT. Therefore, we depend on ChatGPT to provide process supervision, and ask it to assess the correctness of each step in the solutions generated by our model.
- 3.
PPO training. We evolve the original math (GSM8k + MATH) instructions by 8 turns, increasing the data size from 15k to 96k. We use IRM and PRM to generate the instruction reward (${r^{I}}$) and the answer reward (${r^{A}}$). Then apply a product as the final reward $r={r^{I}}\cdot{r^{A}}$.

Figure: Figure 2: The pass@1 performance of main LLM models on the GSM8k benchmark, our model is currently ranked in the top five, slightly outperforming some close-source models such as ChatGPT-3.5[3](#footnote3), Claude Instant-1[4](#footnote4), PaLM 2 [], and substantially surpassing all open-source models.
Refer to caption: /html/2308.09583/assets/Figures/gsm8k_sort_all.png

**Table 1: Results of pass@1 (%) on GSM8k and MATH. In this study, to ensure equitable and cohesive evaluations, we report the socres of all models within the settings of greedy decoding and CoT []. We report the improvement between WizardMath and baseline model with similar parameter size.**
| Model | Params | GSM8k | MATH |
| --- | --- | --- | --- |
| Closed-source models |  |  |  |
| GPT-4  [] | - | 92.0 | 42.5 |
| Claude 2[5](#footnote5) | - | 88.0 | - |
| Claude 1.3[5](#footnote5) | - | 85.2 | - |
| Flan-PaLM 2 [] | 540B | 84.7 | 33.2 |
| Claude Instant[5](#footnote5) | - | 80.9 | - |
| ChatGPT [] | - | 80.8 | 34.1 |
| PaLM 2 [] | 540B | 80.7 | 34.3 |
| Minerva [] | 8B | 16.2 | 14.1 |
| 62B | 52.4 | 27.6 |  |
| 540B | 58.8 | 33.6 |  |
| GPT-3.5 [] | - | 57.1 | - |
| PaLM [] | 8B | 4.1 | 1.5 |
| 62B | 33.0 | 4.4 |  |
| 540B | 56.5 | 8.8 |  |
| RFT-13B  [] | 13B | 55.4 | - |
| Chinchilla  [] | 70B | 43.7 | - |
| ChatGLM 2 [] | 12B | 40.9 | - |
| Text-davinci-002  [] | 175B | 40.7 | 19.1 |
| GPT-3 [] | 175B | 34.0 | 5.2 |
| GPT-2 [] | 1.5B | - | 6.9 |
| Open-source models |  |  |  |
| GAL [] | 30B | - | 12.7 |
| 120B | - | 20.4 |  |
| LLaMA 2 [] | 7B | 14.6 | 2.5 |
| 13B | 28.7 | 3.9 |  |
| 34B | 42.2 | 6.24 |  |
| 70B | 56.8 | 13.5 |  |
| Qwen [8](#footnote8) | 7B | 51.6 | - |
| LLaMA 1 [] | 7B | 11.0 | 2.9 |
| 13B | 17.8 | 3.9 |  |
| 33B | 35.6 | 7.1 |  |
| 65B | 50.9 | 10.6 |  |
| RFT-7B  [] | 7B | 50.3 | - |
| GPT-J-6B  [] | 6B | 34.9 | - |
| ChatGLM 2 [] | 6B | 32.4 | - |
| InternLM-7B  [] | 7B | 31.2 | - |
| Vicuna v1.3  [] | 13B | 27.6 | - |
| Baichuan-chat [7](#footnote7) | 13B | 23.9 | - |
| Falcon [] | 7B | 6.8 | 2.3 |
| 40B | 19.6 | 2.5 |  |
| GPT-Neo-2.7B  [] | 2.7B | 19.5 | - |
| MPT[6](#footnote6) | 7B | 6.8 | 3.0 |
| 30B | 15.2 | 3.1 |  |
| *WizardMath* | 7B | 54.9 (+3.3) | 10.7 (+7.7) |
| *WizardMath* | 13B | 63.9 (+35.2) | 14.0 (+10.1) |
| *WizardMath* | 70B | 81.6 (+24.8) | 22.7 (+9.2) |

**Table 2: Results of pass@1 (%) on MATH Subtopics with WizardMath 70B model.**
| MATH subtopics | WizardMath 70B |
| --- | --- |
| Intermediate Algebra | 7.1 |
| Precalculus | 12.6 |
| Geometry | 15.7 |
| Number Theory | 16.3 |
| Counting & Probability | 17.3 |
| Prealgebra | 41.7 |
| Algebra | 33.3 |
| Overall | 22.7 |

## 3 Experiment

This section provides a comprehensive overview of the baseline models in our experiments. Subsequently, we mainly elucidate the performance metrics of our models on two prevalent mathematical benchmarks: GSM8k [] and MATH [].

### 3.1 Baselines

#### Close-Source Models.

Numerous technology companies have effectively created exceptionally proficient Large Language Models (LLMs) [, , , , , , , , , ], but have opted against making them publicly available, so they are referred to as close-source models. In our research, we extensively integrate a significant number of close-source models as the foundational benchmarks. Specifically, our baselines encompass the following models: (i) OpenAI’s GPT-3  [], GPT-3.5, ChatGPT(^3^33 [https://openai.com/](https://openai.com/)), GPT-4 []; (ii) Google’s PaLM 2 [], PaLM [], and Minerva []; (iii) Anthropic’s Claude Instant [], Claude 1.3(^4^44 [https://www.anthropic.com/index/introducing-claude](https://www.anthropic.com/index/introducing-claude)), Claude 2(^5^55 [https://www.anthropic.com/index/claude-2](https://www.anthropic.com/index/claude-2)), DeepMind’s Chinchilla [].

#### Open-Source Models.

Massive open-source LLMs [, , , , , , , ] have been accessible to the AI community. Nonetheless, their performance consistently tends to significantly lag behind the close-source models. As part of our research, we incorporate a significant number of these open-source models as our baselines, which mainly contain the following: Llama 1 [] & Llama 2 [], GAL [], GPT-J [], GPT-Neo [], Vicuna [], MPT(^6^66 [https://github.com/mosaicml/llm-foundry/](https://github.com/mosaicml/llm-foundry/)), Falcon[], Baichuan(^7^77 [https://github.com/baichuan-inc/Baichuan-13B](https://github.com/baichuan-inc/Baichuan-13B)), ChatGLM [], Qwen(^8^88 [https://github.com/QwenLM/Qwen-7B/](https://github.com/QwenLM/Qwen-7B/)) and RFT [].

### 3.2 Evaluate Benchmarks

We mainly evaluate WizardMath on two benchmarks (GSM8k [] and MATH []). The GSM8k [] dataset contains approximately 7500 training data and 1319 test data, mainly on grade school level math problems, each of which consists of basic arithmetic operations (addition, subtraction, multiplication, and division), and generally requires 2 to 8 steps to solve. The MATH [] dataset collects math problems from prestigious math competitions such as AMC 10, AMC 12, and AIME. It contains 7500 training data and 5,000 challenging test data in seven academic areas: Prealgebra, Algebra, Number Theory, Counting and Probability, Geometry, Intermediate Algebra, and Precalculus. Furthermore, these problems are divided into five levels of difficulty, with ‘1’ denoting the relatively lower difficulty level and ‘5’ indicating the highest level.

### 3.3 Train and Evaluation prompt

The Llama 2 [] base serves as our foundation model.

We undertake the training of our *WizardMath* by employing the prompt from Alpaca []:

We evaluate GSM8k [] and MATH benchmarks [] by employing the following CoT [] prompt:

### 3.4 Evaluation on GSM8k and MATH

Notably, in the Figure [2](#S2.F2) and Table [1](#S2.T1), we cite the metrics of GPT-4 and GPT-3.5 from  []. The evaluation of the ChatGPT model’s scores are from  []. For the assessment of Claude Instant, Claude 1.3, and Claude 2, the scores are extracted from [5](#footnote5). The scores of PaLM 1, PaLM 2, and Minerva are garnered from  [, , ]. Finally, the scores associated with Text-davinci-002, GPT-3 and GPT-2 are garnered from  [, ]. On the open-source models, most scores are retrieved from the paper of Llama 2 [] or their self-reports. Additionally, we evaluate the Baichuan-chat, Vicuna v1.3 by ourselves. In the Table [2](#S2.T2), we show the detailed results of MATH subtopics with our WizardMath 70B model.

#### Comparing with the Close-Source Models.

In Table [1](#S2.T1), our *WizardMath 70B* slightly outperforms some close-source LLMs on GSM8k, including ChatGPT, Claude Instant and PaLM 2 540B. And as shown in Figure [2](#S2.F2), our model is
currently ranked in the top five on all models. Simultaneously,*WizardMath 70B* also surpasses the Text-davinci-002 on MATH. The detailed results are as follows:

- 1.
*WizardMath 13B* outperforms PaLM 1 540B (63.9 vs 56.5), Minerva 540B (63.9 vs 58.8), and GPT-3.5 (63.9 vs 57.1) on GSM8k. Meanwhile,it surpasses PaLM 1 540B (14.0 vs. 8.8), GPT-3 175B (14.0 vs. 5.2) on MATH.
- 2.
*WizardMath 70B*, our largest model, achieves the superior or comparable performance with Claude Instant (81.6 vs 80.9), ChatGPT (81.6 vs 80.8) and PaLM 2 (81.6 vs 80.7) on GSM8k. Concurrently, WizardMath 70B also exceeds Text-davinci-002 (22.7 vs. 19.1) by a margin of 3.6% on the MATH benchmarks.

#### Comparing with the Open-Source Models.

The findings illustrated in the table [1](#S2.T1) explicitly demonstrate that our *WizardMath 70B*, distinctly manifest a substantial performance advantage over all the open-source models across both the GSM8k and MATH benchmarks. The detailed results are as follows:

- 1.
*WizardMath 7B* surpasses most open-source models with parameter counts ranging approximately from 7B to 40B, including MPT, Falcon, Baichuan-chat, Vicuna v1.3, ChatGLM 2, Qwen, Llama 1 and Llama 2 on the GSM8k and MATH benchmarks. Even though its parameter counts are significantly lower.
- 2.
*WizardMath 13B* is significantly superior to Llama 1 65B (63.9 vs. 50.9) and Llama 2 70B (63.9 vs. 56.8) on GSM8k. Additionly, it substantially outperforms both Llama 1 65B (14.0 vs. 10.6) and Llama 2 70B (14.0 vs. 13.5) on MATH.
- 3.
*WizardMath 70B*, our most extensive model, exemplifies a substantial advancement in performance, surpassing Llama 2 70B (81.6 vs. 56.8) by a significant margin of 24.8% on GSM8k. Concurrently, it also outperforms Llama 2 70B (22.7 vs. 13.5) by a margin of 9.2% on MATH.

### 3.5 Case Study

Appendix [A](#A1) shows some examples generated by our *WizardMath*. The examples demonstrate that our model consistently generates accurate response answers accompanied by clear explanations.

## 4 Related Work

#### Large Language Models.

LLMs have achieved substantial advancements within the realm of Natural Language Processing (NLP), providing a valuable and task-agnostic foundation for widespread applications. These models typically encompass parameter counts reaching into the hundreds of billions, which are trained on extensive large-scale corpuses of textual data. The prominent instances entail OpenAI’s GPT3&4 [, ], Anthropic’s Claude[5](#footnote5), Google’s PaLM [, ], Bard(^9^99 [https://bard.google.com/](https://bard.google.com/)), DeepMind’s Chinchilla [], and Gopher []. However none of them have been open-sourced so far, and some of them can only be exclusively accessible through APIs.

Recently, the AI landscape has borne witness to the emergence of numerous open-source LLMs, characterized by publicly accessible model codes and weight parameters. EleutherAI has contributed GPT-NeoX-20B [] and GPT-J-6B []. BigScience has introduced BLOOM []. Similarly, Meta has made strides by releasing OPT [], Llama 1 [], Llama 2 [], and GAL []. Tsinghua University has unveiled GLM-130B and ChatGLM []. TII has facilitated the release of Falcon []. Additionally, LLMs such as Baichuan[7](#footnote7) and Qwen[8](#footnote8) have also surfaced. Presently, Llama assumes a pivotal role as the foundational model for supervised fine-tuning, ushering in the emergence of several extremely remarkable models, including Alpaca [], Vicuna [], Guanaco [], WizardLM [], and Orca [], RFT [] etc.

#### Large Language Models For Mathematical reasoning.

It’s well known that complex reasoning problems are challenging for NLP models, which include mathematical reasoning [, , , , , ], common-sense reasoning [, ], and logical reasoning []. A substantial body of current research is centered around the intricate task reasoning of the Mathematical Word Problems(MWP) [, , , , , , ], which requires the ability to understand mathematical concepts, computation and multi-step reasoning [, , , , , , ]. Addtitionly, models are evaluated across different levels of MWP benchmarks on some mathematical reasoning datasets such as AddSub [], MultiArith [], SingleEQ [], SVAMP [], GSM8K [], AQuA [] and MATH [].

To enhance the reasoning ability of LLMs,  [] proposed Chain-of-Thought Prompting, which attaches multiple reasoning steps before obtaining the answer for a question. By employing the simple few-shot reasoning strategy, LLMs are able to perform better in complex reasoning problems. Least-to-Most [] prompting decomposes the problem into sub-problems that are then solved incrementally. Additionally each step has a more detailed reasoning process. Similarly, the Complex CoT [] underscores the pivotal role of prompt complexity by strategically choosing the most intricate problems and their corresponding solutions to function as prompts. To alleviate the burden of manual efforts,  [] introduced Auto-CoT, an approach that automates the process of acquiring k samples through the application of clustering techniques on a provided dataset. With the objective of mitigating manual intervention,  [] proposed Zero-shot-CoT, which entails the straightforward practice of appending the phrase "Let’s think step by step" to each answer, eliciting the inference steps without examples. Moreover,  [] expanded upon this notion by suggesting the exploration of diverse inference paths throughout the reasoning process. Consequently, the ultimate outcome is determined through either the aggregation of answers using majority voting or by leveraging a validation mechanism, as posited by [].  [] employs a straightforward approach for generating augmented samples, focusing on probing the correlation between LLMs and math reasoning ability.

#### Large Language Models For Reinforcement Learning.

Nevertheless, even state-of-the-art models frequently manifest logical errors and a range of illusions [, ]. These anomalies become especially challenging within domains necessitating multi-step reasoning, where a singular logical misstep maybe precipitate the unraveling of an entire solution. An effective strategy involves the training of reward models aimed at discriminating between favorable and unfavorable outputs []. Early outcome-based approaches were mainly performed on algorithmic tasks [, , , ].  [] demonstrated the significant benefits of reward models or validators, and  [] proposed a heuristic-based step-size-aware RM. [, , , ] proposed the use of reward models for a reinforcement learning pipeline.  [, , , , , , , ] employed rejection sampling for searching to achieve alignment of LLMs with human preferences.

The differences between outcome-based and process-based reward modelling are further discussed by []. Outcome-supervised reward models (ORMs) undergo training exclusively utilizing the ultimate outcomes derived from the model’s chain-of-thought process. Conversely, process-supervised reward models (PRMs) are designed to solicit feedback for each individual step within the chain-of-thought progression. In the domain of logical reasoning, ORMs frequently employ incorrect reasoning pathways yet yield the correct final answer [, ]. Notably, PRMs has been demonstrated to effectively alleviate this phenomenon of inconsistent behavior [].  [, , ] amassed an expansive corpus of process-based supervised signals through meticulous manual annotation, which verified that PRMs and supervision with manual annotation yielded more pronounced advantages for LLMs as compared to ORMs.

#### Large Language Models For Instruction Fine-Tuning.

The initial endeavors in instruction-following training work primarily focused on enhancing the language model’s capacity for generalization across diverse tasks. This often involves the process of fine-tuning across substantially available Natural Language Processing datasets, and evaluates on the different NLP tasks. T5 [] undertake the earliest attempts to train a range of NLP tasks, including Question and Answer, Document Summarization, and Sentiment Classification, by employing a consistent prompt format across all the data. Subsequently, instruction fine-tuning work such as FLAN [], ExT5 [], T0 [], UnifiedQA [], ZeroPrompt [], and FLAN-T5 [] emerged to adapt for a large number of downstream tasks. To address the challenge of misalignment between model outputs and human requirements, OpenAI manually annotates the instruction library to construct a diverse range of tasks. Simultaneously, Reinforcement Learning from Human Feedback technology is employed, which facilitate the rapid development of LLMs such as InstructGPT [], ChatGPT[3](#footnote3), GPT-4 []. To reduce manual involvement, self-instruct [] improves instruction-following through self-generated instructions. Alpaca [] used a dataset of 50k instructions generated from a limited (e.g., 175
samples) seed set of manually-written instructions. Vicuna [] used 70k user-shared conversations with ChatGPT collected from ShareGPT.com. Meanwhile, WizardLM [] introduces the evol-instruct approach, which seeks to refine the existing instruction data by enhancing both its complexity and diversity.

## 5 Conclusion and Future Work

This paper introduces *WizardMath*, a mathematics model fine-tuned with *RLEIF*. The experimental results demonstrate that *WizardMath* achieves SOTA performance surpassing all existing open-source LLMs on two widely recognized mathematical reasoning benchmarks: GSM8k and MATH. Furthermore, *WizardMath* exhibits superior performance compared to some of the largest close-source LLMs, including ChatGPT, GPT-3.5, Claude Instant, PaLM-2, PaLM-1 and Minerva on the GSM8k benchmark.

#### Future Work.

Although our *WizardMath* achieves impressive mathematics performance, as depicted in Figure [2](#S2.F2), our model still falls significantly behind the SOTA LLM, GPT-4 and Claude-2. Therefore, future work will prioritize the enhancement of the *RLEIF* or better method to further augment the performance of our model.

#### Broader Impact.

Similar to the other LLMs, our *WizardMath* could also generate unethical, harmful, or misleading information sometimes. Therefore, future research to address the ethical and societal implications is needed.

## Appendix A Appendix

### A.1 GSM8k Case Study

**Table 3: A comparison case on different scale size models**
| Instruction: Meredith is a freelance blogger who writes about health topics and submits to clients each day as her permanent job. A blog article takes an average of 4 hours to research and write about. Last week, she wrote 5 articles on Monday and 2/5 times more articles on Tuesday than on Monday. On Wednesday, she wrote twice the number of articles she wrote on Tuesday. Calculate the total number of hours she spent writing articles in the three days. |  |
| --- | --- |
| Ground Truth Answer: 104 |  |
| Note: WizardMath 7B, 13B and 70B answer correctly. |  |
| WizardMath-7B |  |
|  | Step 1: Calculate the number of articles Meredith wrote on Monday. |
|  | Meredith wrote 5 articles on Monday. |
|  | Step 2: Calculate the number of articles Meredith wrote on Tuesday. |
|  | Meredith wrote 2/5 times more articles on Tuesday than on Monday. So, she wrote 2/5 * 5 = 2 articles more on Tuesday than on Monday. Therefore, she wrote 5 + 2 = 7 articles on Tuesday. |
|  | Step 3: Calculate the total number of articles Meredith wrote in the two days. |
|  | In the two days, she wrote 5 + 7 = 12 articles. |
|  | Step 4: Calculate the number of articles Meredith wrote on Wednesday. |
|  | Meredith wrote twice the number of articles she wrote on Tuesday on Wednesday. So, she wrote 2 * 7 = 14 articles on Wednesday. |
|  | Step 5: Calculate the total number of articles Meredith wrote in the three days. |
|  | In the three days, she wrote 12 + 14 = 26 articles. |
|  | Step 6: Calculate the total number of hours Meredith spent writing articles in the three days. |
|  | Each article takes 4 hours to research and write. So, she spent 4 hours * 26 = 104 hours writing articles in the three days. |
|  | Therefore, Meredith spent 104 hours writing articles in the three days. |
|  | The answer is: 104. |
| WizardMath-13B |  |
|  | Step 1: Find the number of articles Meredith wrote on Tuesday. |
|  | We know that Meredith wrote 5 articles on Monday and 2/5 times more articles on Tuesday than on Monday. To find the number of articles she wrote on Tuesday, we can multiply the number of articles she wrote on Monday by 2/5: |
|  | 5 articles * 2/5 = 2 articles |
|  | So, Meredith wrote 2 more articles on Tuesday than on Monday. To find the total number of articles she wrote on Tuesday, we can add the 2 articles to the 5 articles she wrote on Monday: |
|  | 5 articles + 2 articles = 7 articles |
|  | Step 2: Find the number of articles Meredith wrote on Wednesday. |
|  | We know that Meredith wrote twice the number of articles she wrote on Tuesday on Wednesday. So, we can multiply the number of articles she wrote on Tuesday by 2: |
|  | 7 articles * 2 = 14 articles |
|  | Step 3: Find the total number of articles Meredith wrote in the three days. |
|  | Now, we can add the number of articles she wrote on each day to find the total number of articles: |
|  | 5 articles (Monday) + 7 articles (Tuesday) + 14 articles (Wednesday) = 26 articles |
|  | Step 4: Calculate the total number of hours Meredith spent writing articles in the three days. |
|  | We know that each article takes an average of 4 hours to research and write about. So, to find the total number of hours Meredith spent writing articles in the three days, we can multiply the total number of articles by the number of hours per article: |
|  | 26 articles * 4 hours/article = 104 hours |
|  | Therefore, Meredith spent a total of 104 hours writing articles in the three days. |
|  | The answer is: 104. |
| WizardMath-70B |  |
|  | 1. On Monday, Meredith wrote 5 articles. |
|  | 2. On Tuesday, she wrote 2/5 times more articles than on Monday. To find out how many articles she wrote on Tuesday, we need to multiply the number of articles she wrote on Monday by 2/5. So, 5 articles * 2/5 = 2 articles. Therefore, she wrote 7 articles on Tuesday (5 articles + 2 articles). |
|  | 3. On Wednesday, she wrote twice the number of articles she wrote on Tuesday. To find out how many articles she wrote on Wednesday, we need to multiply the number of articles she wrote on Tuesday by 2. So, 7 articles * 2 = 14 articles. |
|  | Now, let’s calculate the total number of articles she wrote in the three days: |
|  | Total articles = Monday’s articles + Tuesday’s articles + Wednesday’s articles |
|  | Total articles = 5 articles + 7 articles + 14 articles = 26 articles |
|  | Since each article takes an average of 4 hours to research and write about, we can calculate the total number of hours she spent writing articles in the three days: |
|  | Total hours = Total articles * Time per article |
|  | Total hours = 26 articles * 4 hours/article = 104 hours |
|  | So, Meredith spent a total of 104 hours writing articles in the three days. |
|  | The answer is: 104. |

**Table 4: A comparison case on different scale size models**
| Instruction: A mechanic charges different rates to repair the tires of trucks and cars. For each truck tire that is repaired, the mechanic will charge $60 and for each car tire that is repaired, the mechanic will charge $40. On Thursday, the mechanic repairs 6 truck tires and 4 car tires. On Friday, the mechanic repairs 12 car tries and doesn’t repair any truck tires. How much more revenue did the mechanic earn on the day with higher revenue? |  |
| --- | --- |
| Ground Truth Answer: 40 |  |
| Note: WizardMath 7B answers incorrectly, while 13B and 70B answer correctly. |  |
| WizardMath-7B |  |
|  | On Thursday, the mechanic repaired 6 truck tires and 4 car tires. Therefore, the revenue from repairing truck tires on Thursday was 6 * $60 = $360, and the revenue from repairing car tires on Thursday was 4 * $40 = $160. On Friday, the mechanic repaired 12 car tires, but did not repair any truck tires. Therefore, the revenue from repairing car tires on Friday was 12 * $40 = $480. |
|  | To calculate the total revenue from repairing truck tires and car tires on Thursday and Friday, we need to add up the revenue from each day. Therefore, the total revenue from repairing truck tires and car tires on Thursday and Friday was $360 + $160 + $480 = $900. |
|  | To calculate the revenue from repairing truck tires on Friday, we need to know how many truck tires the mechanic repaired on Friday. However, this information is not given in the question. Therefore, we cannot calculate the revenue from repairing truck tires on Friday. |
|  | To calculate the revenue from repairing car tires on Thursday and Friday, we need to add up the revenue from each day. Therefore, the total revenue from repairing car tires on Thursday and Friday was $160 + $480 = $640. |
|  | To calculate the difference between the revenue from repairing truck tires and car tires on Thursday and Friday, we need to subtract the revenue from repairing car tires on Thursday and Friday from the total revenue from repairing truck tires and car tires on Thursday and Friday. Therefore, the difference between the revenue from repairing truck tires and car tires on Thursday and Friday was $900 - $640 = $260. |
|  | Therefore, the mechanic earned $260 more on the day with higher revenue. The answer is: 260. |
|  | The answer is: 260. |
| WizardMath-13B |  |
|  | Step 1: Calculate the revenue earned on Thursday. |
|  | On Thursday, the mechanic repaired 6 truck tires and 4 car tires. The mechanic charges $60 for each truck tire and $40 for each car tire. Therefore, the revenue earned on Thursday can be calculated as follows: |
|  | Revenue on Thursday = (Number of truck tires) * (Price per truck tire) + (Number of car tires) * (Price per car tire) |
|  | Revenue on Thursday = 6 * $60 + 4 * $40 |
|  | Revenue on Thursday = $360 + $160 |
|  | Revenue on Thursday = $520 |
|  | Step 2: Calculate the revenue earned on Friday. |
|  | On Friday, the mechanic repaired 12 car tires and didn’t repair any truck tires. Therefore, the revenue earned on Friday can be calculated as follows: |
|  | Revenue on Friday = (Number of car tires) * (Price per car tire) |
|  | Revenue on Friday = 12 * $40 |
|  | Revenue on Friday = $480 |
|  | Step 3: Compare the revenue earned on both days. |
|  | Now, we need to compare the revenue earned on Thursday and Friday to find out which day earned more revenue. |
|  | Difference in revenue = Revenue on Thursday - Revenue on Friday |
|  | Difference in revenue = $520 - $480 |
|  | Difference in revenue = $40 |
|  | Therefore, the mechanic earned $40 more in revenue on the day with higher revenue, which is Thursday. The final answer is $40. |
|  | The answer is: 40. |
| WizardMath-70B |  |
|  | Step 1: Calculate the revenue from repairing truck tires on Thursday. |
|  | The mechanic repaired 6 truck tires on Thursday, and each truck tire repair costs $60. |
|  | So, the revenue from repairing truck tires on Thursday is 6 * $60 = $360. |
|  | Step 2: Calculate the revenue from repairing car tires on Thursday. |
|  | The mechanic repaired 4 car tires on Thursday, and each car tire repair costs $40. |
|  | So, the revenue from repairing car tires on Thursday is 4 * $40 = $160. |
|  | Step 3: Calculate the total revenue on Thursday. |
|  | The total revenue on Thursday is the sum of the revenue from repairing truck tires and car tires. |
|  | So, the total revenue on Thursday is $360 + $160 = $520. |
|  | Step 4: Calculate the revenue from repairing car tires on Friday. |
|  | The mechanic repaired 12 car tires on Friday, and each car tire repair costs $40. |
|  | So, the revenue from repairing car tires on Friday is 12 * $40 = $480. |
|  | Step 5: Calculate the total revenue on Friday. |
|  | The mechanic didn’t repair any truck tires on Friday, so the total revenue on Friday is the revenue from repairing car tires. |
|  | So, the total revenue on Friday is $480. |
|  | Step 6: Calculate the difference in revenue between the two days. |
|  | The difference in revenue is the difference between the total revenue on the day with higher revenue and the total revenue on the day with lower revenue. |
|  | So, the difference in revenue is $520 - $480 = $40. |
|  | Therefore, the mechanic earned $40 more revenue on the day with higher revenue. The final answer is $40. |
|  | The answer is: 40. |

**Table 5: A comparison case on different scale size models**
| Instruction: Adrien’s total salary was 30 percent higher than Lylah’s. Four years later, his salary had increased, and he was earning 40% more than what he was making four years ago. If Adrien’s and Lylah’s salary increased simultaneously, and Adrien earned $40000 four years ago, calculate the total salary the two were receiving four years later? |  |
| --- | --- |
| Ground Truth Answer: 95200 |  |
| Note: WizardMath 7B and 13B answer incorrectly, while 70B answers correctly. |  |
| WizardMath-7B |  |
|  | Step 1: Calculate Adrien’s salary four years ago. |
|  | Since Adrien earned $40000 four years ago, his total salary four years ago was $40000. |
|  | Step 2: Calculate the increase in Adrien’s salary four years later. |
|  | Four years later, Adrien’s salary had increased by 40% of his salary four years ago. Therefore, the increase in Adrien’s salary was 40/100*$40000 = $16000. |
|  | Step 3: Calculate Adrien’s total salary four years later. |
|  | Adrien’s total salary four years later was the sum of his salary four years ago and the increase in his salary. Therefore, Adrien’s total salary four years later was $40000+$16000 = $56000. |
|  | Step 4: Calculate the increase in Lylah’s salary four years later. |
|  | Since Adrien’s total salary was 30% higher than Lylah’s, the increase in Lylah’s salary four years later was 30/100*$56000 = $16800. |
|  | Step 5: Calculate Lylah’s total salary four years later. |
|  | Lylah’s total salary four years later was the sum of her salary four years ago and the increase in her salary. Therefore, Lylah’s total salary four years later was $56000+$16800 = $72800. |
|  | Step 6: Calculate the total salary the two were receiving four years later. |
|  | The total salary the two were receiving four years later was the sum of Adrien’s and Lylah’s total salary four years later. Therefore, the total salary the two were receiving four years later was $72800+$56000 = $128800. |
|  | Therefore, the total salary the two were receiving four years later was $128800. |
|  | The answer is: 128800. |
| WizardMath-13B |  |
|  | Step 1: Calculate Adrien’s total salary four years ago. |
|  | We know that Adrien’s total salary was 30 percent higher than Lylah’s. If Adrien was earning $40000 four years ago, then Lylah was earning $40000/30/100 = $12000. |
|  | Step 2: Calculate Adrien’s total salary four years later. |
|  | Four years later, Adrien’s salary had increased, and he was earning 40% more than what he was making four years ago. Therefore, Adrien’s total salary four years later is $40000 + $40000*40/100 = $56000. |
|  | Step 3: Calculate Lylah’s total salary four years later. |
|  | If Adrien’s salary increased by 40%, then Lylah’s salary also increased by 40%. Therefore, Lylah’s total salary four years later is $12000 + $12000*40/100 = $16800. |
|  | Step 4: Calculate the total salary the two were receiving four years later. |
|  | The total salary the two were receiving four years later is the sum of Adrien’s and Lylah’s total salaries. Therefore, the total salary the two were receiving four years later is $56000 + $16800 = $72800. |
|  | Therefore, the total salary the two were receiving four years later is $72800. |
|  | The answer is: 72800. |
| WizardMath-70B |  |
|  | Step 1: Determine Lylah’s salary four years ago. |
|  | Since Adrien’s total salary was 30 percent higher than Lylah’s, and Adrien earned $40000 four years ago, we can find Lylah’s salary by subtracting 30% of $40000 from Adrien’s salary. |
|  | Lylah’s salary = $40000 - (30/100)*$40000 = $40000 - $12000 = $28000 |
|  | Step 2: Calculate Adrien’s salary four years later. |
|  | Four years later, Adrien’s salary had increased by 40% of what he was making four years ago. So, we need to add 40% of $40000 to his salary from four years ago. |
|  | Adrien’s salary four years later = $40000 + (40/100)*$40000 = $40000 + $16000 = $56000 |
|  | Step 3: Calculate Lylah’s salary four years later. |
|  | Since their salaries increased simultaneously, Lylah’s salary also increased by 40% of what she was making four years ago. So, we need to add 40% of $28000 to her salary from four years ago. |
|  | Lylah’s salary four years later = $28000 + (40/100)*$28000 = $28000 + $11200 = $39200 |
|  | Step 4: Calculate the total salary the two were receiving four years later. |
|  | Now, we just need to add Adrien’s and Lylah’s salaries from four years later to find their total salary. |
|  | Total salary = Adrien’s salary four years later + Lylah’s salary four years later = $56000 + $39200 = $95200 |
|  | So, the total salary the two were receiving four years later was $95200. |
|  | The answer is: 95200. |

### A.2 MATH Case Study

**Table 6: A comparison case on different scale size models**
| Instruction: The midpoint of the line segment between $(x,y)$ and $(-9,1)$ is $(3,-5)$. Find $(x,y)$. |  |
| --- | --- |
| Ground Truth Answer: (15,-11). |  |
| Note: WizardMath 7B, 13B and 70B answer correctly. |  |
| WizardMath-7B |  |
|  | Step 1: Understand the problem |
|  | We are given the midpoint of the line segment between $(x,y)$ and $(-9,1)$ as $(3,-5)$. We need to find the coordinates of $(x,y)$. |
|  | Step 2: Recall the formula for the midpoint |
|  | The midpoint of a line segment with endpoints $(x_{1},y_{1})$ and $(x_{2},y_{2})$ can be found using the midpoint formula: |
|  | (x1+ x22, y1+ y22) |
|  | Step 3: Apply the midpoint formula |
|  | In this case, the endpoints are $(x,y)$ and $(-9,1)$. So, we can plug these values into the midpoint formula: |
|  | (x + (-9)2, y + 12) |
|  | Step 4: Simplify the expression |
|  | Now, we can simplify the expression by performing the arithmetic operations: |
|  | (x - 92, y + 12) |
|  | Step 5: Compare the coordinates |
|  | We are given that the midpoint is $(3,-5)$. So, we can compare the coordinates of the simplified expression with the given midpoint: |
|  | x - 92 = 3 |
|  | y + 12 = -5 |
|  | Step 6: Solve for the coordinates |
|  | Now, we can solve for the coordinates of $(x,y)$ by multiplying both sides of the equations by 2: |
|  | x - 9 = 6 |
|  | y + 1 = -10 |
|  | Step 7: Write the final answer |
|  | Now, we can write the final answer by adding the results of the equations: |
|  | x = 6 + 9 |
|  | y = -10 - 1 |
|  | x = 15 |
|  | y = -11 |
|  | So, the coordinates of $(x,y)$ are $\boxed{(15,-11)}$. |
|  | The answer is: (15,-11). |
| WizardMath-13B |  |
|  | Step 1: Understand the problem |
|  | We are given that the midpoint of the line segment between $(x,y)$ and $(-9,1)$ is $(3,-5)$. We need to find the coordinates of the point $(x,y)$. |
|  | Step 2: Recall the midpoint formula |
|  | The midpoint of a line segment with endpoints $(x_{1},y_{1})$ and $(x_{2},y_{2})$ can be found using the midpoint formula: |
|  | Midpoint = $\left(\frac{x_{1}+x_{2}}{2},\frac{y_{1}+y_{2}}{2}\right)$ |
|  | Step 3: Apply the midpoint formula |
|  | In our case, the endpoints are $(x,y)$ and $(-9,1)$. So, we can plug these values into the midpoint formula: |
|  | Midpoint = $\left(\frac{x+(-9)}{2},\frac{y+1}{2}\right)$ |
|  | Step 4: Simplify the midpoint |
|  | Now, we can simplify the midpoint by performing the arithmetic operations: |
|  | Midpoint = $\left(\frac{x-9}{2},\frac{y+1}{2}\right)$ |
|  | Step 5: Set up an equation |
|  | Since the midpoint is given as $(3,-5)$, we can set up an equation using the simplified midpoint formula: |
|  | $\frac{x-9}{2}=3$ and $\frac{y+1}{2}=-5$ |
|  | Step 6: Solve the equations |
|  | Now, we need to solve these two equations to find the values of $x$ and $y$. Let’s start with the first equation: |
|  | $\frac{x-9}{2}=3$ |
|  | Multiply both sides by 2 to get rid of the fraction: |
|  | $x-9=6$ |
|  | Now, add 9 to both sides: |
|  | $x=15$ |
|  | Next, let’s solve the second equation: |
|  | $\frac{y+1}{2}=-5$ |
|  | Multiply both sides by 2 to get rid of the fraction: |
|  | $y+1=-10$ |
|  | Now, subtract 1 from both sides: |
|  | $y=-11$ |
|  | Step 7: Write the final answer |
|  | We have found the values of $x$ and $y$. So, the coordinates of the point $(x,y)$ are $(15,-11)$. Therefore, the final answer is: |
|  | $(x,y)=\boxed{(15,-11)}$ |
|  | The answer is: (15,-11). |
| WizardMath-70B |  |
|  | Step 1: Understand the problem |
|  | We are given the midpoint of a line segment and the coordinates of one of its endpoints. We need to find the coordinates of the other endpoint, which is $(x,y)$. |
|  | Step 2: Recall the midpoint formula |
|  | The midpoint of a line segment with endpoints $(x_{1},y_{1})$ and $(x_{2},y_{2})$ is given by the formula: |
|  | Midpoint = $\left(\frac{x_{1}+x_{2}}{2},\frac{y_{1}+y_{2}}{2}\right)$ |
|  | Step 3: Plug in the given information |
|  | We know the midpoint is $(3,-5)$ and one endpoint is $(-9,1)$. So, we can plug these values into the midpoint formula: |
|  | $(3,-5)=\left(\frac{-9+x}{2},\frac{1+y}{2}\right)$ |
|  | Step 4: Solve for $x$ and $y$ |
|  | Now, we need to solve the system of equations: |
|  | 1) $\frac{-9+x}{2}=3$ |
|  | 2) $\frac{1+y}{2}=-5$ |
|  | Step 5: Solve equation 1 for $x$ |
|  | Multiply both sides by 2: |
|  | $-9+x=6$ |
|  | Add 9 to both sides: |
|  | $x=15$ |
|  | Step 6: Solve equation 2 for $y$ |
|  | Multiply both sides by 2: |
|  | $1+y=-10$ |
|  | Subtract 1 from both sides: |
|  | $y=-11$ |
|  | Step 7: Write the final answer |
|  | Now that we have found the values of $x$ and $y$, we can write the final answer as: |
|  | $(x,y)=(15,-11)$ |
|  | So, the coordinates of the other endpoint are $(15,-11)$. |
|  | The answer is: (15,-11). |

**Table 7: A comparison case on different scale size models**
| Instruction: Evaluate $\tan 10^{\circ}\tan 20^{\circ}\tan 30^{\circ}\tan 40^{\circ}\tan 50^{\circ}\tan 60^{\circ}\tan 70^{\circ}\tan 80^{\circ}.$ |  |
| --- | --- |
| Ground Truth Answer: 1 |  |
| Note: WizardMath 7B answers incorrectly, while 13B and 70B answer correctly. |  |
| WizardMath-7B |  |
|  | First, we can rewrite the given expression using the identity $\tan(90^{\circ}-x)=\frac{1}{\tan x}$: |
|  | $\tan 10^{\circ}\tan 20^{\circ}\tan 30^{\circ}\tan 40^{\circ}\tan 50^{\circ}\tan 60^{\circ}\tan 70^{\circ}\tan 80^{\circ}=\frac{1}{\tan 80^{\circ}}$ |
|  | Now, we can use the double-angle formula for tangent, which is $\tan 2x=\frac{2\tan x}{1-\tan^{2}x}$: |
|  | $\frac{1}{\tan 80^{\circ}}=\frac{2}{\tan 160^{\circ}}=\frac{2}{\frac{1-\tan^{2}80^{\circ}}{2\tan 80^{\circ}}}=\frac{1-\tan^{2}80^{\circ}}{2\tan 80^{\circ}}$ |
|  | Next, we can use the double-angle formula for tangent again: |
|  | $\frac{1-\tan^{2}80^{\circ}}{2\tan 80^{\circ}}=\frac{1-\frac{1}{2}\tan^{2}80^{\circ}}{2\tan 80^{\circ}}=\frac{1-\frac{1}{2}\tan^{2}80^{\circ}}{2\tan 80^{\circ}}\cdot\frac{2\tan 80^{\circ}}{1-\tan^{2}80^{\circ}}$ |
|  | Now, we can multiply the fractions: |
|  | $\frac{1-\frac{1}{2}\tan^{2}80^{\circ}}{2\tan 80^{\circ}}\cdot\frac{2\tan 80^{\circ}}{1-\tan^{2}80^{\circ}}=\frac{1-\frac{1}{2}\tan^{2}80^{\circ}}{1-\tan^{2}80^{\circ}}$ |
|  | Finally, we can simplify the expression: |
|  | $\frac{1-\frac{1}{2}\tan^{2}80^{\circ}}{1-\tan^{2}80^{\circ}}=1-\frac{1}{2}\tan^{2}80^{\circ}$ |
|  | So, the rewritten answer is: |
|  | $\tan 10^{\circ}\tan 20^{\circ}\tan 30^{\circ}\tan 40^{\circ}\tan 50^{\circ}\tan 60^{\circ}\tan 70^{\circ}\tan 80^{\circ}=\boxed{1-\frac{1}{2}\tan^{2}80^{\circ}}$ |
|  | The answer is: $1-\frac{1}{2}\tan^{2}80^{\circ}$. |
| WizardMath-13B |  |
|  | Step 1: Recall the angle addition formula for tangent: |
|  | $\tan(A+B)=\frac{\tan A+\tan B}{1-\tan A\tan B}$ |
|  | Step 2: Use the formula to find $\tan 110^{\circ}$: |
|  | $\tan 110^{\circ}=\tan(10^{\circ}+100^{\circ})=\frac{\tan 10^{\circ}+\tan 100^{\circ}}{1-\tan 10^{\circ}\tan 100^{\circ}}=\frac{1+0}{1-1\cdot 0}=1$ |
|  | Step 3: Observe that $\tan 110^{\circ}$ is the only term in the given expression that is not equal to 0. |
|  | Step 4: Multiply both sides of the equation by $\tan 110^{\circ}$: |
|  | $\tan 10^{\circ}\tan 20^{\circ}\tan 30^{\circ}\tan 40^{\circ}\tan 50^{\circ}\tan 60^{\circ}\tan 70^{\circ}\tan 80^{\circ}\tan 110^{\circ}=1\cdot\tan 110^{\circ}$ |
|  | Step 5: Simplify the equation: |
|  | $\tan 10^{\circ}\tan 20^{\circ}\tan 30^{\circ}\tan 40^{\circ}\tan 50^{\circ}\tan 60^{\circ}\tan 70^{\circ}\tan 80^{\circ}=\tan 110^{\circ}$ |
|  | Step 6: Since $\tan 110^{\circ}=1$, the final answer is: |
|  | $\tan 10^{\circ}\tan 20^{\circ}\tan 30^{\circ}\tan 40^{\circ}\tan 50^{\circ}\tan 60^{\circ}\tan 70^{\circ}\tan 80^{\circ}=\boxed{1}.$ |
|  | The answer is: 1. |
| WizardMath-70B |  |
|  | Step 1: Recall the definitions of tangent and cotangent. |
|  | We know that $\tan x=\frac{\sin x}{\cos x}$ and $\cot x=\frac{\cos x}{\sin x}$. |
|  | Step 2: Find the tangent and cotangent of each angle. |
|  | We need to find the tangent and cotangent of each angle in the given expression: |
|  | $\tan 10^{\circ}=\frac{\sin 10^{\circ}}{\cos 10^{\circ}}$ |
|  | $\tan 20^{\circ}=\frac{\sin 20^{\circ}}{\cos 20^{\circ}}$ |
|  | $\tan 30^{\circ}=\frac{\sin 30^{\circ}}{\cos 30^{\circ}}$ |
|  | $\tan 40^{\circ}=\frac{\sin 40^{\circ}}{\cos 40^{\circ}}$ |
|  | $\tan 50^{\circ}=\frac{\sin 50^{\circ}}{\cos 50^{\circ}}$ |
|  | $\tan 60^{\circ}=\frac{\sin 60^{\circ}}{\cos 60^{\circ}}$ |
|  | $\tan 70^{\circ}=\frac{\sin 70^{\circ}}{\cos 70^{\circ}}$ |
|  | $\tan 80^{\circ}=\frac{\sin 80^{\circ}}{\cos 80^{\circ}}$ |
|  | Step 3: Find the cotangent of each angle. |
|  | Now, we need to find the cotangent of each angle: |
|  | $\cot 10^{\circ}=\frac{\cos 10^{\circ}}{\sin 10^{\circ}}$ |
|  | $\cot 20^{\circ}=\frac{\cos 20^{\circ}}{\sin 20^{\circ}}$ |
|  | $\cot 30^{\circ}=\frac{\cos 30^{\circ}}{\sin 30^{\circ}}$ |
|  | $\cot 40^{\circ}=\frac{\cos 40^{\circ}}{\sin 40^{\circ}}$ |
|  | $\cot 50^{\circ}=\frac{\cos 50^{\circ}}{\sin 50^{\circ}}$ |
|  | $\cot 60^{\circ}=\frac{\cos 60^{\circ}}{\sin 60^{\circ}}$ |
|  | $\cot 70^{\circ}=\frac{\cos 70^{\circ}}{\sin 70^{\circ}}$ |
|  | $\cot 80^{\circ}=\frac{\cos 80^{\circ}}{\sin 80^{\circ}}$ |
|  | Step 4: Use the identity $\tan x\cot x=1$. |
|  | We know that $\tan x\cot x=1$. So, we can rewrite the given expression as: |
|  | $\tan 10^{\circ}\tan 20^{\circ}\tan 30^{\circ}\tan 40^{\circ}\tan 50^{\circ}\tan 60^{\circ}\tan 70^{\circ}\tan 80^{\circ}=\frac{\sin 10^{\circ}}{\cos 10^{\circ}}\cdot\frac{\sin 20^{\circ}}{\cos 20^{\circ}}\cdot\frac{\sin 30^{\circ}}{\cos 30^{\circ}}\cdot\frac{\sin 40^{\circ}}{\cos 40^{\circ}}\cdot\frac{\sin 50^{\circ}}{\cos 50^{\circ}}\cdot\frac{\sin 60^{\circ}}{\cos 60^{\circ}}\cdot\frac{\sin 70^{\circ}}{\cos 70^{\circ}}\cdot\frac{\sin 80^{\circ}}{\cos 80^{\circ}}$ |
|  | Step 5: Simplify the expression. |
|  | Now, we can simplify the expression by canceling out the terms: |
|  | $\frac{\sin 10^{\circ}\sin 20^{\circ}\sin 30^{\circ}\sin 40^{\circ}\sin 50^{\circ}\sin 60^{\circ}\sin 70^{\circ}\sin 80^{\circ}}{\cos 10^{\circ}\cos 20^{\circ}\cos 30^{\circ}\cos 40^{\circ}\cos 50^{\circ}\cos 60^{\circ}\cos 70^{\circ}\cos 80^{\circ}}$ |
|  | Step 6: Calculate the final answer. |
|  | Now, we can calculate the final answer: |
|  | $\tan 10^{\circ}\tan 20^{\circ}\tan 30^{\circ}\tan 40^{\circ}\tan 50^{\circ}\tan 60^{\circ}\tan 70^{\circ}\tan 80^{\circ}=\boxed{1}$ |
|  | So, the final answer is $\boxed{1}$. |
|  | The answer is: 1. |

**Table 8: A comparison case on different scale size models**
| Instruction: Let $f(x)$ be an odd function, and let $g(x)$ be an even function. Is $f(f(g(f(g(f(x))))))$ even, odd, or neither?<br>Enter "odd", "even", or "neither". |  |
| --- | --- |
| Ground Truth Answer: even |  |
| Note: WizardMath 7B and 13B answer incorrectly, while 70B answers correctly. |  |
| WizardMath-7B |  |
|  | 1. We know that $f(x)$ is an odd function, which means that $f(-x)=-f(x)$ for all $x$ in its domain. |
|  | 2. We also know that $g(x)$ is an even function, which means that $g(-x)=g(x)$ for all $x$ in its domain. |
|  | 3. Now, let’s find the value of $f(g(-x))$ and $f(-g(x))$. Since $g(-x)=g(x)$ and $f(-x)=-f(x)$, we can conclude that: |
|  | $f(g(-x))=f(-g(x))$ |
|  | 4. Next, let’s find the value of $f(f(g(-x)))$ and $f(f(-g(x)))$. Since $f(-x)=-f(x)$ and $f(g(-x))=f(g(x))$, we can conclude that: |
|  | $f(f(g(-x)))=f(f(-g(x)))$ |
|  | 5. Now, let’s find the value of $f(f(f(g(-x))))$ and $f(f(f(-g(x))))$. Since $f(f(-x))=-f(f(x))$ and $f(f(g(-x)))=f(f(-g(x)))$, we can conclude that: |
|  | $f(f(f(g(-x))))=-f(f(f(-g(x))))$ |
|  | 6. Finally, let’s find the value of $f(f(f(f(g(-x)))))$ and $f(f(f(f(-g(x)))))$. Since $f(f(-x))=-f(f(x))$ and $f(f(f(g(-x))))=-f(f(f(-g(x))))$, we can conclude that: |
|  | $f(f(f(f(g(-x))))))=-f(f(f(f(-g(x)))))$ |
|  | Therefore, the final answer is $\boxed{\text{odd}}$. |
|  | The answer is: odd. |
| WizardMath-13B |  |
|  | 1. We know that $f(x)$ is an odd function, which means that $f(-x)=-f(x)$. |
|  | 2. We also know that $g(x)$ is an even function, which means that $g(-x)=g(x)$. |
|  | Now, let’s analyze the given expression: $f(f(g(f(g(f(x))))))$. |
|  | 3. Replace $x$ with $-x$ in the expression: $f(f(g(f(g(f(-x))))))$. Since $f$ is odd, we have $f(-x)=-f(x)$, so the expression becomes $-f(f(g(f(g(-x)))))$. |
|  | 4. Since $g$ is even, we have $g(-x)=g(x)$. So, we can rewrite the expression as $-f(f(g(f(g(x)))))$. |
|  | 5. Now, we have another instance of $f$ and $g$. Since $f$ is odd and $g$ is even, their product is an odd function. So, we can rewrite the expression as $-[f(g(f(x)))]$. |
|  | 6. We have another instance of $f$ and $g$. Since $f$ is odd and $g$ is even, their product is an odd function. So, we can rewrite the expression as $-f(g(f(x)))$. |
|  | 7. Now, we have another instance of $f$. Since $f$ is odd, we have $f(-x)=-f(x)$. So, we can rewrite the expression as $-f(f(x))$. |
|  | 8. Finally, we have another instance of $f$. Since $f$ is odd, we have $f(-x)=-f(x)$. So, we can rewrite the expression as $-f(f(x))$. |
|  | Now, we can see that the expression is equal to $-f(f(x))$, which is an odd function. Therefore, the answer is $\boxed{\text{odd}}$. |
|  | The answer is: odd. |
| WizardMath-70B |  |
|  | 1. We are given that $f(x)$ is an odd function and $g(x)$ is an even function. |
|  | 2. Recall the definitions of odd and even functions: |
|  | - A function $f(x)$ is odd if $f(-x)=-f(x)$ for all $x$ in its domain. |
|  | - A function $g(x)$ is even if $g(-x)=g(x)$ for all $x$ in its domain. |
|  | 3. Now, let’s analyze the given function: $f(f(g(f(g(f(x))))))$. |
|  | 4. We can rewrite this function as: $f(f(g(f(g(f(x))))))=h(x)$, where $h(x)=f(f(g(f(g(f(x))))))$. |
|  | 5. Now, let’s find $h(-x)$: |
|  | - $h(-x)=f(f(g(f(g(f(-x))))))$. |
|  | 6. Since $g(x)$ is even, we have $g(-x)=g(x)$. So, we can rewrite the expression as: |
|  | - $h(-x)=f(f(g(f(g(f(x))))))$. |
|  | 7. Now, we can see that $h(-x)=h(x)$. This means that the function $h(x)$ satisfies the definition of an even function. |
|  | 8. Therefore, the given function $f(f(g(f(g(f(x))))))$ is $\boxed{\text{even}}$. |
|  | The answer is: even. |

## Peer Reviews

confidence: 3
contribution: 4
initial_rating: 8
presentation: 2
questions: - I find Figure 1 confusing. Why is there a pyramid in the top left and why is it pointing to a pie chart, cube, etc? What are these supposed to be showing? I feel like I am not understanding much from this figure. 
- I feel like there's a few missing entries in Table 1. For example, Table 1 shows the results for WizardMath-Mathstral and WizardMath-Qwen2.5, but the base scores of these base models are not shown in the table, so the readers don't really know how much improvement there is.

Typos:
- L45: struggles --> struggle
- L80: "in recent" 
- L91: should say what is IRM first before using the acronym. 
- L91: We --> capitalization
- L93: later --> latter
- L105: Jiang et al mentioned without a model (should be mistral?)
- L107: as following --> as follows
- L140,143: spacing
- L145: should be Reinforcement Learning for Large Language Models instead of the other way around?
- L487: "reasoing"
soundness: 3
strengths: **1. Strong Results** -- I put a lot of premium on this strength and use this to justify my overall rating. Many of the gains from training on Math Evol-Instruct are more than 10 points. More importantly, it is quite impressive to design something that outperforms strong proprietary models, so if this method is as strong as the paper claims, then this is something that the community will definitely quickly pick up on.

**2. Thorough Experiments and Baseline Comparisons** -- Various scales ranging from 100M to 1B to 70B, with different base models. Has comparisons with several math-specific models (e.g. Mammoth, Math-Shepherd, etc.) I also liked the analysis and ablations in Section 4.4

**3. Scalable Method** -- The whole process is fully automated, which makes it scalable. I imagine this can also be adapted to other process-intensive reasoning domains outside math (e.g. coding).
summary: The paper takes Evol-Instruct (from the WizardLM paper) and extends it to the math domain, while at the same time integrating process reward models into the training pipeline.

## Method
- Generate questions of various complexities by prompting GPT to sequentially generate easier questions ("downard evolution") and harder questions ("upward evolution")
- Train Instruction Reward Model (IRM) to predict quality of instruction
- Train Process Reward Model (PRM) to predict quality of each individual step
- Use PPO

## Experiments
- GSM8k and MATH datasets
- Main base models are Llama3 and Mistral. Experiments done across various scales.
- Outperforms a number of strong closed-source models such as GPT and Claude2
weaknesses: **1. PRM labels from GPT-4** -- Not really sure what to think of this. On one hand, I feel such direct distillation like this would limit the effectiveness of a method at larger data scales. On the other hand, the results seem to be good (and also this is one key part that makes the process fully AI-automated.)

**2. Unclear presentation** -- The paper assumes that readers are already previously familiar with Evol-Instruct, as it devotes very little time to talking about it in the intro or related work. The narrative is messy -- there are certain concepts (e.g. "grade school" and "high school" questions) that were introduced once out of nowhere then never mentioned again. There are a number of rows on Table 1 that were never discussed in the main text. Figure 1 is difficult to understand. There are also several typos and poorly-worded sentences.

**3. Somewhat marginal contribution** -- Evol-Instruct previously existed. PRM previously existed. This paper basically took Evol-Instruct and PRM and used them to train a model. To nitpick a bit, I think a more comprehensive paper would cover more domains such as code.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: - In Equation (1), how is the parameter ( m ) set? Additionally, how is the Instruction Reward Model (IRM) trained?

- Lines 256-258 suggest that you retain solutions with incorrect answers. How might this influence the results? Have you considered using the IRM to filter out low-quality examples for supervised fine-tuning (SFT)?

- During PPO, do you use two reward models? Using two reward models in PPO can be time-consuming and computationally expensive. What are your strategies for addressing this?

- In Lines 89-90, you state that you "innovatively introduce PRM to address the False-Positive issue in the problem-solving process." This claim should be validated by comparing the false-positive rate on a test set both with and without your method.

- In Lines 88-89, you mention that existing methods "mainly focus on the SFT stage and are susceptible to learning hallucinated information from the teacher model." However, in Line 95, you still use GPT-4 to annotate step-level labels. Isn’t there a risk of obtaining incorrect step labels from GPT-4 as well?
soundness: 4
strengths: This paper is well-written and rich in detail. The introduction of the Instruction Reward Model (IRM) is novel and useful. The experiments are comprehensive, and the analysis is thorough, providing deep insights into the framework's effectiveness. I believe the idea of integrating IRM with PRM could be useful for any math LLMs (only undergoing SFT).
summary: This paper proposes the Reinforcement Learning from Evol-Instruct Feedback (RLEIF) framework to enhance the mathematical reasoning capabilities of language models. RLEIF involves two main steps: instruction tuning with MATH EVOL-INSTRUCT and reinforcement learning using an Instruction Reward Model (IRM) and a Process Reward Model (PRM).

The experimental results demonstrate that their model, after both supervised fine-tuning (SFT) and reinforcement learning (RL), surpasses many existing math-focused language models that have undergone only SFT.

The ablation study highlights the crucial role of the IRM during the reinforcement learning phase. Additionally, they show that their PRM, which utilizes GPT-4 as a step annotator, delivers superior performance compared to Math-Shepherd and PRM800K.
weaknesses: The primary concern with this paper is the unfair comparison of baseline models in the results. While the authors claim that both supervised fine-tuning (SFT) with Math Evol-Instruct and reinforcement learning (RL) with the Instruction Reward Model (IRM) and Process Reward Model (PRM) are beneficial for enhancing mathematical reasoning, these approaches—SFT with synthesized data and the use of various reward models for RL—represent parallel research lines.

- In Table 1, the authors compare their model, which has undergone both SFT and RL, with models that have only undergone SFT. This comparison is unfair because these SFT models could also be further enhanced with RL techniques to improve mathematical reasoning (e.g., using ORM for RL on DartMath). It would be more appropriate to isolate the effects of SFT and RL for a fair comparison. In Table 1, the authors should compare the performance of models that have undergone SFT with Math Evol-Instruct against existing baselines such as MetaMath and DartMath. Additionally, comparisons with baselines like MetaMath and DartMath on the LLaMA-3.2 backbone would be valuable, as their training data is publicly available.

- In analyzing the impact of training data size, the authors should compare their approach with the best method for SFT using synthesized data, specifically DartMath. MetaMath, which was developed around a year ago, uses GPT-3.5-turbo for data augmentation, making it an outdated and potentially unfair baseline.

- It appears that SFT with Math Evol-Instruct yields inferior results compared to other SFT methods. From Table 4, the LLaMA2-7B: WizardMath-SFT scores 35.6 on MATH, which lags behind models like XwinMath and Skywork. Likely, it would also lag behind LLaMA2-7B fine-tuned on the DartMath training data. This suggests that the main contribution of the paper is in the RL component. Therefore, the primary focus should be on the results obtained with different reward models, as presented in Table 4, utilizing various SFT backbones.

- Table 7 lacks adequate baselines; at least, the authors should include LLaMA-2-7B trained on the DartMath training set. This table also suffers from the same fairness issues as Table 1.

I recommend that the authors reorganize the paper better to emphasize their contributions to the "RL part."

---

confidence: 3
contribution: 4
initial_rating: 8
presentation: 3
questions: NA
soundness: 4
strengths: The paper is well-written, with comprehensive analysis and ablation experiments. 

It represents a valuable exploration of process supervision and IRM in math reasoning, achieving impressive performance.

At the same time, their method is data efficient.
summary: This paper proposes a method to enhance math reasoning using Reinforced-Evol-Instruct, called WizardMath. The process begins with Supervised Fine-Tuning (SFT) on evol-instruct data, followed by training Instruction Reward Model (IRM) and Process Reward Model (PRM). Finally, they train PPO using the trained IRM and PRM. The WizardMath model achieves high performance on GMS8K and Math datasets.
weaknesses: NA

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 3
questions: see above
soundness: 3
strengths: 1. This paper provides a scalable approach to enhance LLM reasoning capabilities in math without external computational tools.
2. Notable improvements in metrics across diverse benchmarks.
3. The introduction of dual reward models (IRM and PRM) effectively improves model reliability and accuracy. 
4. The synthetic data and process supervision paradigm have had a great impact on the community.
summary: This paper introduces WizardMath, an innovative approach to improving mathematical reasoning capabilities in large language models (LLMs). WizardMath employs a unique reinforcement learning method called Reinforcement Learning from Evol-Instruct Feedback (RLEIF), which optimizes the model’s performance on mathematical reasoning without external tools. It specifically targets pretrained model to enhance their reasoning by leveraging evolutionary instruction (Evol-Instruct) to generate diverse math problems and using process-based supervision to ensure correctness at each reasoning step. Experiments demonstrate that WizardMath achieves higher accuracy than other leading models, even outperforming proprietary models like GPT-3.5 and Claude on GSM8k and MATH benchmarks.
weaknesses: 1. The training data for RLEIF is derived partly from a designated training set and synthetic from existing models, such as GPT-4. How might the reliance on specific LLMs for generating synthetic data (like GPT-4) affect the scalability of this approach for researchers without access to such models? Have the authors explored alternative methods for generating diverse mathematical problems that don't depend on proprietary models?
2. While RLEIF has undergone extensive evaluation in the context of mathematical reasoning, it may also hold potential for applications in other reasoning domains. Have the authors considered applying RLEIF to other reasoning tasks like logical deduction or causal inference? It would be valuable to see if the benefits observed in mathematical reasoning transfer to these domains, and what modifications might be necessary.

Meta-review: This paper takes the Evol-Instruct method invented in the WizardLM paper and applied that to the math domain to create a strong math SFT dataset. Then the authors also perform reinforcement learning to further boost the performance. 

This paper receives a high score of 8, 8, 8, 8 and I have no doubt to accept this paper.

Decision: Accept (Oral)