WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct
Authors: Haipeng Luo
Qingfeng Sun
Can Xu
Pu Zhao
Jian-Guang Lou
Chongyang Tao
Xiubo Geng
Qingwei Lin
Shifeng Chen
Yansong Tang
Dongmei Zhang
Venue: ICLR 2025 Oral (2025)
Keywords: Mathematical Reasoning
Evol-Instruct
Reinforcement Learning
Primary area: foundation or frontier models, including LLMs

Abstract: Large language models (LLMs), such as GPT-4, have shown remarkable performance in natural language processing (NLP) tasks, including challenging mathematical reasoning. However, most existing open-source models are only pre-trained on large-scale internet data and without math-related optimization. In this paper, we present WizardMath, which enhances the mathematical reasoning abilities of LLMs, by applying our proposed Reinforcement Learning from Evol-Instruct Feedback (RLEIF) method to the domain of math. Through extensive experiments on two mathematical reasoning benchmarks, namely GSM8k and MATH, we reveal the extraordinary capabilities of our model. Remarkably, WizardMath-Mistral 7B surpasses all other open-source LLMs by a substantial margin. Furthermore, WizardMath 70B even outperforms ChatGPT-3.5, Claude Instant, Gemini Pro and Mistral Medium. Additionally, our preliminary exploration highlights the pivotal role of instruction evolution and process supervision in achieving exceptional math performance.

Published as a conference paper at ICLR 2025
WizardMath: EMPOWERING MATHEMATICAL REASON- ING FOR LARGE LANGUAGE MODELS VIA Reinforced Evol-Instruct
Haipeng Luo²> Qingfeng Sun
Can Xu
Pu Zhao
Jianguang Lou" Chongyang Tao?
Xiubo Geng" Qingwei Lin Shifeng Chen
Yansong Tang Dongmei Zhang
"Shenzhen International Graduate School, Tsinghua University
2 Microsoft Corporation
Shennhen Institute of Advanced Technology, Chinese Academy of Sciences
(luohp248mal 44mails. tang. yansongesz. itsinghua. edu. en
(caxu, qins, puzhao, jlou, chotao, x geng, glin, dongmeiz ) 8mi www.ccooooo..oomooo com (shi feng. chen) @s auiiatc. ac.
ABSTRACT
Large language models (LLMs), such as GPT-4, have shown remarkable perfor- mance in natural language processing (NLP) tasks, including challenging mathe- matical reasoning. However, most existing open-source models are only pre-trained on large-scale internet data and without math-related optimization. In this paper, we present WizardMath, which enhances the mathematical CoT reasoning abilities of LLMs without using externll python tools, by applying our proposed Reinforce- ment Learning from Inss-unst Feedback (RLEIF) method to the domain of math. Through extensive experiments on two mathematical reasoning benchmarks, namely GSMSk and MATH, we reveal the extraordinary capabilities our model. Remarkably, WizardMath-Msster 7B surpasses top-tier open-source LLMs by a substantial margin with higher data efficiency. Furthermore, WizardMath 70B even outperforms GPT-3.5- 5urbbo, Claude 2. Gemini Pro and GPT-4-carly-vrrsion Additionally, our preliminary exploration highlights the pivotal role of instruction evolution and process supervision in achieving exceptional math performance. For more details refer to /ttpp://gittbbs www.c///wwwccc////cccc zaazzai
INTRODUCTION
Recently, Large-scale language models (LLMs) have garnered significant attention and become the go-to approach for numerous natural language processing (NLP) tasks, including open domain
conversation (Ouyang al. 2022 OpenAl 2023 Touvron al.) (023a).. coding (Chen al. 202T
Wang et al. 202 Li et al. 2023b) and math (Taylor et al.) 2022 Lewkowye et al.] 2022
Shao
ct al. 2024 Yang et al. 2024). conspicuous example ChatGP TV. developed by OpenAl. This
model uses extensive pre- training large-scale internet data and further fine-tuning with specific instruction data and methods. As a result, it achieves state-of-the-arr zero-shot performance on various benchmarks. Subsequently, Anthropic, Google, and Meta also launched their competitive products one after another. Notably, Meta's series of Llama (Touyron et al.) 2023a b) Dubey et al.) 2024) have sparked open-source revolution and quickly narrowed the gap with those closed-source LLMs. This trend also gradually stimulates the releases of Mistral (Jiang et al. 2023), Alpaca (Taori al. 2023), Vicuna (Chiang et al., 2023). and WizardLM (Xu et 2023), etc. However, these open models still struggle with the scenarios which require complex multi-step quantitative reasoming,
such solving mathematical and science challenges (Ahn al. 2024 Long al. 2024).
Chain-of-thougee (CoT) (Wei et al. 2022) proposes to design better prompts to generate step-by- step solutions, which can lead improved performance. consisonnsttecyy (Wang al.) 2022)
Equal contribution. Work done during the internship of Luo Microsoft Research.
1 Corresponding author.
https://opens. comm

---

Published as a conference paper at ICLR 2025
Skp Math Eval-instred and superviedd finx-tuning
Sep-
Sep Reinfrremunt Learaing with IRM and PRM.
Instruction Roward Medel (II and supeeess-ppevviidd Reward Madel (PRM).
Pooi-lratnet
E
C2523
Instrudtin Banking
SupsL Tablling
SFT
C>A-E-B-D
CB
Figure 1: diagram illustrating the three steps of our Reinforcement Learning from Evol-Instruct Feedbuck (RLEIF). For detailed explanation of the training pipeline, refer to Appendix A.6
also achieves remarkable performance on many reasoning benchmarks, which generates several possible answers from the model and selects the correct one based on majority vote (Fu et al. 2022). Llemma (Azerbayev al. 2023) and MathPile (Wang al., 2023c) continue pretraining LLMs with math corpus to improve domain capacity. MetaMath (Yu al.) 2023b) and Xwin-Math (Li et al. 2024a) bootstraps mathematical questions by augmenting the question from multiple perspectives. MAmmoTT (uue et al. 2023) and TORA (Gou et al.) 2023) presents a unique hybrid of CoT and program-of-thon
(PoT) ensure extensive coverage of diverse fields in math. Recently, Evol- Instruct is an effective method for large-scale data synthesis using LLMss. It has been widely verified and proven be effective in enhancing the model's instruction following capability. It employs In-depth Evolving and In in-breadth Evolving to automate the generation of diverse and complex open-domain instructions using L.Lss, instead of relying on human-crafted instruction datasets. In- depth Evolving incrementally enhances instruction complexity by introducing additional constraints, deepening, concretizing, increasing reasoning steps, and complicating input. In- In-breadth Evolving focuses improving topic diversity and dataset richness by creating entirely new instructions. To enhance the correctness of each step in the model's generation process, (Wang et al., 2024a Chen et al. 2024a: Lightman et al. 2023) finds that process supervision with reinforeement learning significantly outperforms outcome supervision for solving challenging MATH problems.
Inspired by Evot-Instruct and Process-supervicn Reinforcement Learning, this work aims to enhance the mathematical reasoning abilities the LLMs. As shown in the Figure | we propose a new method named Reinforcement Learning from Evol-Instruct Feedback (RLEIF), which could firstly generate diverse math instructions data by brand-new Math Evol-Instruct, which includes two downward evolution and upward evolution progress produce the grade school math and challenging high school math respectively. However different from WizardLM (Xu et al. 2023) and WizardCoder (Luo al., 2023). which mainly focus on the SFT stage and are susceptible to learning hallucinated information from the teacher model, we innovatively introduce PRM to address the issue in the problem-solving process. Moreover, to prevent instruction evolution from spiraling out of control, we incorporate instruction reward model (IRM) as mitigating strategy. Thus, we train an instruction reward model (IRM) and process-supervised reward model (PRM) (Lightman) et al. 2023 Uesato et al. 2022 Wang al.) 2024a Chen et al. 2024a). the former indicates the quality of the evolved instruction and the latter offers feedback for each reasoning step in the solution. Initially, finetune LLMs with the evolved math data. Immediately,
leverage GPT-4 to produce the ranking order instructions, and the correctness of each reasoning step, then optimize the LLMs obtain the reward models. Finally, implement the step-by-step PPO to train our WizardMath.
2

---

Published as a conference paper at ICLR 2025
We perform experiments on two widely used mathematical reasoning benchmarks, namely GSM8K (Cobbe et al.] (02T) and MATH (Hendrycks et al.) 2021) covering math problems from grade to high school levels, the results show that our WizardMath outperforms all other open-source LLMs at the same model size, achieving state-of-the-an performance. For instance, WizarlMath-700 signif- icantly outperforms MetaMatth7708 by a significant margin GSM8K (92.8 vs. 82.3) and on MATH (58.6 vs. 26.6). Specifically, WizardMath- Mistrall-B observed a substantial improvement in pass @ with an increase of +2288 (90.7. vs. 77.9) on GSM8K, and +26.8 (55.4 vs. 28.6) on MATH compared to MetaMath-Mistral BB.. Notably, our 70B model even also significantly surpasses those powerful proprietary LLMs, such as GPT-3.5-Turbo, Claude 2 (Bai et al. 2022), Mistral Medium (Jiang et al., 2024). Gemini-Pro (Team] 2023). PaLM-2 (Anil et al.) 2023) and GPT-44early-versionn
The main contributions of this work are as follows:
We introduce WizardMath model, which enhances the LLMs' mathematical reasoning abilities across range of problem difficulties, from grade to high school levels.
We propose new fully Al-powered automatic reinforcement learning method, Reinforce- ment Learning from Evol-Instruct Feedback (RLEIF), alongside Math Evol-Instruct and Process Supervision, for improving reasoning performance.
WizardMath surpasses top-tier open-source LLMs by substantial margin with higher data efficiency and also significantly outperforms various proprietary LLMs on both GSM8k and MATH, demonstrate the effectiveness of our RLEIF.
2 RELATED WORK
Large Language Models. LLMs have significantly advanced Natural Language Processing, with
models like OpenAl's GPT Series (Brown et al., 2020a OpenAl 2023), Anthropic's s Claude (Bai
ct al. 2022). Google's PaLM (Chowdhery et al. 2022 Anil et al. 2023). Gemini (Team 2023
and Gemma (Team et al. 2024) featuring billions of parameters and trained on massive textual
datasets. The AI field has also seen rise in open-source LLMs such Mistral (Jiang et al. 2023).
Llama Series (Touvron et al. 2023a b Dubey et al. 2024 Taylor et al.) 2022), DeepSeek (Bi et al.
2024 [Shao et al.] 2024), Qwen (Bai et 2023 Yang et al. 2024) etc. Notably, Llama serves as
foundational model for supervised line-tuning leading to the development of models like Alpaca,
Vicuna (Taori et al., 2023 Chiang et al. 2023).
Large Language Models For Mathematical reasoning, NLP models face challenges with complex reasoning, including mathematical (Long et al., 2024 Zhang et al. 2024b Xia al. 2024), common- sense (Talmor et al. (2019). Significant research focuses on Mathematical Word Problems (MWP).
which demand understanding of mathematical concepts and multi-step reasoning (Zheng et al. 2023
Zhao et al. 2023 Yuan et al.) (2023a). Models are tested on various MWP benchmarks (Roy & Roth 2015 Hendrycks et 22. Techniques like Chain-of-Thoughh Prompting (Wei et al. 2022).
Least-to-Mos prompting (Zhou al.) (022), and Complex CoT (Fu et al. (022) enhance reasoning by introducing multiple steps and breaking problems into sub-problems. There are some models 2023h), MathScale (Tang 2024) etc. Some models enhance mathematical reasoning by integrating python tools, such TORA (Gou et al. (023). TAmmoTT (Yue et al. 2023), Openmathinstruct (Toshniwal et al. 2024). NuminaMath (Li et al. 2024b) etc. In our work, we mainly improve the CoT reasoning ability of mathematics without using external Python tools.
aimed at improving math CoT reasoning skills such MetaMath (Yu et al.
eet al.] 2024). Xwin-Math (Li et al.] (024a). DART-Math (Tong et al.,
Reinforcement Learning for Large Language Models. State-of-the-at models often display logical eπrors and illusions, particularly in domains requiring complex, multi-step reasoning, leading to significant challenges (Bubeck et al. 2023 Maynez et al. 2020). Strategies such as training reward models help discriminate between desirable and undesirable outputs (Lightman et al.) 2023: Wu
approaches focused on algorithmic tasks (Li et al. 2016 Cai et al. 2017; Yu et al., 2023a), while recent research demonstrates the efficacy of reward models or validators in enhancing model performance (Cobbe et al.) 2021 Wang let al.) 2223a bl Li al. 2022). Reward models have also been incorporated into reinforcement learning pipelines and employed in rejection sampling to align Large Language Models (LLMs)
ct al. 2023 Chen et al. 2024b). Historically, outcome-
with human preferences (Shen al. 2021
Bai et al. 2022
Yuan et al. 2023b Dong al. 2023
3

---

Published as a conference paper at ICLR 2025
Song al. 2023
Touvron et al. 2023b Rafailov et al. 2024 Meng al. 2024 A contrast is
drawn between outcome-superrisee supervised and process-supervised reward models, with the latter being more effective at addressing discrepancies arising from incorrect reasoning paths leading to correct outcomes (Uesato et al. 2022; Zelikman et al., 2022 Creswell et al. 2022). Recent advances have promoted process-bases supervision through manual annotation, significantly benefiting LLMs
outcome-based approaches (Lightman et al. 2023 Wang et al., (024a) Sun et al., [2024] Chen et al.)
2024a Wang al. 2024b Zhang et al. 2024a). In our study, we leverage Al models like ChatGPT
to automatically offer process annotation to improve the efficiency of this research line.
3 METHOD
In this section, we elaborate the details of our WizandMath. Following WizardLM and PRMs (Light man et al. 2023), we propose Reinforcement Learning from Iyvol-Isttuct Feedbuck (RLEIF) method, which integrates the math Evol-Instruct and reinforced instruction and process supervision to evolve GSM8k and MATH, and fine-tune the pre-trained language models with the evolved data and reward models.
3.1 MATH EVOL-INSTRUCT
Motivated by the Evol-Instruct (Xu et al. 2023) method proposed by WiazrdLM and its effective application on WizardCoder (Luo et 2023). this work attempts to make math instructions with various complexities and diversity to enhance the pre-trained LLMs. Specifically, we adapt Evol- Instruct to a new paradigm including two evolution lines:
1) Downward evolution: It enhances instructions by making the questions easier. For example i): revising high difficulty questions to lower difficulty, or ii) producing a new and easier question with another different topic.
2) Upward evolution: Derived from original Evol-Instruce method, it deepens and generates new and harder questions by i) adding more constraints, ii) concretizing, iii) increasing reasoning.
The complete prompts of above evolution are shown in Appendix A.I For each instruction, we use GPT-4 to evolve 5 rounds (2 downward and 3 upward) of new instructions progressively, each new one is generated by the previous round evolution.
3.2 REWARD MODELS
Considering the necessity of quality control for evolved instructions and inspired by PRMs (Lightman) et al.) 2023), we train two reward models to predict the quality of the instructions and the correctness of each step in the answer respectively:
Instruction Reward Model (IRM) This model aims judge the quality of the evolved instructions on two aspects: i) Difficulty, and ii) Definition. To produce the ranking list training data of IRM, we leverage GPT-4 rank the quality between those evolved instructions and original instruction. The one with high difficulty and clear defimition will deserve higher ranking. The detailed prompt of above ranking process is shown in the Appendix A.2
Specifically, given an math instructions IRM (Q IR) assigns a score to to indicate its quality. We optimize ORM via the following pairwise ranking loss:
CIRM log (r3 r m)
where r is the reward of chosen instruction and r is the reward of rejected instruction, m is the margin.
Process-superviee Reward Model (PRM) As there is no simple way to support highly precise process supervision without professional and expensive human-labelers, we depend on GPT-4 to provide process supervision. and ask it to assess the correctness of each step in the solutions generated by our model to produce PRM training data. The detailed prompt of above step level labeling process shown in the Appendix A.3
4

---

Published as a conference paper at ICLR 2025
For exactly, given an math instructions and its answer PRM (Q x A + R+) assigns to each step of a, we train PRM with the following cross-entropy loss:
RPRM- Σ ys log rf (1 y:) log rf)
(2)
where L is the reasoning steps of answer a. yi is the ground-trum truth label of the i-th step of answer a, y/ 1 if a, is correct, otherwise y: 0. r. is the reward score (assigned by PRM) of the i-th step of answer a.
3.3 REINFORCEMENT LEARNING WITH IRM AND PRM
Immediately, exploit reinforcement learning to optimize LLMs. Following (Lightman al. )023), we employ step by step Proximal Policy Optimization (PPO) to reward both instruction and each reasoning step.
For each math instruction and generated answer a, we use IRM to assign instruction reward r", and the minimum score across all reasoning steps to represent the final reward score r" of the answer a assigned by PRM. Then we apply a product as the final reward of this instruction-answer pair:
r r
3.4 PRM FOR VERIFICATION
Following (Lightman et 2023) and (Li et al., 2023c). we leverage both majority voting and PRM verifier to aggregate the predictions of different reasoning paths.
arg max Σ Ue,=a
PRM a.)
(4)
where PRM a,) is the score of the i-th reasoning path assigned by PRM for instruction g. Ia,= an indicator function that returns 1(or 0) if a; a.
EXPERIMENT
This section provides comprehensive overview of the advanced models. Subsequently, mainly elucidate the performance metrics of our models on two prevalent mathematical benchmarks from grade high school problems: GSM8k (Cobbe et al. (02T) and MATH (Hendrycks et al.) 2021)
4.1 EXPERIMENTAL SETUP
SFT Training Data. Firstly, use the GSM8k and MATH training sets as the initial seed collection, then employ both upward and downward math Evol-Instruch approach for five rounds. Each round need to evolve the initial instructions times, and the temperature parameter is set to 0.7. Next, remove duplicate instructions 17k. Hence, total of 448k unique instructions Subsequently, 30k data were excluded by the data filtering method to avoid contamination, ultimately leaving 418k data. Finally, we use GPT-4-0613 3 generate the answer with a step-by-step format, and leverage them for supervised fine-tuning.
obtained.
Reward Models Training Data. To train the reward models, We conducted additional rounds of evolution on the initial instruction set and obtain 90k instructions. we use GPT-4-0613 to rank each instruction list with the quality from 1 to 6 as the training data of IRM. To obtain the training data of PRM, We use our Llama-2 70B SFT model to generate 5 answers for each instruction, and GPT-4-0613 employed to assign correctness judgement for each reasoning step.
Implementation Details. We employ our method on two open-source foundational models Llama 2 (Touvron et al.) 2023b) and Mistral-7B (Jiang et al. 2023). Llama encompasses three distinct parameter sizes: 7B, 13B, and 70B. We utilize GPT-4-0613 for instruction evolution and the training data construction of reward models. For SFT, we train epochs, and the learning rate is 2e-5, le-5 and 5e-6 for Llama 7B/13B, 70B and Mistral-7B. The batch size is 512, and the sequence length is 2048. For the reward model, we train Llama 2 and Mistral-7B with learning rate 4c-6 and 1e-6 for one epoch. For RI, the Ir is 4 and le-7 for Llama 2 and Mistral-7B and train epoch.
5

---

Published as a conference paper at ICLR 2025
4.2 MAIN RESULTS
Table I: The models' CoT pass @1 I results on GSM8k and MATH without using any external python tool.
Table shows the CoT (Wei al., 2022
pass results of the current state-of models on GSM8K and MATH. In this study, to ensure equitable and cohesive evaluations, we report the socres of all models within the settings of greedy decoding and CoT with- out using any external python tool.
Purams
GSMSK MATH
Proprictary model
of OpenA) 2003
9888
GPT-ol-mtt
90.0
Gemiai-1 002
$6.5
Claude Sounet Baieral 2022
96.4
71.1
40 2024-0517
96.1
76.6
GiMI----artm-0123 OpenA] 10.3
GPTT4-2814
(origital xeminu)
Buichun:3 Yankel
Comparing with the proprietary Models. As shown in the Table [1 our WizardMath demonstrates notable superiority over various proprietary LLMs the GSM8k and MATH benchmarks in terms of pass@1: 1:
Chanke
PaLM2 Aniletal R03
Minero
120
5400
GP133
57.1
Upen Scurce Models (0.18 (B)
CIPr--2-Seall Rowwn al 202
0.18
6.0
5.4
GPT Medram
0.38
11.2
6.2
GPT-2 Large (boe ce
0.70
13.6
6.4
1) WizardMath-Liama 70B, the largest model, demonstrated exceptional perfor- mance the GSM8k and MATH, surpass- ing earlier versions of GPT-4, Claude-2, and Gemini Pro, and performing on par with GPT- 4-0314. It significantly outperformed GPT- 3.5-Turbo by 11.2% GSM8K and by 15.5% on MATH.
GPI-2-XL bhoe e 2000
1.50
15.4
6.9
GFT:2-Sualt
0:18
GFT-2.
0.38
Large
1,59
Wissr
Quan-Maab-2.5
1.58
86.7
68.6
Dubsyy
16
WitriMall-la Limma I-inntnnt
Llama
516
WizarMaant- lina
Elima $2
38
85.5
49.9
Open Source Modele (7II-11)
Limed Tenveomstal 5023h
7B
14.6
2.5
MAmIn 11 Aisee 2055
1laus-2
7B
50.5
10.4
2) WizardMath-Mistral 7B, the smaller- sized model, outperformed Baichuan on GSM8k (90.7 vs. 87.6) and sur- passed GPT-4-0314 on MATH (55.4 vs. 52.6), significantly exceeding the perfor- mance of GPT-3.5-Turbo and Gemini Pro. Meanwhile, WizardMath-Mainn Mathssral,, trained on Mathstral-7B-vO. demonstrated perfor- mance comparable to GPT-4-turbo-0125 Ad- ditionally, WizardMath- Mawn-Qwen, trained Qwen2.5 5-Mahh, surpassed 05T-4-024-00113 on MATH (77.8 vs. 76.6).
MuthScale Langttia 2014
Liama
715
06.3
31.1
MenaMatt TE baraug
1Jama-2
716
66.3
19.8
MugglMMalr In parso)
1aama-2
7B
68
Skywork
7B
Shepherd
7B
7B
Misin--n11 Bonselal
7B
42.9
12.9
MathScale TiinBii PiP
Misinl v0.1
7B
24.8
35.2
MMBOC Lime PM61
Misnil 10.1
7B
74.8
36.0
ManaMaal
Mismn--fi.1
7B
KPMan-Pai
Mismd--0..1
7B
MART-Mach Till 1. 1-1-
Mismd--6.3
7B
Skywork- Mann 7-0r
Mismd--0.1
7B
MAmmoTH2
7R
faZhang00
Mistnd-sff1
76
Xwin-Malh
Mistnd-s0.1
76
WizardMant
Mistral vo.1
78
WizzrdMantt MMtteMaattMMrrl
Mistral-x0.1
715
90.4
WizzrdMant-Mallet
Mutharal-S01
766
93.8
70.9
Qwes
0wes2.5
7B
Comparing with the Open-Source Models. The results presented in Table unequivo- cally indicate that our WizardMath-LLama 70B exhibits significant performance supe- riority over strong models in both the GSM8k and MATH benchmarks with higher data ef- ficiency across the range from 0. to 70B parameters. The detailed results are as fol- lows:
WizurdMaal Qwee
Qwen2.5
7B
DeepSesk3tan Bage Stan 2024
7B
64.2
36.2
NemitaMallees Lreat 2053h
DespeozkMal
76
75.4
55.2
MMIGC Lin Y.u 2004
DeqpScokktath
76
79.0
45.3
KPMaih-ems pomnn 21 2i 2025
DespScckbdath
7B
83.9
48.8
DeepSeakM Nir
DeepSeakbtath
7B
88.2
51.7
DARI--Mash long 11 7115
DeepSeckMMth
716
86
WiztrdMaall-- Mo
DespSerkMath
7B
MenaMaat ot X0/1
Lama
8B
27.3
20.6
MMBQC Die 1 MA11
Liama
8B
77.6
24.5
DAKT Maar TWnl Cel P0/4
Limm
8B
82.5
45.3
MAmmoT122T0 Tra N015
Llma
8B
36.1
42.8
Limi I-nngret
17 1a 1
1ama
80
86.5
51.9
FaZhang33 Zhno b1 2054
Liuma
8B
88.6
$1.0
WizzrsMalt 1100
Liuna
8B
90.3
58.8
Open-Sourre Models (1
1) With the same model parameter size, our model surpasses the previous best model such as MetaMath, MAmmoTH2-PIucn Math. Particularly, WizardMath- achieves a substantial improvement of 10.5%
Liama Teusron gtall
ManSScll
13B
Xwin- 70B
ManMaai
Liuna
13B
MussleMul
Liuna
13B
KPMath Pluc
Liama
138
Xwin-Mali (Dar 20090
Lhmns
15B
88.1
44.9
WinrrMaal
Llans
13B
89.7
50.6
on GSM8K and 32.0% on MATH compared to MetaMath-Llama 70B testing accuracy. In the Table 2 we show the detailed results of MATH subtopics with our WizardMath 70B model. Specifically, WizardMath-MMstrer 7B also surpasses top-tier open source models, outperforming Petaaath-Misthee 7B with a notable margin (90.7 vs 77.9 on GSM8k) and (55.4 4 vs 28.6 on MATH). It demonstrats the effectiveness
Open-Soume Morkch (708)
Liuma Teuston stall 20233
70B
56.8
13.5
MAmno TEL B
Liama 2
700
12.4
21.1
MansMal: Yegtii
1aan-=2
708
82.3
KPMall ioniin
1lan--2
70B
Xein-Mah,
70B
708

---

Published as a conference paper at ICLR 2025
Table 2: Results of pass (%) on MATH
Table 3: Explore the effects of PRM and IRM during PPO training.
subtopics (i.e., Intermediate Algebra, Geome- try) with WizardMath 70B model.
Models
GSM8K
MATH
MATH subtopics
WazardMath 70B
GPT-2-XI--1.5B WizariMath-SFT
51.9
18.3
Intermediate Algebra
36.3
PRM
55.8
22.1
Precalculus
38.9
PRM + IRM
58.9
25.4
Geometry
48,3
IJama2 7B: WizardMath SFT
77.4
35.6
Number Theory
58.5
PRM
81.7
39.9
Counting & Probability
54.8
Prealgebra
74.6
Mistral-BB: WizardMath-SFT
82.8
48.1
Algebra
78.5
PRM PRM + IRM
87.2
52.7
Overall
58.6
90.7
55.4
of our RLEIF method in enhancing mathematical reasoning capabilities across range of problem difficulties, from grade to high school levels.
2) By employing diverse pre-trained models (i.e., GPT-2, Llama 2, Mistral, Qwen, DeepSeek) base models, WizardMath demonstrated notable advancements on the GSM8k and MATH benchmarks. Specifically, MatarddWat---laaa22-- GSMSK and 41.0% on MATH. Similarly, SrrardMath-GPT2--1 43.5% improvement on GSM8k and 18.5% on MATH, performing on par with Llama2-70B and outperforming GPT-3.5 on GSM8K. This demonstrates that our RLEIF method is equally effective for smaller models in enhancing mathematical reasoning capabilities, proving scalability and robustness across various model backbones.
based Llama2-7B, improved performance by 69.5%
built on GPT2-XI, achieved
4.3 ANALYSIS
The impact training data size
We are curious about to how the training data size of different dataset construction methods impact the reasoning capacity of LLMs. Thus we conduct different num- ber of training instances from ours evolved data and MetaMathQA to fine tune Mistral 7B. As shown in the Figure 2 Math Evol- Instruct achieves superior data efficiency. Specifically, model constantly outper- forms MataMath by more than 3% ~ 6% on GSM8k and 15% 20% MATH un- der the same number of conditions. Our findings indicate that Math Evol-Instruct exhibits a higher potential upper bound compared MetaMath, thus demonstrat- ing the effectiveness of Evol-Instruct for math reasoning senario.
GSMBKIOurs!
CSMOKOMetMMch) MATH(DUrs)
MATH(MctaMath)
COMOINN
MATHHN)
68
-8
62
21
56
1.cc
60k
120k
200k
390k
Data Sizc
Figure 2: Accuracy of Mistral-7B fine-tuned in different sizes of augmentation data GSMSK and MATH
The impact of PRM and IRM during PPO training
To verify the contributions of the instruction reward model and process-supervisee reward model, we consider the following variants: (1) SFT + PRM: only use PRM in the PPO training. (2) SFT + PRM + IRM: use both IRM and PRM the PPO training. As shown in Table applying PRM alone for PPO training GSMSk and MATH yields 3%-4% improvement. When combined with IRM, an additional 2.5%-4% gain is observed. Thus, the integration of PRM and IRM results a substantial overall improvement of 6%.8%. So, the variant with PRM significantly outperforms the SET one without any PPO training (2) IRM also plays key role in the success of reinforcement learning, there is remarkable improvement when
conclude that (1) PRM is crucial to WizardMath, since
7

---

Published as a conference paper at ICLR 2025
combine PRM with IRM, further demonstrating the necessity of taking instruction's quality into account and correcting false positives in the problem-soke solviig process when we optimize the LLMs.
Table 4: The effect of different reward models during PPO training
Table 5: Results of reinforcement learning com- bined with validation. The SFT and Reward models are trained based on Mistral-7B. The ver- ifier is based on 256 sample outputs.
Models
GSM8K
MATH
Llama2-7B: WizardMath-SS -FT
77.4
35.6
Generators
Verifiers
GSM8K
MATH
ORM (ours)
79.1
36.8
Self Self-Consistenyy
90.7
57.5
PRM800K
79.7
38.7
SET
ORM PRM
93.0
58.3
Sheh--h
80.3
38.2
93.9
61.7
PRM (ours)
81.7
39,9
Self-Consisenncy
91.2
57.7
Mistral-7B: WizardMath-SFT
82.8
48.1
ORM
ORM
93.4
59.4
PRM
94.1
63.3
ORM (ours)
84.6
49.6
Self Consistency
92.3
59.3
PRM800k
85.4
50.8
SFT + PRM
ORM
94.1
60,8
MatthShepperd
86.1
50.3
PRM
95.2
64.7
PRR (ours)
87,2
52.7
The impact of Evol-Instruct turns. Ta- olluusissssssses the impact of combining downward and upward evolution SFT
Table 6: Impact of different Downward and Upward Evol- Instruct turns on Mistral-7B SFT. D-i refers to the i round of downward evolution, whereas U-i denotes the round of upward evolution. Ori is the original manually annotated 7.5k data of GSM8k and MATH.
training. Two rounds of downward evo- lution improved GSM8k by 14.8% (74.5 vs. 59.7) and MATH by 19.6% (34.7 vs.
15.1) over the original. Three rounds of upward evolution yielded 18.9% im- provement on GSM8k (78.6 vs. 59.7) and 27.4% improvement MATH (42.5 vs. 15.1). Furthermore, combining downward evolution based upward evolution resulted in an additional 2.6% improvement on GSM8k (81.2 vs. 78.6), total improvement of 21.5% over the original. Similarly, a 1.9% improvement on MATH (46.5 vs. 42.5). a 31.4% total
GSMSK
MATH
Ori
D-1 D-2
U-1
U-2
U-3
pas@1
Ori
D-1
D-2
U-1
U-2
U-3
pass1
On
X
X
X
13.1
30.3
X
70.5
X
X
X
28.7
X
8
X
33.4
32.6
Mach
Evol
x
70.2
X
X
X
V
30.9
X
34,7
V
77.1
V
38.6
V
x
x
78.6
V
42.5
40.3
V
v
V
V
79.8
V
X
44.6
V
46.2
improvement. These results underscore the complementary and significant effectiveness of upward and downward evolution.
ORM v.s. PRM; Human v.s. AI. Table 4 presents the performance of different answer reward methods for LLMs in terms of pass @ 1. As is shown: 1) Our step-by-step PRM significantly enhances the performance of both Llama and Mistral based SFT models. Specifically, the Mistral-7B powered by PRM achieves 87.2% and 52.7% GSM8K and MATH respectively. 2) PRM models consistently outperforms ORM on both GSM8k and MATH, indicating the effectiveness of step-by-step supervision. 3) The PRM trained on our fully Al-labeled data outperforms both the manually annotated PRM800K and Math-Shepherd, which utilizes MCTS tree search for annotation. When training WizardMath-MMistral--S
with PPO, our PRM improves upon PRM800K by 1.8% and Math-Shepherd by 1.1% % on GSM8k, while surpassing PRM800K by 1.9% and Math Sath-Sheeperr by 2.4% on MATH. This demonstrates powerful AI can also provide good process supervision quality, highlighting the effectiveness of utilizing AI to construct PRM training data.
PRM as Verifier. Table presents the performance comparison various generators with different verifiers GSM8K and MATH in terms of pass @ 256. We find that: 1) PRM verifier consistently demonstrates superior performance compared to Self-Consistency and ORM. Specifically, our SFT + PRM generator, enhanced by the PRM verifier, achieves 95.2% and 64.7% accuracy on GSM8K and MATH respectively. 2) When compared to ORM, PRM exhibits a more significant advantage on the more challenging MATH dataset which aligns with the findings in (Uesato al. 2022) and (Lightman et al.] 2023). This can be attributed to the fact that GSM8K involves fewer and less complex steps in problem-solving than MATH. 3) Particularly, the generator with PRM PPO training
8

---

Published as a conference paper at ICLR 2025
GSMBk
MATH
96 Problems Solved (Best-off (fst-of-N)
94
Problems Solved [Best- of--t)
62
PAM (ours)
90
56
54
52
SC
50
ORM
PAM (ours)
N number of solutions per problem
N number of solutions per problem
Figure 3: Performance of Mistral-7B SFT with different verification strategies.
surpasses those SFT and ORM PPO trained generators regardless of employing Seff-Consistency. ORM, and the PRM verifiers. This further demonstrates the effectiveness of our PRM.
Figure 3 also shows the performance of different Verification strategies across a range of candidate numbers from 1 to 256 on two benchmarks. The main observations are as follows: 1) PRM verifiers consistently achieves superior performance compared to both ORM and majority voting, and this superiority becomes more evident N increases. 2) For MATH benchmark, PRM trained on the Al-annotated datasets slightly surpassed the aanan-annotated PRMS00K.
Performance of Out-of-Domain. Ta- ble 7 presents the results of Wizard- Math the 7 out-of-domain evalu- ation results covering K-12, college, and competition level math problems, highlighting the following salient obser- vations: (1) With math Evol-Instruct and reinforcement learning, WizardMath consistently surpasses prior state-of-the. art open-source models (e.g. Meta- Math, MathScale) across scales, and achieves improvement of 5%-10% across tasks on average. (2) The accuracy of WizardMath-
Table 7: Performance of WizardMath on the 7 out-of- domain evaluation results covering K-12, college, and competition level math problems. The results of models the table refer to MWPBENCHI (Tang et al. 2024). "AGIE" stands for AGIEval. We report the models CoT pass results on MwpBench without using any external python tool
Models
College Math
TAL
Math23k
ApeZ10k
Hench Math
Gaokao
AGIE
Gaokao Math
SAT Math
AGLE
AVG
Propricsary models
GPE4 Turboo
24.4
51.8
76.5
61.5
35.4
28.2
68.6
49.5
44.0
23.2
Models buaved on LL-MA-2 138
LLaMA-2
7.9
0.7
is about 5.0% on the
MAmm/TII-Co CoT
17.3
39.5
28.1
5.9
4.9
20.5
17.5
higher than WizardMath-Llamm same size. Especially it exceeds GPT-3.5- Turbo (45.7 vs. 37.9) while being compa- rable GPT-4. This also indicates that Mistral-7B has more potential in math- ematical reasoning. (3) Especially difficult benchmarks (i.e., College Math, AGIE Gaokao Math), WizardMath out- performs MetaMath by significant mar- gin This demonstrates our model and RLEIF method has stronger robustness and better significant generalization abil- ity for invisible mathematical problems.
GAIR-Abel
7.9
21.1
42.2
27.8
7.0
4.9
30.3
20.2
31.6
9.6
MathScale 13B
20.4
38.1
61.1
43.7
20,0
12.3
55.8
35.9
Models based on LLaMA 2 78
7B
MAmmeTH-CaT
6.2
13.3
34.6
21.4
3.9)
2.7
19.6
14.5
GAIR-Abel
24.5
4.3
MenaMah
22.5
44.0
29.9
5.9
5.1
36.2
21.9
MathScale 7B
20.9
35.2
59.0
41.8
19.6
12.6
57.8
35.3
WizniMaath
Mudill hased Misral 7R
15.5
MetaMath Mistrl
15.7
31.4
55.1
38.1
15.3
10.1
50.9
30.9
Misral
WizeeMaal Misral
24.8
44.8
71.2
52.6
37.2
24.5
64.7
45.
Employ Open-source Model Math Evol-Instruct. In Table
19 we investigate the use of
open-source models (i.e., Llama-3-700 70--Instr as substitute for GPT-4 during the SET stage for Evol Instruct, employing the same evolution strategy. The results demonstrate that WazardMath-

---

Published as a conference paper at ICLR 2025
Table 9: A case study from GSM8k test set. We rate the response using PRM and ORM. Red text denotes the wrong reasoning steps which PRM successfully detected, but ORM failed.
Question
The farm has 30 cows and the 200 has 20 sheep. The zoo has twice as many cows as the farm does. The farm has half as many sheep the wo does. How many animals do the farm and zoo have combined?
Ground truth answer: 120
Step 1: we know that the farm has 30 cows. PRM Score: 0.95.
Step 2: are told that the zoo has twice many cows the farm does. PRM Score: 0.97.
Step 3: So, we can calculate the number cows in the zoo by multiplying the number of cows
in the farm by 2. So, 2 30 = 60, So, the zoo has 60 cows. PRM Score: 0.97.
Step 4; we know that the farm has half as many sheep as the zoO does. PRM Score: 0.96.
Steps
calculate the number of sheep in the farm by dividing the number of sheep in the zoo by 2. So, 20/ / 10. So, the farm has 10 sheep. PRM Score: 0.93.
Step 5; So, we
Step 6: Now, we can calculate the total number of animals in the farm and the zoo by adding the number of cows and the number of sheep in both places. PRM Score: 0.92.
Step 7: So, 30 (farm's s cows) 60 (zoo's 'ss cows) + 10 (farm's 's sheep) 100. PRM Score: 0.11.
Step 8: So, the farm and the zoo have total of 100 animals combined. PRM Score: 0.10.
Step 9: Therefore, the final answer is 100 The answer is: 100. PRM Score: 0.06. ORM Score: 0.89.
Llama3-Evol achieved 33.8% improvement GSM8K and a 30.6% improvement on MATH, indicating that the math evol instruct strategy remains effective on open-source models. How- ever, compared to GPT-4 evolution, there is still a 5%-66 performance gap. Despite this, the strategy shows significant potential in balancing computational cost and accuracy.
Table 8: The impact of using open source models for Math-Evol and use Mistral-7B-v0.11 for SFT Models
GSM8k
MATH
Mistral-7B 7B-v0.1
42.9
12.9
82.8
48.1
WizarMMath-SFT-LLammad
76.7
43.5
4.4 DATA CONTAMINATION CHECK
Apart from the performance analysis, we also investigate whether evolution leads to the data contami- nation between training data and test set. To address this consideration, we employ instructions in the GSM8k and MATH test set as queries to retrieve the top-5 samples from all evolved training data with embedding model, gte-large (Li al., 2023d). Additionally, employ GPT-4 provide similarity judgement between the test sets and the retrieved samples, and remove the top-2 similar instructions. The prompt and details are shown in Appendix An4 A.5 Figure 4 illustrates that the evolution process does not yield higher similarity scores.
4.5
CASE STUDY
Evol-unstruc.. The Examples 3 and 4 in the Appendix shows the prompt and corresponding cases of GSM8k and MATH instruction evolution, demonstrating that the evolved instructions exhibit more complexity and diversity than the original training set.
PRM v.s. ORM. We present comprehensive case study to illustrate the effectiveness of our PRM. As delineated in Table PRM demonstrates precise performance on a challenge math problem from PRM effectively distinguished the incorrect solution, in the
the GSM8k test set. Remarkably, meanwhile the ORM struggled in this task. Furthermore, PRM demonstrated exceptional insight by accurately detecting the incorrect steps of the solution chosen by ORM, specifically the steps 7, 8, and 9. Subsequently, PRM also assigned lower score logits to these erroneous steps.
5 CONCLUSION
This paper introduces WizardMath, a mathematics model fine-tuned with RLEIF, The experimental results demonstrate that WizardMath achieves SOTA performance surpassing existing open-source LLMs on GSM8k and MATH from grade to high school problems. Notably, WizardMath 70B exhibits superior performance compared to some of the well-known proprietary LLMs, including ChatGPT- 3.5, Claude Instant, PaLM-22, Gemini Pro. Furthermore, our preliminary exploration highlights the pivotal role of instruction evolution and process supervision in achieving exceptional performance.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. Large language models for mathematical reasoning: Progresses and challenges. arXiv preprint arXiv:2402.001577 2024.
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305. 10403, 2023.
Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics. arXiv preprint arXiv:2310 10. 10631, 2023.
Jinze Bai, Shuai Bai, Yunfei Chu. Zeyu Cui, Kai Dang. Xiaodong Deng. Yang Fan, Wenhang Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, K. Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan. Jianhong Tu, Peng Wang. Shijie Wang. Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang. Jian Yang, Shusheng Yang. Shusheng Yang, Bowen Yu, Yu Bowen, Hongyi Yuan, Zheng Yuan, Jianwei Zhang. Xing Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. ArXiv, abs/2309. 16609, 2023. URL https: /atp./app..aa semant westtciccchharrc Www.aaorrooCoouuool/ 263134555
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08007. 2022.
DeepSeek-AI Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng. Honghui Ding, Kai Dong, Qiushi Du Zhe Fu, Huazuo Gao, Kaige Gao, Wenjun Gao, Ruiqi Ge, Kang Guan, Daya Guo, Jianzhong Guo, Guangbo Hao, Zhewen Hao, Ying He, Wen-Hui Hu, Panpan Huang. Erhang Li, Guowei Li, Jiashi Li, Yao Li, Y. K. Li, Wenfeng Liang, Fangyun Lin, A. X. Liu, Bo Liu, Wen Liu, Xiaodong Liu, Xin Liu, Yiyuan Liu, Haoyu Lu, Shanghao Lu, Fuli Luo, Shirong Ma, Xiaotao Nic, Tian Pei, Yishi Piao, Junjic Qiu. Hui Qu, Tongzheng Ren, Zehui Ren, Chong Ruan, Zhangli Sha, Zhihong Shao, Jun-Mei Song. Xuecheng Su, Jingxiang Sun, Yaofeng Sun, Min Tang, Bing-Li Wang, Peiyi Wang, Shiyu Wang, Yaohui Wang, Yongji Wang, Tong Wu, Y. Wu, Xin Xie, Zhenda Xie, Ziwei Xie, Yi Xiong, Hanwei Xu, Ronald X Xu, Yanhong Xu, Dejian Yang, Yu mei You, Shuiping Yu, Xin yuan Yu. Bo Zhang, Haowei Zhang, Lecong Zhang, Liyue Zhang, Mingchuan Zhang, Minghu Zhang, Wentao Zhang, Yichao Zhang, Chenggang Zhao, Yao Zhao, Shangyan Zhou, Shunfeng Zhou, Qihao Zhu, and Yuheng Zou. Deepseek 1lm: Scaling ooen-source language models with longtermism. ArXiv, 0ass/2001.229555 2024. URL htt.httt:://eet semant wwcholar.......oooooccc www.hhollrrrrgg/ooopppoo TD: 26681 8336
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Marc Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020a. URL https::/ppocceeiinottto/ooooo/et wtwp:eeeeeddinn...uuuisss.p.s.ps CC wttp:/appe//22/////// 1457c0d6bfcb496774 2rtroe/c2....11aaaoaa2222226a-e2
html
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Horbert-Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter. Christopher Hesse, Mark Chen. Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020b.
11

---

Published as a conference paper at ICLR 2025
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4, arXiv preprint arXiv: 2303.12.03.. 2023.
Jonathon Cai, Richard Shin, and Dawn Song. Making neural programming architectures generalize 2017.
via recursion.. arXiv preprint
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray. Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang. Igor Babuschkin. Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021.
Zhaorun Chen, Zhuokai Zhao, Zhihong Zhu, Ruiqi Zhang, Xiang Li, Bhiksha Raj. and Huaxiu Yao. Autoprm: Automating procedural supervision for multi-step reasoning via controllable question decomposition. arXiv preprint arXiv:2402.11452, 2024a.
Zhipeng Chen, Kun Zhou, Wayne Xin Zhao, Junchen Wan, Fuzheng Zhang, Di Zhang, and Ji- Ji-Rong Wen. Improving large language models via fine-grained reinforcement learning with minimum editing constraint. arXiv preprint urXiv:2401.0600 2024b.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality, March 2023. URL https
7 //vicuna. Imsas..mmsss.oommmmmmmm org
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan. Shivani Agrawal. Mark Omernick, Andrew M. Dai. Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways, 2022.
Douglas Eck, Jeff Dean,
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 14166. 2021.
Antonia Creswell, Murray Shanahan, and Irina Higgins. Selection-inference: Exploiting large language models for interpretable logical reasoning. arXiv preprint arXiv:2205.09711 2022.
Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint ar -XXx:2334.00677. 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178.3 2024.
Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. Complexity-basee prompting for multi-step reasoning. arXiv preprint arXiv:22 10.0022, 2022.
12

---

Published as a conference paper at ICLR 2025
Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, et al. Chatglm: A family of large language models from glm-1300 to glm-4 all tools. arXiv preprint vrXiv:2406.12793 2024.
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujuu Yang, Minlie Huang, Nan Duan, and Weizhu Chen. Tora: A tool-integrated reasoning agent for mathematical problem solving, 2023.
Dan Hendrycks. Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv: 103.0:2.03.08874.. 2021.
Yiming Huang. Xiao Liu, Yeyun Gong, Zhibin Gou, Yelong Shen, Nan Duan, and Weizhu Chen. Key-point-drrven data synthesis with its enhancement on mathematical reasoning. arXiv preprint arXiv:2403.02333, 2024.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b, arXiv preprint arXiv:2310.0683 2023.
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie- Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang. Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral experts, 2024.
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. arXiv preprint arXiv:2206. 14858, 2022.
Chen Li, Weiqi Wang, Jingeheng Hu, Yixuan Wei, Nanning Zheng, Han Hu, Zheng Zhang, and Houwen Peng. Common 7b language models already possess strong math capabilities. arXiv preprint 4rXiv:2403.04706, 2024a.
Chengpeng Li, Zheng Yuan, Guanting Dong, Keming Lu, Jiancan Wu, Chuanqi Tan, Xiang Wang, and Chang Zhou. Query and response augmentation cannot help generalization. ArXiv, abs/2310.05506, 2023a. URL http:::t/////t. api. semanticscholaa (org/Crrpus
math reasoning
Chengtao Li, Daniel Tarlow, Alexander L. Gaunt, Mare Brockschmidt, and Nate Kushman. Neural program lattices. In International Conference on Learning Representations, 2016. URL https
7/api. www.eantnsscchho.ooooo...o.
wtg. Sttg/CorpussDD. 34811674478
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. 2024b.
Raymond Li, Loubna Ben Allal, Yangtian Zi. Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06/60 2023b.
Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, B. Chen, Jian-Guang Lou, and Weizhu Chen. Making language models better reasoners with step-aware verifier. In Annual Meeting of the Associa- tion for Computational Linguistics, 2022. URL httpt::t///app wwwant
org/
Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. Making language models better reasoners with step-aware verifier. In Proceedings of the 61st Annual Meeting the Association for Computational Linguistics ( (Volume 1: Long Papers). pp. 5315-5333 Toronto, Canada, July 2023c. Association for Computational Linguistics. doi: 10. 10.18653/v1/2023 1/2023. //acl acl-long.221. URL ot..ntttpe///eeiiaatooooo...... wrg.:llloooorr//233333 acl ong. 291
13

---

Published as a conference paper at ICLR 2025
Zehan Li, Xin Zhang, Yanzhao Zhang. Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi multi-stage contrastive learning. ArXiv, 0122223080022811 2023d.
htttht:s///aat semant i weschocachrcrcc wrg/Corpusli:i 20682255
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20050, 2023.
Haoxiong Liu and Andrew Chi-Chih Yao. Augmenting math word problems via iterative question composing, arXiv preprint arXiv:2401.09003, 2024.
Lin Long, Rui Wang, Ruixuan Xiao, Junbo Zhao, Xiao Ding, Gang Chen, and Haobo Wang. On llms-driven synthetic data generation, curation, and evaluation: A survey. arXiv preprint arXiv:2406.1511 15..0 2024.
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardeoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.0888 08566, 2023.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. On faithfulness and factuality in abstractive summarization, arXiv preprint arXiv:2005.000 0,066, 2020.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference- free reward. arXiv preprint arXiv:2405. 14734, 2024.
OpenAI. Gpt-4 technical report, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In /hash/ http:////iiieppea/////eee/ee
NeurlPS, 2022. TRL.ttpp::t. b 1.5.6653le33644331111553383 oa0eaa4400444400000033332222---0 html]
n hww.hhtte//pppprr...sss..s////es
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
Subhro Roy and Dan Roth. Solving general arithmetic word problems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1743-1752, Lishon, Portugal, September 2015, Association for Computational Linguistics. doi: 10. 10.18863/11/D15-12000 URL
org.111 1202
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingehuan Zhang, YK Li, Yu Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Jianhao Shen, Yichun Yin, Lin Li, Lifeng Shang, Xin Jiang, Ming Zhang, and Qun Liu. Generate & rank: multi-task framework for math word problems. arXiv preprint 2rXiv:2109.03034 2021.
Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. Preference ranking optimization for human alignment, arXiv preprint arXiv:2306. 1742,, 2023.
Zhiqing Sun, Longhui Yu, Yikang Shen, Weiyang Liu, Yiming Yang, Sean Welleck, and Chuang Gan. Easy-to-hard generalization: Scalable alignment beyond human supervision. arXiv preprint arXiv:2403.000 09472, 2024.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA:: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 4149-4158. Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18653/ 1421. URL https
7/acl hot ogy.org/Nl N-111111111111
14

---

Published as a conference paper at ICLR 2025
Zhengyang Tang, Xingxing Zhang, Benyou Wan, and Furu Wei. Mathscale: Scaling instruction tuning for mathematical reasoning. arXiv preprint arXiv:2403.02884 2024.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An enstruction-folloiing Ilama model. https://githuit www coom/tttss-llabb.... st anforddalppaaccc 2023.
Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science. arXiv preprint arXiv:22 .09085, 2022.
Gemini Team.
Gemini: A family of highly capable multimodal models, 2023.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, -ean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.11800 2023.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.0829. 2024.
Yuxuan Tong, Xiwen Zhang, Rui Wang, Ruidong Wu, and Junxian He. Dart-math: Difficulty-aware rejection tuning for mathematical problem-solving. arXiv preprint arXiv:2407.13690 2024.
Shubham Toshniwal, Ivan Moshkov, Sean Narenthiran, Daria Gitman, Fei Jia, and Igor Git- man. Openmathinstruct--l: A 1.8 million math instruction tuning dataset. arXiv preprint 2:Xi::2402.10176: 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprini arXiv:2302.. 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307 09288, 2023b.
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14275. 2022.
Peiyi Wang, Lei Li, Liang Chen. Feifan Song. Binghuai Lin. Yunbo Cao, Tianyu Liu, and Zhifang Sui. Making large language models better reasoners with alignment. ArXiv, abs/2339.022144 2023a. URL https:/ api www.ap....aaaaa.mm.mmm.........a semant
wttp:/ooopu..::22555555
Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. ArXiv, abs/2305. 179266, 2023b. URL
https:/ api semanticschen
org/ wttg//orpuslii ID 000225589333
Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce lims step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 9426-9439 2024a.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv: 2202222..1177 1171, 2022.
Yue Wang, Weishi Wang, Shafiq R. Joty, and Steven C. H. Hoi. Codet5: Identifier-awanl unified pre- trained encoder-decoder models for code understanding and generation. In Marie-Francinn Moens, Xuanjing Huang. Lucia Specia, and Scott Wen-tau Yih (eds.). Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 1 November, 2021, pp. 6696-8708 Association for Computational /tt:::/doii..ooo Linguistics, 2021. doi: 18.18653/v http:///112221mmmm/lll mammlp-miin6885. URL /
18653/v1/2021 emn 1p pm 685
15

---

Published as a conference paper at ICLR 2025
Zengzhi Wang, Rui Xia, and Pengfei Liu. Generative ai for math: Part matthpiee: A billion-token scale pretraining corpus for math. arXiv preprint arXiv:2312.17120, 2023c.
Zihan Wang, Yunxuan Li, Yuexin Wu, Liangehen Luo, Le Hou, Hongkun Yu, and Jingbo Shang. Multi-step problem solving through a verifier: An empirical analysis on model-induced process supervision. arXiv preprini arXiv:2402.002658, 2024b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint (tf:::2001.19900 2022.
Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for language model training. ArXiv, abs/2306.01693, 2023. URL https://api
semanticscholarr org/ http:/ooopuusoss...o
Shijie Xia, Xuefeng Li, Yixin Liu, Tongshuang Wu, and Pengfei Liu. Evaluating mathematical reasoning beyond accuracy. arXiv preprint vrxiv:2404.05692 2024.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. WizardIm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304. 20ti:22304.22244,, 2023.
Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309. 10300 2023.
An Yang, Beichen Zhang, Binyuan Hui, Bofci Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, al. Qwen2. 5-math technical report: Toward mathematical expert model via self- self-improvement. arXiv preprint arXiv:2409.1212.. 2024.
Fei Yu, Anningzhe Gao, and Benyou Wang. Outcome-superviseed verifiers for planning in mathe- matical reasoning. ArXiv, 025/23111.9922 2023a. URL Uptt icschoolitccccoaaa
ort//oorpusID::2552155
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models, 2023b.
Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Chuanqi Tan, and Chang Zhou. Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308.00885. 2023a.
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprins arXiv:2304.05330. 2023b.
Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning. arXiv preprint arXiv:2309.05653, 2023.
Xiang Yue, Tuney Zheng, Ge Zhang, and Wenhu Chen. Mammoth2: Scaling instructions from the web. arXiv preprint wrxiv:2405.03544 2024.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 15311566--5588, 2022.
Liang Zeng. Liangjun Zhong. Liang Zhao, Tianwen Wei, Liu Yang, Jujie He, Cheng Cheng, Rui Hu, Yang Liu, Shuicheng Yan, et al. Skywork-mn math: Data scaling laws for mathematical reasoning in large language models-the story goes on. arXiv preprint arXiv:2407.08.8
2024.
Dan Zhang, Sining Zhoubian. Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mets' Lim self- trffttraining via process reward guided tree search. arXiv preprint arXiv:2406.0334 2024a.
16

---

Published as a conference paper at ICLR 2025
Jiaxin Zhang, Zhongzhi Li, Mingliang Zhang, Fei Yin, Chenglin Liu, and Yashar Moshfeghi. Geoeval: benchmark for evaluating llms and multi-modal models on geometry problem-soie solving.. arXiv preprint arXiv:2402.10110 2024b.
Xu Zhao, Yuxi Xie, Kenji Kawaguchi, Junxian He, and Qizhe Xie. Automatic model selection with large language models for reasoning. arXiv preprini arXiv:2305. 14333, 2023.
Chuanyang Zheng, Zhengying Liu, Enze Xie, Zhenguo Li, and Yu Li. Progressive-hing hint prompting improves reasoning in large language models. arXiv preprint arXiv:2304.09797, 2023.
Denny Zhou, Nathanael Scharli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schu- urmans, Olivier Bousquet, Quoe Le, and Ed Huai hsin Chi. Least-to-most prompting ables complex reasoning in large language models. ArXiv, abs/2205. abs/2205.10625, 2022. URL
Kun Zhou, Beichen Zhang, Jiapeng Wang, Zhipeng Chen, Wayne Xin Zhao, Jing Sha. Zhichao Sheng, Shijin Wang, and Ji-Rong Wen. Jiuzhang3. O: Efficiently improving mathematical reasoning by training small data synthesis models. arXiv preprint arXiv:2405. /4365, 2024.
17

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 MATH EVOLUTION PROMPTS
Example I: Upward Evolution Prompt
Step 1:
Understand the core concept and structure of the "#Instructionf". Identify the key elements such as variables, conditions, participants, actions, or processes can manipulated increase com- plexity. Also, recognize the theme of the instruction and ensure it remains consistent throughout the evolution.
Step 2: Formulate comprchensive plan increment the complexity of the "#Instruction#" based the identified elements Step The plan should involve modifying expanding least three components from list. is crucial ensure that components the instruction are logically interconnected the complexity increase coherent and justified. The plan should avoid introducing variables or conditions without clear criteria for determining their values or without contributing the overall complexity. In this step, consider adding real-world constraints and dependencies between variables to make the problem more challenging.
you can also add more constraints, concretizing, increasing reasoning.
Step 3: Implement the plan step by step to create the "#Rewritten Instruction## Ensure the rewritten instruction maintains logical sequence and avoids ambiguity confusion. If additional variables conditions are introduced, provide clear and unambiguous methods or criteria for determining their values. "#Rewritten Instruction# should not exceed the original "#Instructionf!" by more than 30 words ensure readability and comprehension.
Step 4: Review the "#Rewriten Instruction#" thoroughly identify any unrcasonable elements inconsistencies. Make sure the "#Rewritten Instruction#" more complex version of the "#Instruction#"" and that accurately reflects the intended increase complexity. Adjust any part of the instruction that may lead to misunderstanding or ambiguity, and provide the "#Finally Rewritten Instruction# without any supplementary explanation.
Please reply strictly in following format:
Step 1
#Elements Identified:::
Step 2
#Plan#:
Step #Rewritten Instruction::
Step 4 #Finally Rewritten InstructionH:
#Instruction#:
18

---

Published as a conference paper at ICLR 2025
Example Downward Evolution Prompt
Step 1: Understand the "#instructionf"" and identify all the components that can be modified to decrease complexity, that it makes the instruction easier. These components be variables, conditions, participants, actions, etc. The key keep the core scenario unchanged while ensuring any new elements introduced do not cause ambiguity or confusion.
Step 2:
Develop comprehensive plan to decrease the complexity of the "#Instruction*" based the components identified in Step 1. The plan should involve modifying least three components from the list. is important to ensure that components
instruction are logically interconnected and that the complexity decrease is justifiable. The plan should avoid introducing variables or conditions without clear criteria for determining their values, Our goal is revising high difficulty questions to lower difficulty, or producing new and casier question with another different topic.
Step 3: Implement plan step by step to create "#Rewritten Instruction#". Make sure rewritten instruction maintains a logical sequence and avoids ambiguity or confusion. If additional variables or conditions are introduced, provide clear and unambiguous methods or criteria for determining their values. The #Rewritten Instructionf"" should not exceed the original "#Instructiont" more than 20 words.
Step 4: Review the "#Rewritten Instructionit" thoroughly to identify any unreasonable elements. Make sure the "#Rewritten Instruction#" a casier version of the "#Instructionf". Adjust any part of the instruction that may lead to misunderstanding ambiguity, and provide the "#Finally Rewritten Instruction# without any explanation.
Please reply strictly in the following format:
Step I
#Elements Identified#:
Step 2
#Planff:
Step 3
#Rewritten Instruction::
Step 4
#Finally Rewritten Instructioni:
#Instruction:::
19

---

Published as a conference paper at ICLR 2025
Example 3: GSM8k Evol Instruction Case
Original Instruction 1: Bill is trying to decide whether to make blueberry muffins or raspberry muffins. Blueberries cost $5.00 per ounce carton and raspberries cost $3.00 per 8 ounce carton. If Bill is going make 4 batches of muffins, and each batch takes 12 ounces of fruit, how much money would he save by using raspberries instcad of blucherries?
Evol Instruction I: Bill and Jane are contemplating between blueberry and raspberry muffins. Blueberries are $5.00 for 6 ounce carton, with a 20% bulk discount. Raspberries are $3.00 for 8 ounce carton. If they each make 6 batches of muffins, with each batch requiring 12 ounces of fruit, calculate the total money they would save by choosing raspberries over the discounted blueberries, given Jane's inclination towards raspberries.
Original Instruction 2: A snake's head is one-tenth its length. If snake is 10 feet long, calculate the length the rest of its body minus the head. Evol Instruction 2: Given a snake's 's head is a certain fraction of its total length, and the snake's 's total length is positive integer, determine the length of the snake's 's head by multiplying the total length by the fraction. Subtract this value from the total length to calculate the length of the rest of the snake's body.
Original Instruction 3: Thomas is training at the gym to prepare for a competition. Hc trained for 5 hours every day for month (30 days). If he continues train for the next 12 days, how many hours will he spend on training in total?
Evol Instruction 3: Thomas and James are preparing for a competition by training at the gym. They trained for 5 hours daily for a month (30 days), excluding a rest day each week. If they persist in training for the subsequent 12 days, adding an extra hour of training each week, what will be the total hours they have spent training?
Original Instruction 4: Travis is hired to take 638 bowls from the factory to the home goods store. The home goods store will pay the moving company a $1 $100 fee, plus $3 for every bowl that is delivered safely. Travis must pay the home goods store $4 each for any bowls that are lost broken. If 12 bowls lost, 15 bowls are broken, and the rest are delivered safely, how much should Travis be paid?
Evol Instruction 4: Travis and his team are tasked with moving 1000 bowls and 500 plates from the factory to a home goods store. The store agrees to pay $200 fee, plus $4 for each safely delivered bowl and $2 for each plate. However, Travis must compensate the store $5 for each lost broken bowl and $3 for each plate. If they lose 20 bowls and 10 plates, and break 25 bowls and 15 plates, how much should the store pay Travis and his tcam?
Original Instruction S: Gary is buying chlorine for his rectangular pool, which is 10 feet long, 8 feet wide, and 6 feet deep. Gary needs to buy one quart of chlorine for every 120 cubic feet of water in his pool, chlorine costs 53 a quart, how much does Gary spend on chlorine?
Evol Instruction 5: Gary and John are purchasing chlorine for their cylindrical pools, with diameters of 12 feet and 10 feet, and depths of 8 feet and 6 feet respectively. They require one quart of chlorine per 100 cubic feet of pool water. Given that chlorine priced at $4 per quart, calculate the total expenditure on chlorine for both Gary and John.
Original Instruction 6: Ken likes to bike when it's raining and can cycle 30 miles in 20 minutes during this time. However, when it's snowing Ken can't stand the cold and can only cycle 10 miles in 20 minutes. If rains times and snows 4 times in one week, how many miles did Ken reach if he cycles hour a day? Evol Instruction 6: In varying weather conditions, Ken's biking speed differs. He can cycle 30 miles in 20 sunny days. In week, if rains 4 times, snows 3 times, and sunny times, and Ken cycles for 1.5 hours each day, how many miles did he cover? Remember, after eycling for an hour, his speed decreases by 10%.
minutes when it's raining, 10 miles in 20 minutes when it's snowing, and 20 miles in 20 minutes
20

---

Published as a conference paper at ICLR 2025
Example MATH Evol Instruction Case
Original Instruction 1: Find the smallest positive integer whose cube ends in 888. Evol Instruction 1: Determine the least positive whole number, denoted by x', whose cube terminates in divisible by Verify the result by checking divisibility the cube by
Original Instruction 2: The sum of all the positive factors of integer is 24, If one of the factors is 3, what is the value of =? Evol Instruction 2: Given that summation of all positive factors an integer is 24, and considering ar is positive integer divisible by 3 with one of its factors being 3, determine the value of a by first calculating the variable S representing the sum of factors, and then solving for x.
Original Instruction 3: What is 2- + 2- 2- 2 2 2 (mod 13)? Express your answer
as an integer from O to 12, inclusive.
Evol Instruction 3: Iet S be the sum of the series 2 + 2- 2 2 2- Calculate S by finding the of each term, then determine the value of S (mod 13). Utilize the properties of modular arithmetic provide step-by-step solution. Express the final answer as an integer from 0 12, inclusive.
Original Instruction 4: Find the greatest common divisor of 40304 and 30203. Evol Instruction 4: Determine the greatest common divisor of the integers 40304 and 30203 by employing the Euclidean algorithm. Utilize prime factorization, considering Fundamental Theorem of Arithmetic, and verify if both numbers are divisible by the same prime factors.
Original Instruction Find remainder when
x 32 x... x 82 92 is divided by Evol Instruction 5: First, let P represent the product the series, which can be expressed as P IT (2 10m). Next, calculate the value of P. Then, determine the remainder, denoted R, when divided by 5. Ensure that R positive integer.
Original Instruction 6: Is the function f(z) |æ] even, odd, or neither? Enter odd, even, neither. Evol Instruction 6: Determine the function f(x) = [x] + exhibits parity (evenness or oddness) or neither, considering the mathematical definitions of even and odd functions. If æ O, introduce variable and compare (xr with ( y². Provide brief explanation for your antwer.Fnter f(x) even, f(x)
odd, or f(x) is neither even nor odd.
21

---

Published as a conference paper at ICLR 2025
A.2 IRM PROMPT
Example Instruction Quality Ranking Prompt
You a senior mathematics grading teacher in university, very skilled in high difficulty fields such as Intermediate Algebra, Precalculus, Prealgebra, Number Theory, Geometry. Counting Probability, Algebra and on.
Your task is to act as an impartial judge to evaluate the quality math problems based on their definition completeness and difficulty and rank set of maths problems according to these criteria. Make sure that your assessment takes into account the following rules:
1. | Problem statement completeness and correctness:**
Assess the clarity and accuracy of the definition of each math problem. Ensure that the problem statement provides sufficient information, conditions, constraints.
Consider whether the problem allows for multiple interpretations or if further clarification is needed.
Evaluate clarity of mathematical notation and terminology used the problem.
2. Con*Couul difficulty:
Evaluates the complexity
mathematical problem terms underlying concepts involved. Ensure solid and sound understanding of the underlying principles, or advanced mathematical concepts.
Consider the depth mathematical knowledge required to address and solve each problem.
Assess whether the problem encourages critical thinking and the application of mathematical principles.
3. Coomputationll complexity;**
Examine the computational complexity of each problem. Judge whether it involves complex calculations, algebraic operations, or non-trivial numerical operations.
Consider whether the problem requires sophisticated computational techniques or algorithms whether it can be answered with existing mathematical knowledge.
4.** Problem contextualisation:
Consider the relevance of each mathematical problem the given context or practical application. as well as being relevant or having meaningful meaning the practical context.
Evaluate whether the theory of the mathematical problem is detached from the facts, spurious, and non-existent.
Avoid any position biases and ensure that the order in which the math problems were presented does not influence your decision.
Do not allow the length of the problems to influence your evaluation. Do favor certain mathematical theory of the problems. Be objective as possible.
Below is a list of a set of math problems that you need to rank according to the rules above from most complete and clear (1) to least complete and clear (N) based on the comprehensiveness and difficulty level of the maths problem. Also, consider the difficulty level from most challenging (1) casiest (N), Your output needs to be placed the <Rank> <cRann>
And Your output is in JSON list format, where each element is a dictionary with three keys:
instruction: represents the math problem.
score: represents the result of your ranking for the problem.
reason: provide your explanation in detail for your ranking result.
|## Math Problems List ###]: <PROBLEMS HERE
22

---

Published as a conference paper at ICLR 2025
Math instructions
product two polynomials integer coefficients, Reason: The problem well-defined and clear. requires deep understanding of polynomial factorization, making it conceptually difficult, The computational complexity is high it involves the factorization of high-degree polynomial.
Instruction I. Write .: 2
Instruction 2. Express the following sum as common fraction:
1.2 2-3
1
9.
Rank: I. Reason: This problem is clear and complete, with well-defined mathematical notation. It roquires a good understanding
fractions, making conceptually challenging. The computational complexity moderate it involves the summation of fractions.
Instruction 3. Given that I read 6 novels monthly, each novel having 200 to 350 pages, and considering have or 13 months duc to a leap year or a year when choose to read more, also taking into account that my reading speed Bluctuate between of the novel can influcnce the number of pages (for instance, thriller may contain 300 to 350 pages, while a romance Reason:
may
GPT-4 Ranking
novels per day, and the number of days month be 28, 30, or 31, and the genre
novel may contuin to 250 puges), calculate the total pages novels problem is complex and requires multiple steps to solve, making it conceptually and computationally challenging, It is well-contextualized relevant,
year.
problem statement could clearer more concise.
Instruction 4. James decides to run sprints times week. He runs 60 meters each sprint. How many total meters does week? Rank: Reason: This problem cleur, plate, and well coniestualized. However, it is conceptually and computationaly less challenging than the others, as it only requires basic multppiccaiion.
Instruction What 2 times v Rank: Reason: This problem clear and complete, but lacks practical context. It is conceptually and computationa Bly less challenging than the others, as it only requires knowledge of the multiplication square roots.
Instruction 6. What is 2 times /3? Rank: 6. Reason: This problem clear and complete, but it lacks a practical context. is conceptually challenging involves multiplication number and imaginary number. However, the computational complexity is low.
Table 10: case study GPT-4 scoring the evolved instructions from two aspects:i) Difficulty, and ii) Definition.
23

---

Published as a conference paper at ICLR 2025
A.3 PRM PROMPT
Example 6: PRM Step Level Labeling Prompt
are a senior mathematics grading teacher in university, very skilled high difficulty fields such as Intermediate Algebra, Precalculus, Prealgebra, Number Theory, Geometry, Counting & Prohability, Algebra and on. Below is mathematical problem and its corresponding solution, well JSON list format for the solution, where each element is dictionary with two keys:
idx: represents the number of each step.
value: represents each step in the problem-solving process.
Firstly please provide your judgement whether the solution is correct. Your judgment (which must be only True or False) needs to be placed in the <Judge> </Judge>section.
And then you need to judge whether each step correct and give for each solving step the JSON list which needs be placed the <Scores> </Scoressseccion. There are three kinds of scores below:
I: indicates that the step correct.
0: indicates that the step is ambiguity, meaningless, or subtly misleading, or not helpful to the entire problem-solving process.
-1: indicates that the step is incorrect.
If this step leads to final wrong answer, then rate -1. If not, rate 1 or O.
Here are some rules about whether the solution's 's each step is correct:
Sobblem-Solving Thoughts ##: You should think about how to solve this problem, and then judge whether this step is correct.
## Calculation Accuracy ##: You should carefully check and verify whether each step calculated
correctly, including various mathematical numerical calculations. Notablely you don't need consider simplification.
## Logical Coherence ##: You should judge whether each step logically coherent and reasonable.
## Basic Theories and Principles #i:: You should judge whether each step correctly is using basic mathematical theories, principles. or formulas.
need constantly verify check repeatedly whether each step is correct. And rate each step carefully, honestly, and without bias, order, or discrimination. Your output is a JSON list format, where each element is dictionary with three keys:
idx: represents number of each step.
score: represents your rating for this step, which can only be -1, 0 and 1.
reason: provide your explanation detail for your rating whether each step of the pobllem-olvving process is correct.
[### Problem ###]: <INSTRUCTION HERE
[## Solution JSON List ###]: <Solution ISON LIST> >
24

---

Published as a conference paper at ICLR 2025
Question
John and his two brothers decide to split the cost apartment. It is 40% more expensive than John 's old opartment which costs $1200 per month. How much does John sove per year by splitting the apartment compared to living alone?
Step I: Calculate the cost of John's 's old apariment. John's is old apartment cosas $1200 per month. Score: 1.
Step 2:
Calculate the cost of the new apartment. Score: 1.
Step 'The upartmmnt custs 40% than John's old spartmennt need to find 40% of $1200 and add that to the cost the old apartment, Score: 1.
Step 4: 40% of $1200 is (40/100) $1200 $480. Sa the new apartment costs $1200 $480 $1680 month, Score:
Step 5: Calculate the savings for John by splitting the apartment. Score: 1.
Step 6: John lived alone, he would pay $1680 per month for the new apartment. Score:
Step 7: However, since be splits the cost with his two brothers, be only needs to pay 1/3 of the total cost. Score: 1.
GPT-4 Labeling
Step 8: So, John's share of the cost $1680 3 S560 month. Score: 1.
Step Calculate the savings per year for John. Senne:
total savings John year would 12 $560 $6720. Reason: The step is incorrect. The calculation is wrong. The total savings for John in a year should be calculated as the difference between what he would pay living alone and what pays when splitting cost, multiplied by The correct calculation should be (1200 560) 12 $7680.
Step 10: There
months year,
Step Il: So, John $6720 per year splitring the apartment compared to living alone. Score: -I. Reason: The step incorrect. The final answer wrong. The correct answer should be $7680.
Step 12: The answer is: 6720. Score: -1. Reason: The step is incurrect. The final answer is wrong. The currect answer should $7680.
Table 11: A case study from Mistral-7B model on GSM8k training set. Red text denotes the incorrect steps that GPT-4 to successfully label errors.
25

---

Published as a conference paper at ICLR 2025
Question
Compute cos 120"
Step First, neod to find right triangle with angle of 120" and then use the Pythagorcan theorem find the length of the hypotenuse. Score:
Step 2:
The first step is to visualize rig triangle that has an angle of 120%, Score: -1. Reason: This step incomrec..
right triangle cannot have angle 120 degrees.
Step can this by drawing line and then rotating it 120" around the midpoint of line. Soore: -1. Reason: 120 dogrees.
This step incorrect. Rotating line around midpoint does not create right triangle with
Step This will triangle with two angles of 60 dogrees and one angle of 120 degroes not right triangle.
triangle with 60 angles and one 120" angle. Saore: -I. Reason: This step incorrect.
Step 5: Next, need to find the length of the hypotenuse of this triangle. Score: O.
Step 6: We do this using the Pythagorean theorem, which stat that right triangle, the square of the length of the hypotemuse is equal to the sum the squares the lengths of the other two sides. Score: D.
Step 7: In this case, the other two sides of the triangle are the two logs of the 60 angles, which are cach unit long. Score:
This step incorrect. sides right triangle not correspond legs the 60 degree angles in this context.
GPT-4 Labeling
A² 1 = means length hypotemuse à V 'Now, to find the cosine of 120". Score: -1. Reason: This step is incorret. The calculation is based on incorrett assumptions from previous steps.
Step
The cosine of an angle defined as the ratio of the length the side opposite the angle to the length of the hypotenuse, Now, need find cosine of 120". Score: -1. Reason; This stop incorrect. The cosine of 120 degrees is not 1/sgrt(2).
Step 9:
Step 10: The cosine of angle defined as the ratio of the length of the side opposite the angle the length of the hypotenuse. Score:
Step 11: In this case, the opposite side is the leg of the 60 angle, which is unit long. So, we have cos 120 / Score: -1. Reason: This stop is entirely correct. The cosine of angle defined as the ratio of the adjacent side to the hypotemuse, not the opposite side the hypotenuse.
Step 12: Finally, we need to simplify thi expression. The square root of 2 is usually written as v2, so we rewrite cos 120 as 7/- Score: -1. Reason: This stop is incorret. The cosine of 120 degrees not 1/sqit(2).
Step 13: So, the final answer is
The answer is: V2
Score: -1, Reuson: This step incorrect, The final answer
not 1/sgrt(2). The correct answer is
Table 12: A case study from Mistral-7B model on MATH training set. Red text denotes the incorrect steps that GPT-4 to successfully label errors.
26

---

Published as a conference paper at ICLR 2025
Simliarity per Round
asMol Similariy
Similaity
6.42 4.13
4.34 4.0
392
6.00
180
Eval
Figure Average similarity scores between GSM8k, MATH samples and the top-1 retrieved data for each round.
A.4 DATA CONTAMINATION CHECK
Apart from the performance analysis, also investigate whether evolution leads to the data contami- nation between training data and test set. address this consideration, we employ instructions the GSM8k and MATH test set queries retrieve the top-5 samples from all evolved training data with an embedding model, gte-large (Li al. 2023d). Additionally, we employ GPT-4 to provide similarity judgement between the test sets and the retrieved samples, and remove the top-2 similar instructions. The prompt and details are shown in Appendix AA.5 Figure Appendix illustrates that the evolution process does yield higher similarity scores. Furthermore, similarity scores across all rounds remain relatively low. These findings indicate that the primary source of performance gain the introduction of more complex and comprehensive data based on our downward and upward instruction evolution.
A.S SIMILARITY CHECKING AND DATA FILTERING
The prompt formats to compute the similarity score between two given math problem tasks are as follow:
Example 7: System Prompt for Similarity Checking
Your task is evaluate the similarity of the given math problems. Please review the math problem lasks carefully, paying close attention to overlap in variables, conditions. participants, actions, processes, topics, and contents and core concept and structure. Once you have carefully reviewed both math problem tasks, provide similarity score between these two math problem tasks. The score should range from to 10 (1: completely different math problem tasks; 10: identical math problem tasks). You only need to provide your score without any explanation. # Problem-1 [task1] }
# Problem-2 [task2)
Your judgement score:
To thoroughly prevent data leakage from the GSM8k and MATH test datasets to the training dataset, implemented an additional data filtering step. Utilizing the SOTA embeddings model, gar-e,
27

---

Published as a conference paper at ICLR 2025
treated all test samples queries to extract the top 5 samples from the training data. Following this, GPT-4 was employed to evaluate the similarity between the retrieved samples and the test set.
A.6 DETAILED EXPLANATION OF OUR METHOD FLOW..
We offer a detailed clarification of our method flow, including the significance of colors and shapes as well as the direction of the arrows in Figure to facilitate clearer understanding.
In Figure T the various colored squares represent specific elements: blue squares denote original instructions, orange squares indicate evolved instructions, cyan squares signify model-genan -generated solution processes, and grey squares correspond to a series of training-related operations such as supervised fine-tuning (SFT), reward modeling, and reinforcement learning (RI... To enhance the mathematical reasoning capabilities large language models, we propose the RLEIF method, which integrates instruction evolution with reinforcement learning. This method consists of three primary steps:
1. Instruction Evolution and SFT
In the first step, we apply upward and downward instruction evolution on the GSM8K and the three blue arrows, from top to bottom, represent:
MATH datasets, generating evolved instructions for the SFT. On the leftmost side of Figure
(a) the adoption of the instruction evolution technique,
(b) the generation of evolved instruction data, and
(c) its application SFT training.
2. Reward Model Training
The second step involves two reward models: the Instruction Quality Scoring Reward Model (IRM) and the Process-Supervised Reward Model (PRM), depicted in the central section of Figure
IRM: We employ upward and downward evolution on a seed instruction, yielding five instructions (original evolved). These instructions are ranked by quality (e.g., C> = >>> B >>> D) using GPT-4. Based on the rankings, we train the Instruction Ranking Model (IRM) to assess instruction quality. In Figure []] this process is shown in the left-central segment: "A" represents the original instruction, while "B," "C," "D," and "E" denote the evolved instructions. The first blue arrow illustrates the ranking process via GPT-4, the second arrow shows the ranking outcomes, and the third arrow highlights the use of this ranked data to train the IRM.
PRM: In the middle-right section of Figure 1 the process for training the PRM is depicted. The SFT model generates step-by-step solutions from the given instructions, which are then evaluated and labeled by GPT-4. This labeled data subsequently used to train the PRM.
3. Reinforcement Learning with PPO
In the final step, we integrate the IRM and PRM within Proximal Policy Optimization (PPO)- based reinforcement learning framework. As depicted in the far-right section of Figure the process is as follows:
(a) The first blue arrow represents instruction scoring by the IRM.
(b) The second blue arrow shows PPO initialization and the start of reinforcement.
(c) The third blue arrow illustrates the policy model generating responses based on instruc- tions.
(d) The fourth blue arrow shows the scoring of each response step using the PRM.
(e) Arrows five through eight depict the combination of IRM and PRM scores to calculate the final reward score.
(f) The ninth blue arrow highlights the use of the reward score for the PPO training.
By integrating instruction evolution and reward- based optimization, the RLEIF method significantly enhances the reasoning capabilities of large language models.
28

---

Published as a conference paper at ICLR 2025
A.7 COMPARE OUR WIZARDMATH-SPF MODEL ACROSS VARIOUS BASE MODELS (0.1B-70B) WITH THE SOTA MODELS ON THE GSM8K AND MATH BENCHMARKS.
To provide more comprehensive and fair comparison, we have included the WizardMath-STi results in Table 13 and Table [4] These results evaluate the performance of WizardMath-SFT, trained exclusively using SFT, against current SOTA models across various base models. The key findings are summarized as follows:
1. Performance Comparison:
On Llama-2-7B and Mistral-7B-v0.1, WizardMath-SFT performs marginally below SOTA models Awin and Skywork-Math) and outperforms existing other excellent models (i.e., MART
On Llama-2-13B and Llama-2-70B, WizardMath-SFF achieves comparable perfor- mance to Awth-Math.
On all various base models, WizardMath-SFF surpasses most existing SOTA models trained solely with DAT(Gee,DARTTMath))
Notably, WizardMath-SER achieves these results using only 418K synthetic data points, significantly smaller dataset compared to DART-Math (580k-590k), Xwin-Math (1440K) and Skywork-Math (2500K).
2. Comparison with advaneed data synthesis methods (i.e., DART-Math, MetaMath) As shown in the following Table [5| MART-Math demonstrates strong performance across various base models and the data synthesis method proposed by DART-Math shows the effectiveness and outstanding performance. Meanwhile, WizardMath-STP SFT demonstrates comparable superior performance to advanced data synthesis methods, such as DART- Math and MetaMath, across all base models. Key observations include:
On Mistral-7B-ve and MeepSeekMath, WizardMath-S3e performs on par with DART-Math (Uniform & Prop2Difff GSM8k and surpasses DART-Math (Uniform Prop2Diff) on MATH;
On Llama3.2 1B, Llama3.2 3B, Llama3-8B, and Llama3.1-8B. Llama2-7B.
WizardMath-SEE exhibits 2%-7%% improvement over DART-Math (Uniform & Prop2Difff on the GSM8k benchmark. On the MATH benchmark, WizardMath-SFT outperforms DART-Math (Uniform & Prop2Diff) by approximately 5% 10%.
These findings highlight the effectiveness of the proposed Math Evol-Instruct for enhancing mathematical reasoning capabilities.
Notably, to compare the advanced data synthesis methods such as DART-Math and MetaMath on different base models, and ensure the same training settings as in our paper during the SFT stage, we employ a learning rate of 2e-5 for the Llama series base models (i.e., Llama2 7B, Llama3.1 8B, Llama3.2 2 IB, and Llama3.2 3B) and a learning rate of 5e-6 for Mistral-7B-vO. 1. All models are trained for 3 epochs with a batch size of 256, and 4 checkpoints
saved per epoch. Finally, select the checkpoint with the highest accuracy on the GSM8k and MATH benchmarks for reporting.
29

---

Published as a conference paper at ICLR 2025
Table 13: the study, we compare the VizardMathSSFT/I model across various base models (0.1B-3B) with the SOTA models the GSM8k and Math benchmarks. We report the Chain of Thought (CoT) @ results without using any external Python tools. The results from 7B to 70B shown in Table [4
Model
Base
Params
GSMSK
MATH
Proprictary
GPT-al (OpenAl 21733
94.8
GPT-ol-mam (UN0AA 2025
90.0
Gemini-1.5
Clande Sonner
Ball 21 MIPP2
96.4
71.1
GPT--4-2024405115 UpSIA
96.1
76.6
GPTT-4-urbo-0125
GPT-4-0314 (OperA) 2125
94.7
GPT (ariginh) Veramo [OtenA] 2023
92.0
42.5
Haichwan-1 Yans 217.5
KR.2
p314
Gemini
Piy Team 2175)
86.5
32.6
Clande2
85.2
32.5
Pal M2
Anil et
2174
8017
34.3
Minerva
kmn YCZ er 1022
540B
58.8
33.6
GPT3.5
Open-Source Models (0.1B-3B)
GPI-2-Small (Bren ct 202hh
0.18
6.9
5.4
GPT-2-Medient
GPT-2-Laage [Brow ar 20.00
0.78
13.6
6.4
GPT-2-XL (Browit et 20209
1.50
15.4
6.9
Witan|Mart:><P3311
GPT-2-Small
0.18
21.2
9.1
WiardNMat-GGPT-RL
GPT:2-Smal
0.18
264
WiarrMMath-GPT GPT-SFT
GPT-2- oPe-2-Medium
0.38
30.6
11.4
WiardMath-GPI-RR
GPT-2-Mediiu
0.38
38.7
15.6
WizandMah-GPT.SSt
GPT-2-Large Large
WiaanMMahh-PP--
GPT-2-1 Lare
0.78
50.1
21.2
WizarMMathGGPI-FFT
GPT-2-XL
1.58
51.9
18.3
WiaadMath-GPP-RL
GPT-2-XI
WizardMath- SPTT
82.3
WizardMach Owen RL
Owen Maih:
1.50
86.7
68.6
Llama-34 1.2ma----IInsmmic DDiee et 2024
Llama 3.2
18
44.4
30.6
MetaMath Yo
Liana
18
51.9
15.5
Prop2tii
DART-Matl-Unile Tong oral. 2021
Llama
1B
53.8
22.0
WizardMach Lloma S1T
Llama 3.2
1B
57.1
29.7
WiraddMarh--Jama
Llama
63.3
Llama-3 3.2lsssuet (Dubey el al. 2024
Llana
77.7
MetaMath Yu al. 20236
Llama 3.2
3B
72.6
25.9
DART-Matl Mroo-aop---oaaa Tode 2024
Llama 3.2
3B
74.0
Uniform (Tong crai
77.8
WwardMarhhL.lama--
Llama 3.2
38
80.3
45.2
WizardMath-Llam
Llana
3B
85.5
19.9
30

---

Published as conference paper at ICLR 2025
Table 14: Continue Table 13 in this study, we compare the WizardMath-SFT/AA model across various base models (7B-70B) with the SOTA models the GSM8k and Math benchmarks. We report the Chain of Thought (CoT) pass@1 results without using any external Python tools.
Open-Source Models (7B-8B)
Llans-2
Touvron 20236
7B
14.6
2.5
MathScale
Tane eiam M0244
Llama-2
7B
66.3
31.1
MetaMath
Liama-2
78
66.5
19.8
MugeleMaia
Liama-2
Skywork-Mali
C 5024
Llama-2
7B
72.9
47.7
Math Shepherd Wangg cr 2U36
Llama
7B
73.2
21.6
M14
Llama-2
78
DART-Math-Unnlom Jone
A024
Llama-2
7B
73.8
29.5
Xwin-Math [i 20245
Llama
7B
82.6
40.6
WicardMath-LLuie Lhiaa-h-------FF1
Llama-2
7B
77.4
35.6
Mistral-vo:
7R
MathScall Thane Fi MD24]
Misual-vO.1
78
74.8
35.2
MMIOC Lia
Mistra-vO.1
7B
74.8
36.0
MetaMadi
Misual-vO.1
DART Mais Prpptii Tont et 2024
Misral-vO.1
7B
81.1
45.5
KPMalh- Plus
HHang
P124
Mismll-V.1
7H
82.1
46.8
Skywork-Man Math Zeng 2t M744
Misral-vO.1
7B
83.9
$1.2
Math Shepherdd
Walle LE 2014
Misual-vO.1
7B
84.1
33.0
MAmmoTH2..PM
Mismal-v0.1
7B
84.7
45.0
0
2024
Mismal-V0.1
7H
Xwin-Math
Lid 20218
Misual-vO.1
7B
89.2
43.7
WizardMath-Miiinz Mistaathh-Mssil-333---
Mistral-vO.1
78
82.8
48.1
Mismal-v0.3
7B
84.5
49.9
WizardMath-Miite Ml-
Mistral-V0.3
78
90.4
55.6
WieardMath-Mallssin
Mathstral-v.11
7B
93.8
70.9
Owen2.5 Math Base Yang et 2024
Owen2.5 Math
7B
91.6
55.4
WizardMarh- SaardMMahh-Qeen--S SPT
Qwen2.5.Mine
7B
92.3
72.3
WizardMath-Quann
78
WicardMath- Sael---wen-FFF
Owen2.5
7B
89.8
68.1
WizardMall-Qweaitt
Qwan2.5
78
94.0
74.5
DeepSeekMath Base (Shao 2024
7B
64.2
36.2
NuniniMath-Col [L
DeepeecMath
78
75.4
55.2
MMIQC (Lia & Aux AD141
DogSScckMath
78
79.0
45.3
KPMalh- Plus Hiinng ale 2024
DeepScekMath
7B
83.9
48.8
DART- MatT--Mal--'oppDDDi
2024
DespScolMath
7B
86.8
53.6
DeepSeekMath SSaoo
DeepScekMath
MAKK-MMth-Lnnifari (rong compeeiii m024
DestSeakMathh
78
88.2
52.9
WiardMaath-DeupSeeee
Matt
7B
88.9
58.2
WizardMath DeepSeck
DeepSeeMMah
77.3
MMIOC [Lau 1200 2024
Llama
8B
77.6
29.5
DART-Man-Prppa Trbpe et 2124
Ilama
SB
81.1
46.6
DART-Math-Unllon
MAmmoTTH2 Plus
elll AMM
Llama
8B
84.1
42.8
Llama 3. I-Inssrucl Dibcy 2l 201
Llama
SB
84.5
$1.9
JizZhagg.0 0.0 Zhou
Llama
SB
88.6
$1.0
Hama
8H
WizardMath- Mannaaa
Llama
8B
90.3
58.8
MetaMath Yu 20236
Llama 3.1
8B
80.4
35.4
MART-MaiS-Piopur Trme al 2024
Llama
8B
84.3
46.5
Math-Unifom (Tong
WizardMach-l Laaa----
Llama 3.1
8H
89.2
55.8
WizardMath-Llamee RL
Llama
SB
93.4
62.3
Open-Sounce (13B)
Llama-2 Touvron cl 20230
13B
28.7
3.9
MAmix/ T8-Col Tua
2023
Llama
138
56.3
12.9
MahSSale
13B
MetaMarh Yirerii M1500
Llama
13B
72.3
22.4
MugeleMalit LTc 2124
Llama
138
74.0
KPMath-Plus:
Ilama
138
Xwin-Math Liat 20243
Llana
138
85.
44.9
WizardMath- Lummar
Llama 2
13B
86.8
46.5
WizrrMMath-L
Llama
138
Open-Souree Models (70B)
Llama-2 Touvren et
2023b
70B
56.8
13.5
MAmmo
Llama-2
70B
72.4
21.1
Yu
70B
KPMailsPoo friiin
3024
Llana-2
708
87.4
48.6
Xwin Math Lietal
Llama-2
70B
90.6
52.8
WizarMMaik
Llama-2
70B
92.8
58.6
31

---

Published as a conference paper at ICLR 2025
Table 15: In this study, mainly compare the performance of WizardMath-S3 with advanced data synthesis methods such as DART-Math and MetaMath on different base models under the GSM8K and MATH benchmarks in the SFT stage. We report the CoT pass @ results of the model without relying on any external Python tools.
Model
Base
Params
GSM8k
MATH
DART-Math-Pro2DDin
Llama
1B
49.2
MetaMath
Llama
1B
51.9
DART-Math-- Huah-Uniffrm
1B
55.8
Llama
1B
57.1
29.7
MetaMath
Llama 3.2
3B
72.6
25.9
MART-Math- Math-Prop2DDff
Llama 3.2
3B
74.0
37.8
DART-Math- Cuih-Uniform
Llama 3.2
3B
77.8
36.4
WizardMath MarruMMM------m-SP
Llama 3.2
3B
80.3
45.2
MetaMath
Llama-2
7B
66.5
19.8
DART-Math-Prop2DD5
Llama-2
7B
69.9
30.7
MART-MathUUnioon
Llama-2
7B
73.8
29.5
WizardMath- SaFT--S
Llama-2
7B
77.4
35.6
MetaMath
Mistral-vO.1
7B
77.9
28.6
DART-Matl MART-Matt-Proo2Difff
Mistral-vO.1
7B
81.1
45.5
DART-Math-Unifoom
Mistral-vO.1
7B
82.6
43.5
Mistral-vO.1
7B
82.8
48.1
DART-Mall Prop2Diff
MORT-MMth-Prop2DD
DeepSeekMath
7B
86.8
53.6
DART-Math Math-Unffom
DeepSeekMath
7B
88.2
$2.9
WizardMath- DeepSeek-SFT
DeepSeekMath
7B
88.9
58.2
MetaMath
Llama
77.3
MART--Mth-Prop2DDD
Llama
8B
81.1
DART-Math- Mnith--
Llama
8B
82.5
45.3
WizardMath- -laaa---
I.aama
8B
88.9
53.3
MetaMath
Llama 3.1
8B
80.4
35.4
DART-Math-PAE Muth-Prop2Difff
Llama 3.1
8B
84.3
46.5
DART Narh-Math- Math-Unfoo-
Llama 3.1
8B
86.7
45.1
WizardMath MaaM-Matt---aaaaaa Llama-SFT
Llama 3.1
8B
89.2
55.8
Table 16: The performance of WizardMath on the GSM8k and MATH based on the Mathstral-7B. vO. 1-Bae,, Qwen2.5-7B-Basc 1.5-5-Math--11B--aa and Owen2.5-MMth-7D
Models
Base
Params
GSM8k
MATH
Mathstral-vO.1 1-ase
77.1
56.6
WiizarMaat-MMthsste
Mathstral-vO.1 1-Baee
7B
93.8
70.9
Owwe2..5MMtt--Ban
1.5B
76.8
49.8
WirardMath-Qwen225--MM
Qwen2.5-Math-BBan
1.5B
86.7
68.6
Qwen2.5- Math--Bas
7B
91.6
55.4
WizardMath-Qwee2255-2
Qwen2.5-Math-Bass
7B
93.9
77.8
Qwen2.5 1amn2.5-Bard 5-Base
7B
85.4
49.8
WizardMath- Minee22
Qwen2.5-Bas
7B
94.0
74.5
32

---

Published as a conference paper at ICLR 2025
Table 17: The impact of applying the proposed Instruction Quality Scoring Reward Model (IRM)
and Process Supervised Reward Model (PRM) to PPO training across various SFT backbones (i.c., DART-Math, MetaMath, and Xwin-Math)
Model
Base
Params
GSM8K
MATH
MctaMath-SFT
Ilama-2
7B
66.5
19.8
MetaMath-RI.
Ilama-2
7B
75.6
25.1
MAKT-MMthh-Prpp2Dii--
Llama-2
7B
69.9
30.7
MArT--Math-Prop2Diff
Llama-2
7B
76.8
37.1
Llama-2
7B
73.8
29.5
DART-Math- Beett--nnifom--Ke
Llama-2
7B
79.1
35.2
Xwin-Math Math-SFT
Llama-2
7B
82.6
40.6
Xwin-Math RL
Llama-2
7B
88.2
48.5
WizardMath- BeaaddMat--Llaaa---.. Jam
I.ama-2
7B
77.4
35.6
WizardMath- Rllma-R..
I.ama-2
7B
84.1
43.5
MetaMath-SSP
Mistral-v0.1
7B
77.9
28.6
MetaMath-RR -RL
Mistral-v0.1
7B
86.4
35.2
SART--MathPPro22Diff
Mistral-v0.1
7B
81.1
45.5
DART-Math- DART-Mathh-roooDiif Prop2Diff-RR RL
Mistral-v0.1
7B
87.5
51.4
DART-Math-Unffon Uniform-SFT
Mistral-vO.1
7B
82.6
43.5
DART-Math-Uniicn
Mistral-v0.1
7B
88.1 1
48.7
WizardMath- Mistral-SFF
Mistral-vO.1
7B
82.8
48.1
WizardMath- MhearMMat-MMistral-RL Mistral-RL
Mistral-vO.1
7B
90.7
55.4
A.8 THE PERFORMANCE OF WIZARDMATH ON THE OTHER DIFFERENT BASE MODELS
Table supplementt the performance improvements Mathstral- 7B- 1-Base, Qwen2.5-7B-Ban Qwen2.5-Math-11 1.5BBBase,, and Dween255-Math-7B-BBB on the GSM8K and MATH datasets.
The results demonstrate that using Mathstral-7B--O.1 Base as the base model, WizardMath-Mathstral improves performance by 16.7% GSM8k (93.8 vs. 77.1) and 14.5% MATH (70.9 56.6). When employing
as the base model, SeaarddMtt--wee255--aaa achieves 9.9% improvement on GSM8K (86.7 vs. 76.8) and 18.8% on MATH (68.6 vs. 49.8). Similarly, with Owen2.5-Math- -B-aas, WizardMath- 1etthh-QQn..5aaahb.. GSM8k (93.9 vs. 91.6) and 22.4% on MATH (77.8 vs. 55.4). Finally, using Qwen2.5-7B-BB base model, SatraaddMMMh--Qeee2... MATH (74.5 vs. 49.8).
shows a 2.3% increase on
the
improves by 8.6% on GSM8k (94.0 vs. 85.4) and 24.7% on
Notably, both Mathstral-7B-vil 1-Base and Qwen2.5-MM -rren..--Math-- 5-Math--ase, pre-trained on extensive mathemati- cal corpora, exhibit robust mathematical reasoning capabilities and deliver strong performance on GSM8k and MATH datasets. However, our proposed RLEIF method achieves substantial perfor- mance enhancements even with these highly math-optimized models. Specifically, on the MATH, RLEIF delivers performance boost of 15% 25%, while GSM8k, the improvement ranges from 8% 16% (with the exception of Qwen2.5- 7B---BB- which achieves a high baseline of 91.6 on GSM8k but still benefits from 2.3% enhancement). These results underscore the continuous improvement enabled by our RLEIF method on models pre-trained with specialized mathematical corpora, further validating its effectiveness and scalability.
A.9 THE IMPACT OF APPLYING THE PROPOSED INSTRUCTION QUALITY SCORING REWARD MODEL (IRM) AND PROCESS SUPERVISED REWARD MODEL (PRM) TO PPO TRAINING ACROSS VARIOUS SFT BACKBONES
Table 17 shows the impact of applying the proposed Instruction Quality Scoring Reward Model (IRM) and Process Supervised Reward Model (PRM) to PPO training across various SET backbones The results demonstrate that incorporating IRM and PRM during PPO training led to a performance improvement of 5% to 8% on both GSM8k and MATH for most SFT models. For instance:
(i.e., DART-Math, MetaMath, and
33

---

Published as a conference paper at ICLR 2025
86
51
GSMBK(Ours)
GSMaK(MetaMath)
GSMBK(DART-Math)
80
MATH(Ours)
44
MATH(MetaMath)
MATH(DART-MaI)
74
36
GSMBK(%)
MATH(%)
68
28
62
21
56
13
15k
30k
60k
120k
200k
300k
390k
Data Size
Figure The performance of WizardMath Evol-instruct in comparison with DART-Math and MetaMath across different training data scales on the GSM8k and MATH benchmarks in the SFT stage. We use the Mistral-7B as base model
When using DART-Math as the SFT backbone based on Llama2-7B: On GSM8k, after reinforcement learning training with IRM and PRM, Prop2Diff-RL improved by 6.9% (69.9% vs. 76.8%), and Uniform-RL improved by 5.3% (73.8% vs. 79.1%).
On MATH, Prop2Diff-RL RL achieved 6.4% gain (30.7% vs. 37.1%), and Uniform-Ri improved by 5.7% (29.5% 35.2%).
When using DART-Math the SFT backbone based on Mistral-7B-v0.1: On GSM8k, Prop2Diff-Ro improved by 6.4% (81.1% vs. 87.5%), and Uniform-RL in- creased by 5.5% (82.6% vs. 88.1%).
On MATH, Prop2Diff-RI. rose by 5.9% (45.5% vs. 51.4%), and Uniform-RI. saw a 5.2% enhancement (43.5% 48.9%).
For the MetaMath models based on Llama2-7B and Mistral-7B-v0.1: Training with PPO using IRM and PRM led to performance improvements of 8% to 9% on GSM8K and 5% 8% MATH.
Similarly, for the Xwin-Math-Llama'd MATH improved by 6% to 8%.
model, performance both GSM8k and
These findings highlight the significant contributions of our IRM and PRM during reinforcement learning, consistently enhancing mathematical reasoning abilities of our SFT models while achieving robust generalization different SFT backbones. This represents key contribution of our study. Thus, our study primarily makes two core contributions:
1. The proposed Math Evol Instruct data synthesis method is also effective and practical the current state-of-the-an art data synthesis methods, such as DART-Math, Skywork- and Xwin-Math in the SFT stage. It also significantly enhances the mathematical reasoning capabilities of our models.
The proposed IRM and PRM models substantially improve performance during the reinforcement learning phase. They not only continuously enhance the mathematical reasoning abilities of
34

---

Published as a conference paper at ICLR 2025
Table 18: The performance comparison of WizardMath-SFF with DART-Math, Xwin-Math, and Skywork-Matt on the Llama2-78 base model on the MATH benchmark.
Llama2 7B as the base model
Data size
MATH
DART-Math-Unifom
591k
29.5
DART-Matt-roop2Diif
585k
30.7
Xwin-Math
1440k
40.6
Skytork-Math
360k
29.36
Skywork-Math
720k
34.54
Skywork-Math
2500k
47.7
WizardMath-SFT
418k
35.6
SFT models but also achieve strong generalization across various SFT backbones (i.e., DART-Math).
Outstanding performance is demonstrated the GSM8K and MATH.
A.10 COMPARE OUR APPROACH WITH ADVANCED METHON SFT USING SYNTHESIZED DATA, SUCH AS DARTMATH
As the volume of training data increases, WizardMath-Evol Evol-Instuuct consistently improves its per- formance on the GSM8K and MATH benchmarks, exhibiting slightly higher growth rate than DART-Math in Figure [5] In the initial stages, WizardMath slightly underperforms compared to DART-Math. This advantage may stem from DART-Math being distilled from DeepScekMath RL, an advanced mathematical reasoning model pre-trained on 120B high-quality mathematical tokens, showcasing exceptional proficiency in mathematical reasoning. However, exceeds 60k, its performance begins to surpasse DART-Math. At a data scale of 390k, WizardMath slightly outperforms DART-Math by 2%-3% on GSM8K and by 5 5%-6% on MATH. Additionally, Mizarddaat-BBvo--Inntuc consistently exceeds MetaMath at the same data scales, achieving in- creases of 3%-6% on GSM8k and 15%-20% on MATH. This performance gain is attributed to the efficiency of Math Evol- Inst-unsssst's 's upward and downward evolution processes. These findings demonstrate that our Math Instrunst method also as scalable and effective as DART-Math for the large-scale synthetic data.
the dataset
A.11
OUR MATH EVOI-INSTRUCT COMPARED TO OTHER SFT METHODS, SUCH AS DART-MATH, XWINMATH AND SKYWORK-MATH!
In Table 18 show the performance comparison of WizardMath-SFF with DART-Math Xwin- Math and Skywork-Math on the Llama2-7B base model on the MATH benchmark.
WizardMath-SFT vs. DART-Math:
WizardMath-SSI based on the Llama2-7B model, outperforms DART-Mathe Math-Moot by 6.1% and MARRT-MathPPop22DD by 4.9% on the MATH. Notably, the amount of data used WizardMath-SEF is only 70%-7192 % of DART-Math (418k vs. 591k; 418k vs. 585k).
WizardMath vs. Xwin-Math:
Although WizardMath- -ST--FFFFF is 5% lower than Xwth on the MATH, the amount of data used is only 29.0% of Xwin-Math (418k vs. 1440k), which is much less than Xwin-Math. Moreover, Xwin-Math leverages GPT-4-turbo for data synthesis. However, WizardMath-SIP outperforms Xwin-Maine on the MATH when using different backbones such as Mistral- 7B-v0.1, Llama2-13BB and Llama2-70B as shown in Table [4] For instance, in Table [4] WizardMath-SFT exceeds Xwin-Math by 4.4% (48.1% 43.7% when using the Mistral- 7B-v0.1 as the base model.
WizardMath vs. Skywork-Math: WizardMath-SFT underperforms Skywork-Math-2500 on the MATH benchmark by 12. but it uses only 16.7% of the amount of data used by Skywork-Math-22 2500k (418k vs. 2500k), which is much less than Skytork-Math. Furthermore, according to Figure 5 About Synthetic Data Size in the Skywork-Math vaper(Zeng et al., 2024). Skywork- Srywork-Math-225 Matt- 720k scores 34.54% MATH. and Skywork-Math Math-360t scores 29.36%. Therefore, WizardMath- SFT-418K 8k performs comparably to Mattork-MMth-72225 on MATH, and with the same amount of data, WizardMath-SFF outtofffoorFoooooooooommmm Skytork-Math.
35

---

Published as a conference paper at ICLR 2025
Table 19: The impact of using advanced open-source models(i.e., Llama-3.-44SBDIIssttuu) for PRM training data labeling and we use Mistral-7B-v0.1 as the base model.
Models
Al-Label
GSM8k
MATH
WizardMath-SFT
82.8
48.1
PRMM-Lama-3.11444 PRM-GPT-4
1amma-3.144SBBInntttee
85.8
51.5
GPT-4
87.2
52.7
In summary, the Math Evol Instruct data synthesis method proposed our study as effective and practical the current state-of-the-ar data synthesis methods, such DART-Math, Skywork-Math and Xwin-Math the SFT stage. It significantly enhances the mathematical reasoning capabilities of the model, marking key contribution of our work.
A.12 FEASIBILITY OF USING ADVANCED OPEN-SOURCE MODELS INSTEAD OF GPT-4 TO LABEL PRM TRAINING DATA
We realize that there is a high cost of directly distilling GPT-4 in large-scale data scenarios, which limitation of this study, Additionally, manual annotation demands mathematical expertise and entails a challenging, time-intensive, and costly process. Moreover, our evolved instructions lack correct answers, limiting compatibility with the methods employed by Math-Sha Sath-Shepherd Wang et al.. 2024a) which needs the correct answers.
To mitigate these challenges, we also explore the feasibility of leveraging advanced open-source models, such as Llama-3.1-4000 same label prompts and training settings. As shown in the Table [9 405B achieves 85.8% % on the GSM8k, outperforming WizardMath-SFTT by 3.0% and lagging behind by 1.4%. the MATH, it scores 51.5 %, exceeding WizardMath-SSS by 3.4% with 1.2% gap compared WizardMath- PRMM--aP------PP-....- Balancing cost and accuracy, Llama-3, 3.05- 1-405B--Instruc Instruct demonstrates considerable potential as a substitute for GPT-4 in PRM training data labeling.
instead of GPT-4 for PRM training data labeling, using the PRM--Maat--PM-Lammm
SeraardMath-PRM-GPP
A.13 HIGHLIGHT THE CORE CONTRIBUTIONS OUR METHOD.
We highlight the key contributions of our method as follows:
1. Unlike WizarlI.MWiizaree
which primarily focus on increasing instruction difficulty, the first to propose the novel concept of downward evolution, major distinction in instruction evolution.
In Table 6. provide detailed analysis of the effects of downward evolution. Specifically, two rounds of downward evolution led to remarkable improvement in GSM8K performance by 14.8% (74.5 vs. 59.7) and in MATH performance by 19.6% (34.7 vs. 15.1) compared the original, significantly enhancing the model's 's mathematical reasoming capabilities. This demonstrates that Math Evol-Instruct is instrumental in significantly boosting the model's mathematical reasoning ability.
2. In reinforcement learning (RL) training, we firstly propose the instruction quality scoring reward model (IRM) combined with the process supervision reward model (PRM) further enhancing WizardMath mathematical reasoning ability. As demonstrated in Table 3 our method achieves remarkable 5%-8% improvement GSM8K and MATH performance over the SFT backbone across models of various sizes, leveraging PRM and IRM for the PPO training.
We firstly propose to use AT to annotate the step-level PRM training data. Additionally, the training datasets for SFT, PRM. and IRM are fully synthesized using Al systems. This fully Al-automated data generation pipeline ensures scalability.
4. MazardMath demonstrates outstanding performance across a wide range of model scales, from 100M to 1B and 70B parameters, on the benchmarks such as GSM8k, MATH, and out-of- distribution (OOD) tasks like MWPBench/ (Tang et al., 2024). It surpasses all existing open-source state-of- the models, showcasing the effectiveness and robustness of the RLEIF approach proposed in our study.
36

---

Published as conference paper at ICLR 2025
Table 20:
The performance of MizhrrMMat--FF" on the 7 out-of-domait evaluation results covering K-12, college, and competition level math problems compared with some SOTA models (i.e., DART- Math) in the SFT stage. The results of models in the table refer to МWPBENCH (Tang et al. 2024). "AGIE" stands for AGIEval. We report the models' CoT pass@) any external python tool
results on MwpBench without using
College Math
Gaokao
AGIE
AGIE SAT Math
Models
TAL Math23k
Ape210k
Bench Math
Gaokao
AVG
Math
Proprietary models
GPT-4 GPT-3.5-Turbo
24.4
51.8
76.5
61.5
35,4
28.2
68.6
49.5
21.6
42,9
62.5
44.0
23.2
15.3
55.8
37.9
Models based on LLaMA-2
13B
1JaMA-2 13B
1.2
6.3
9.5
7.9
0.7
0,4
6.8
4.7
MAmmoTH-CoT
6.5
17.3
39.5
28.1
5.9
4,9
20.5
17.5
GAel-Abel
21.1
42.2
27.8
7.0
30.3
20.2
MetaMath
MathScale
20.4
38.1
43.7
20.0
12.3
35.9
WiarrdMth-SFFT
22.2
42.5
65.9
47.6
31.6
23.5
59.7
41.9
WizardMath-RR RL
22,9
43.3
70.3
50.8
33.1
25.7
64.7
44.4
Models based on LLaMA-2
LLaMA-2
7.6
2.1
2.9
4.6
MAmmoTH-CoT
6.2
13.3
34,6
21.4
3.9
2.7
19.6
14.5
GAIR-Abel
6.6
18.3
35.4
24.5
4.3
4.4
23.5
16.7
MetaMath
9.4
22,5
44.0
29.9
5.9
5.1
36.2
21.9
DART-Muth-Unin Unth--ufoonuo
12
27.3
47.9
32.9
14.8
11.1
45.1
27.3
DART-Math MART-MMth-Prop2D00 Prop2Diff'
11.9
27.7
34.3
12.8
10.6
47.1
27.8
Xwin-Mutb-VL.
29.7
15.9
8.4
31.5
MathScale
20.9
35.2
59.0
19.6
12.6
35.3
WizardMath SFT
21.1
38.5
62.4
43.8
26.3
17.7
58.3
38.3
WizardMath-RL
21.2
40.2
67.3
46.1
28.9
18.7
62.7
40.7
Modela based Mistral 7B
7B
17.9
6.2
13.4
MetaMath Mistral
15.7
31.4
15.3
10.1
50.9
30.9
DART-Muth-Uniie
19.4
34.8
61.6
44.8
27.0
16.1
59.8
37.6
MathScale Mistral
21.8
39,9
64.4
46.0
21.4
14.3
57.8
37.9
DART-Math-PP Mfff
19.9
37.4
62.2
44.9
27.2
18.1
62.7
38.9
WizardMath Mistral-it----
24.3
42.7
66.6
49.7
35.2
22.7
63.1
43.5
WizardMath- Mistral-RL
24.8
44.8
71.2
52.6
37.2
24.5
64.7
45.7
A.14 THE PERFORMANCE OF WIZARDMATH-SFTT ON THE OUT-OF-DOMAIN BENCHMARKS COMPARED WITH SOME SOTA MODELS (I.E., DART-MATH) IN THE SFT STAGE.
Table 20 presents the performance of WizardMath SFT on 7 out-of out-of-domman domain (OOD) evaluation tasks covering K-12, college, and competition-leve level math problems in the SFT stage. The results indicate that WitardMath-SFT consistently surpasses open-source state-of-the-. models (i.e., DART-Math, X Xwin-Math, and MathScale) across various scales and tasks, achieving average improvement of 3 -6-6% -6%. For instance:
With the Llama2-7B base model, WizardMath-SFT outperformed DART-Math Uniforr by 11.0% (38.3% vs. 27.3%) and DART-Math-Prop2Diit by 10.5% (38.3% 27.8%) on average.
With the Mistral-7B base model, WizardMath-SFF achieved an average improvement of 5.9% over (43.5% vs. 38.9%).
(43.5% 37.6%) and 4.6 % over DART-Math-Prop2Dir
These findings highlight the effectiveness of Math Evol-Instruct method, demonstrating its robustness and superior generalization capabilities on out-of -o--ain tasks.
37


Reviews:
[{"review_id": "QRJ669FbBC", "reviewer": "Reviewer_dHFe", "summary": "The paper takes Evol-Instruct (from the WizardLM paper) and extends it to the math domain, while at the same time integrating process reward models into the training pipeline.\n\n## Method\n- Generate questions of various complexities by prompting GPT to sequentially generate easier questions (\"downard evolution\") and harder questions (\"upward evolution\")\n- Train Instruction Reward Model (IRM) to predict quality of instruction\n- Train Process Reward Model (PRM) to predict quality of each individual step\n- Use PPO\n\n## Experiments\n- GSM8k and MATH datasets\n- Main base models are Llama3 and Mistral. Experiments done across various scales.\n- Outperforms a number of strong closed-source models such as GPT and Claude2", "questions": "- I find Figure 1 confusing. Why is there a pyramid in the top left and why is it pointing to a pie chart, cube, etc? What are these supposed to be showing? I feel like I am not understanding much from this figure. \n- I feel like there's a few missing entries in Table 1. For example, Table 1 shows the results for WizardMath-Mathstral and WizardMath-Qwen2.5, but the base scores of these base models are not shown in the table, so the readers don't really know how much improvement there is.\n\nTypos:\n- L45: struggles --> struggle\n- L80: \"in recent\" \n- L91: should say what is IRM first before using the acronym. \n- L91: We --> capitalization\n- L93: later --> latter\n- L105: Jiang et al mentioned without a model (should be mistral?)\n- L107: as following --> as follows\n- L140,143: spacing\n- L145: should be Reinforcement Learning for Large Language Models instead of the other way around?\n- L487: \"reasoing\"", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "**1. Strong Results** -- I put a lot of premium on this strength and use this to justify my overall rating. Many of the gains from training on Math Evol-Instruct are more than 10 points. More importantly, it is quite impressive to design something that outperforms strong proprietary models, so if this method is as strong as the paper claims, then this is something that the community will definitely quickly pick up on.\n\n**2. Thorough Experiments and Baseline Comparisons** -- Various scales ranging from 100M to 1B to 70B, with different base models. Has comparisons with several math-specific models (e.g. Mammoth, Math-Shepherd, etc.) I also liked the analysis and ablations in Section 4.4\n\n**3. Scalable Method** -- The whole process is fully automated, which makes it scalable. I imagine this can also be adapted to other process-intensive reasoning domains outside math (e.g. coding).", "weaknesses": "**1. PRM labels from GPT-4** -- Not really sure what to think of this. On one hand, I feel such direct distillation like this would limit the effectiveness of a method at larger data scales. On the other hand, the results seem to be good (and also this is one key part that makes the process fully AI-automated.)\n\n**2. Unclear presentation** -- The paper assumes that readers are already previously familiar with Evol-Instruct, as it devotes very little time to talking about it in the intro or related work. The narrative is messy -- there are certain concepts (e.g. \"grade school\" and \"high school\" questions) that were introduced once out of nowhere then never mentioned again. There are a number of rows on Table 1 that were never discussed in the main text. Figure 1 is difficult to understand. There are also several typos and poorly-worded sentences.\n\n**3. Somewhat marginal contribution** -- Evol-Instruct previously existed. PRM previously existed. This paper basically took Evol-Instruct and PRM and used them to train a model. To nitpick a bit, I think a more comprehensive paper would cover more domains such as code.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OTIKKu4TVp", "reviewer": "Reviewer_8CQg", "summary": "This paper proposes the Reinforcement Learning from Evol-Instruct Feedback (RLEIF) framework to enhance the mathematical reasoning capabilities of language models. RLEIF involves two main steps: instruction tuning with MATH EVOL-INSTRUCT and reinforcement learning using an Instruction Reward Model (IRM) and a Process Reward Model (PRM).\n\nThe experimental results demonstrate that their model, after both supervised fine-tuning (SFT) and reinforcement learning (RL), surpasses many existing math-focused language models that have undergone only SFT.\n\nThe ablation study highlights the crucial role of the IRM during the reinforcement learning phase. Additionally, they show that their PRM, which utilizes GPT-4 as a step annotator, delivers superior performance compared to Math-Shepherd and PRM800K.", "questions": "- In Equation (1), how is the parameter ( m ) set? Additionally, how is the Instruction Reward Model (IRM) trained?\n\n- Lines 256-258 suggest that you retain solutions with incorrect answers. How might this influence the results? Have you considered using the IRM to filter out low-quality examples for supervised fine-tuning (SFT)?\n\n- During PPO, do you use two reward models? Using two reward models in PPO can be time-consuming and computationally expensive. What are your strategies for addressing this?\n\n- In Lines 89-90, you state that you \"innovatively introduce PRM to address the False-Positive issue in the problem-solving process.\" This claim should be validated by comparing the false-positive rate on a test set both with and without your method.\n\n- In Lines 88-89, you mention that existing methods \"mainly focus on the SFT stage and are susceptible to learning hallucinated information from the teacher model.\" However, in Line 95, you still use GPT-4 to annotate step-level labels. Isn’t there a risk of obtaining incorrect step labels from GPT-4 as well?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "This paper is well-written and rich in detail. The introduction of the Instruction Reward Model (IRM) is novel and useful. The experiments are comprehensive, and the analysis is thorough, providing deep insights into the framework's effectiveness. I believe the idea of integrating IRM with PRM could be useful for any math LLMs (only undergoing SFT).", "weaknesses": "The primary concern with this paper is the unfair comparison of baseline models in the results. While the authors claim that both supervised fine-tuning (SFT) with Math Evol-Instruct and reinforcement learning (RL) with the Instruction Reward Model (IRM) and Process Reward Model (PRM) are beneficial for enhancing mathematical reasoning, these approaches—SFT with synthesized data and the use of various reward models for RL—represent parallel research lines.\n\n- In Table 1, the authors compare their model, which has undergone both SFT and RL, with models that have only undergone SFT. This comparison is unfair because these SFT models could also be further enhanced with RL techniques to improve mathematical reasoning (e.g., using ORM for RL on DartMath). It would be more appropriate to isolate the effects of SFT and RL for a fair comparison. In Table 1, the authors should compare the performance of models that have undergone SFT with Math Evol-Instruct against existing baselines such as MetaMath and DartMath. Additionally, comparisons with baselines like MetaMath and DartMath on the LLaMA-3.2 backbone would be valuable, as their training data is publicly available.\n\n- In analyzing the impact of training data size, the authors should compare their approach with the best method for SFT using synthesized data, specifically DartMath. MetaMath, which was developed around a year ago, uses GPT-3.5-turbo for data augmentation, making it an outdated and potentially unfair baseline.\n\n- It appears that SFT with Math Evol-Instruct yields inferior results compared to other SFT methods. From Table 4, the LLaMA2-7B: WizardMath-SFT scores 35.6 on MATH, which lags behind models like XwinMath and Skywork. Likely, it would also lag behind LLaMA2-7B fine-tuned on the DartMath training data. This suggests that the main contribution of the paper is in the RL component. Therefore, the primary focus should be on the results obtained with different reward models, as presented in Table 4, utilizing various SFT backbones.\n\n- Table 7 lacks adequate baselines; at least, the authors should include LLaMA-2-7B trained on the DartMath training set. This table also suffers from the same fairness issues as Table 1.\n\nI recommend that the authors reorganize the paper better to emphasize their contributions to the \"RL part.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ekLpZ21lAD", "reviewer": "Reviewer_8UVP", "summary": "This paper proposes a method to enhance math reasoning using Reinforced-Evol-Instruct, called WizardMath. The process begins with Supervised Fine-Tuning (SFT) on evol-instruct data, followed by training Instruction Reward Model (IRM) and Process Reward Model (PRM). Finally, they train PPO using the trained IRM and PRM. The WizardMath model achieves high performance on GMS8K and Math datasets.", "questions": "NA", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The paper is well-written, with comprehensive analysis and ablation experiments. \n\nIt represents a valuable exploration of process supervision and IRM in math reasoning, achieving impressive performance.\n\nAt the same time, their method is data efficient.", "weaknesses": "NA", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ybD0gkBjHn", "reviewer": "Reviewer_Hirr", "summary": "This paper introduces WizardMath, an innovative approach to improving mathematical reasoning capabilities in large language models (LLMs). WizardMath employs a unique reinforcement learning method called Reinforcement Learning from Evol-Instruct Feedback (RLEIF), which optimizes the model’s performance on mathematical reasoning without external tools. It specifically targets pretrained model to enhance their reasoning by leveraging evolutionary instruction (Evol-Instruct) to generate diverse math problems and using process-based supervision to ensure correctness at each reasoning step. Experiments demonstrate that WizardMath achieves higher accuracy than other leading models, even outperforming proprietary models like GPT-3.5 and Claude on GSM8k and MATH benchmarks.", "questions": "see above", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. This paper provides a scalable approach to enhance LLM reasoning capabilities in math without external computational tools.\n2. Notable improvements in metrics across diverse benchmarks.\n3. The introduction of dual reward models (IRM and PRM) effectively improves model reliability and accuracy. \n4. The synthetic data and process supervision paradigm have had a great impact on the community.", "weaknesses": "1. The training data for RLEIF is derived partly from a designated training set and synthetic from existing models, such as GPT-4. How might the reliance on specific LLMs for generating synthetic data (like GPT-4) affect the scalability of this approach for researchers without access to such models? Have the authors explored alternative methods for generating diverse mathematical problems that don't depend on proprietary models?\n2. While RLEIF has undergone extensive evaluation in the context of mathematical reasoning, it may also hold potential for applications in other reasoning domains. Have the authors considered applying RLEIF to other reasoning tasks like logical deduction or causal inference? It would be valuable to see if the benefits observed in mathematical reasoning transfer to these domains, and what modifications might be necessary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)