MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning
Authors: Xiang Yue
Xingwei Qu
Ge Zhang
Yao Fu
Wenhao Huang
Huan Sun
Yu Su
Wenhu Chen
Venue: ICLR 2024 spotlight (2024)
Keywords: Math Reasoning
Instruction Tuning
Large Language Model
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: We introduce MAmmoTH, a series of open-source large language models (LLMs) specifically tailored for general math problem-solving. The MAmmoTH models are trained on MathInstruct, our meticulously curated instruction tuning dataset. MathInstruct is compiled from 13 math datasets with intermediate rationales, six of which have rationales newly curated by us. It presents a unique hybrid of chain-of-thought (CoT) and program-of-thought (PoT) rationales, and also ensures extensive coverage of diverse fields in math. The hybrid of CoT and PoT not only unleashes the potential of tool use but also allows different thought processes for different math problems. As a result, the MAmmoTH series substantially outperform existing open-source models on nine mathematical reasoning datasets across all scales with an average accuracy gain between 16% and 32%. Remarkably, our MAmmoTH-7B model reaches 33% on MATH (a competition-level dataset), which exceeds the best open-source 7B model (WizardMath) by 23%, and the MAmmoTH-34B model achieves 44% accuracy on MATH, even surpassing GPT-4’s CoT result. Our work underscores the importance of diverse problem coverage and the use of hybrid rationales in developing superior math generalist models.

Published as a conference paper at ICLR 2024
MAMMOTH: BUILDING MATH GENERALIST MODELS THROUGH HYBRID INSTRUCTION TUNING
Xiiang Yue", Xingwei Qu, 'Ge Zhang, Yao Fu, WWnnao Huang,
"Huan Sun, Yuu Su, 'Wnhuu Chen"
"University of Waterloo, "hhe Ohio State University, *HKUST, UUiversity of Edinburgh, $01.AI
yue. 1490osu.. edu, wenhuchen@uwaterllog
ABSTRACT
We introduce MAmmOTH, series of open-source large language models (LLMs) specifically tailored for general math problem-solving The MAmmoTH models are trained Mat SetthInstruuu MathInst ruct is compiled from 13 math datasets with intermediate rationales, six of which have rationales newly curated by It presents unique hybrid of chain-of-thought (CoT) and moogaam-ff-hhoug sures extensive coverage of diverse fields math. The hybrid of CoT and PoT not only unleashes the potential of tool use but also allows different thought processes for different math problems. As result, the MAmmоTH series substantially outper- form existing open-source models on nine mathematical reasoning datasets across all scales with an average accuracy gain between 16% and 32%. Remarkably, our MAmmoTTH77B model reaches 33% MATH (a competition-level dataset), which exceeds the best open-source 7B model (WizardMath) by 23%, and the MA-moTH-34B model achieves 44% accuracy MATH, even surpassing GPT- 4's CoT result. Our work underscores the importance of diverse problem coverage and the use of hybrid rationales in developing superior math generalist models.
meticulously curated instruction tuming dataset.
(PoTT) rationales, and also en-
MathInstruct
Hybrid Instruction Tuning
Chain-of- Chain-of-Thought Thought (CoT)
Weng hour for babysitting. Yesterday, she just did 50 minures of buhysitting. How much did she eam?
Weng earms 12/60 0.2 per minute. Doing 50 mins, she earned 0.2 50 10
Program-of- Program-of-Thought (PoT)
XX
hourly 12; time, worked 50/60; earnings hourly_rate time_ workedd
MAmmoTH
printfrunddcarnings 2))
Diverse Math Problems
In-domain datasets
Out-of-domain datasets
Base B SoTA
D MAmmooTH (Ours)
Base
SoTA
MMAmmeTH (Ours)
+28
+32
+21
Accurncy (%)
++20
+11
Accuracy (%)
+24
+26
7B
13B
30B
70B
7B
13B
30B
70B
Figure 1: The superior performance of MAmmoTH, series of models instruction-tuned to solve diverse of mathematical problems using hybrid CoT and PoT rationales. MA significantly outperforms base and SoTA models on both in in-oomain and out-of-domain test sets, across all scales.
Xiing Yue and Wenhu Chen the leading authors of the paper. They contributed equally this project.

---

Published as a conference paper at ICLR 2024
INTRODUCTION
This work focuses on mathematical reasoning. critical capability of modern large language models (LILMs) (OpenAl| 2023 Anil et al. 2023). Despite the recent advances in this field, noticeable gap models like GPT-4 (OpenAT, 2023). P-LM-2 (Anil et al. 2023), and Claude 2 (Bai et al., (022) dominate popular mathemat- ical reasoning benchmarks such GSM8K (Cobbe et al.] 2021 and MATH (Hendrycks et al. 2021b), while open- source models like Llama (Touvron et al., 2023a Falcon (Penedo et al,
exists between closed-soure source and open-source L.MMs closed-soure
2023).
OPT (Zhang et al. 2022) lag behind on all benchmarks by wide margin.
Current efforts to bridge this gap are twofold: (1) Continued pre-training like Galactica (Tylor et al, 2022) and MINERVA (Lewkowyez al., 2022). which continues train an LLM math-related web data of more than T00B tokens. This approach improves a model's s general scientific reasoning capability but incurs high computation cost. (2) Dataset-specific fitisspeciiiffee--tueee fine-tuning like rejection sampling fine-tuning (RFT) (Yuan et al. 2023) and WizardMath (Luo et al. 2023), which fine-tunes LLMs using supervised data specific to certain datasets. Although such approaches improve in-domain performance, they cannot generalize to a wider range of math reasoning tasks beyond their fine- tuning data. For instance, both RFT and WizardMath can increase the accuracy on GSM8K (Cobbe) et al., 2021) by 30%++ one of their fine-tuning datasets, but hurt the accuracy on out-of-domain datasets like MMLUU-Mtth (Hendrycks et al. 2021a) or AQuA (Ling et al. (017)) by up to 10%.
In this paper, we aim to propose a lightweight yet generalizable math instruction-tuning approach to enhance the general (i.e., not limited to the fine-tuning tasks) mathematical reasoning capabilities of LLMs. Existing methods (Luo et al. 2023 Yuan et al., 2023 Taylor et al. 2022) primarily focus on Chain-of-Thought
Thought (CoT) approaches (Wei et al. [2026b] Nye et al. 2022 to solve math problems through step-by-step natural language descriptions. This approach excels in its generality to cover most math subjects but struggles with computation precision. and complex mathematical or algorith- mic reasoning procedures (e.g., solving quadratic equation roots and calculating matrix eigenvalues). In contrast, prompts in the format of code like Program-of-Thooe Thought (PoT) approaches (Chen et al.) 2022) and PAL (Madaan et al. 2022 Gao et al. (2023) utilize external tools (i.c.. Python interpreter) to greatly simplify the math solving process. This approach advocates offloading the computation process to the external Python interpreter to solve complex mathematical and algorithmic reason- ing procedures (e.g.. solving quadratic equations with sympy or calculating matrix eigenvalues with numpy). However, PoT falls short in dealing with more abstract reasoning scenarios, like common- sense reasoning, formal logic, and abstract algebra, especially when there exist no built-in APIs
To leverage the strengths of both CoT and PoT approaches, we introduce a new math hybrid instruction-tuning dataset MashInst ruct, which has two main characteristics: (1) broad cov- erage of different math fields and complexity levels, and (2) hybrid CoT & PoT rationales. MathInst is based on seven existing math rationale datasets and six newly-curated datasets to fine-tune Llama (Touvron et al., 2023a bj Rozière et al. 2023) models of different scales ranging from 7B to 70B. The resulting MAmmo TH
(see details in Table 1). We use
models (Figure 1) demonstrate unprecedented potential in serving as math generalists.
We evaluate MAmmoTH on spectrum of datasets, including in-domain (IND) test sets GSM8K (Cobbe et af., (022), MATH (Hendrycks et al.) (20215). AQuA-RAT (Ling et al.. 2017). NumGLUE (Mishra al. 2022b)- and out-of-domain (OOD) test sets SVAMP (Patel et al. 2021 SAT Zhong et al. 2023). MMLU-Math (Hendrycks et al. (021a)). Mathematics (Davies) al. 202T). and Simullq Konneel-Keziioskii et al., 2016). Compared with existing meth- ods, our models generalize better to OOD datasets and substantially improve the performance of open-source LLMs in mathematical reasoning. Notably, on the popular competition- MATH dataset (Hendrycks et al.) 2021b), our 7B model can beat WizardMath (open-source MATH SoTA) (Luo et 2023) by 3.5x (35.2% vs 10.7%), and our 34B MAmmoTH-Codes (fine-tuned on Code Llama (Rozière et al. 2023) can even beat the result of GPT-4 (using CoT).
We highlight our contributions from two perspectives: (1) From the data engineering perspective, we present MathInstruct, high-quality math instruction tuning dataset, combining vari- ety of math problems and hybrid rationales. (2) From the modeling perspective, we investigate the impact of various data sources and onput-output formats through training and evaluating over 50 different models and basclines ranging from 7B to 70B. Our models, including MArmoTH and MAmmoTH-Code r, achieve substantial accuracy gains over existing open-source models.
2

---

Published as a conference paper at ICLR 2024
Training Dataset
Type
Annotation
# Samples
Characteristics
Fields
GSMSK (Cobbe et al. 2021
CoT
Human
Grade Schol Exam
GSMSK-RE Tuan
2023
CoT
Llama
Llama + Validated
AQuA-RAT Ling et al.
CoT
Human
90K
GRE/GMAT Exam
MATH Hendrycks
202
CoT
Human
Math Competition
TheoremQA Chen et al
202.
*
CoT
GPT-4
600
GPT4 Validated
Camel-Math
(L ct al.
2025a
CoT
GPT-4
50K
GPT4 (Unvalidated)
College-Math *
CoT
GPT-4
1.8K
GPT4 (Unvalidated)
GSM8K *
PoT
GPT4
GPT4 Validated
AQuA-RAT 1
PoT
GPT4
9.7K
GPT4 Validated
MATH *
PoT
GPT4
GPT4 Validated
TheoremQA
Pol
GPT4
700
GPT4 Validated
MathQA (Amini et al. 2019
PoT
Human
25K
IQuA-RAT Subset
NumGLUE Mishra et al. 2022a
PoT
Human
13K
Lila Annotated
Mathinstruct
260K (72% COT. 28% PoT)
Table 1: Overview of our MathInstruct means with NEW rationales curated by us by prompting GPT-4. We have filtered out augmented samples that have answers inconsistent with the original dataset's annotations. Different colored squares represent different fields in mathematics: Pre-Algebra; Inter-Algebra; Algebra; Probability; NumTheory; Calculus; Geometry.
2
OUR APPROACH
Mathematical reasoning serves as a vital gauge for assessing the ability of LLMs to execute com- plex multi-hop and quantitative reasoning. Previously, this has been challenging task for neural networks, which struggle to solve even hasic addition and subtraction problems (Yang al. 2023). However, recent LLMs have considerable advancements in mathematical reasoning. Key break- throughs have been made through CoT prompting (Wei et al., 2022b) Nye et al. 2022) and PoT prompting (Chen et al., 2022; Gao et al. (023). CoT prompting encourages LLMs to solve prob- lems incrementally on scratchpad, enhancing both accuracy and explainability in mathematical reasoning. This approach contrasts with traditional methods that generate answers directly. PoT prompting, on the other hand, formulates the intermediate reasoning process as a program, executed with external tool like Python, compute the answer. This method improves robustness in solv- ing complex mathematical problems by offloading the calculations to external tools. However, most existing work (Zhou al., [223ab in PoT limited to proprictary models like GPT-4 (OpenAI, 2023) and Codex (Chen et al. 202T). The PoT potential of open-source models is yet to be seen. Our work aims optimizing LLMs' CoT and PoT reasoning capabilities through instruction tuning.
2.1 CURATING A DIVERSE AND HYBRID INSTRUCTION TUNING DATASET
Our study aims to compile a list of high-qualit, and diverse math instruction perructioo-tuning datasets, stand- ing out with two main characteristies: (1) broad coverage different mathematical fields and com- plexity levels, and (2) hybrid CoT & PoT rationales.
We aim for broad repre- sentation of math fields and complexity levels in our dataset. This ensures exposure to diverse set of mathematical knowledge, fostering versatility in our models. Based on these criteria, we narrow down our choices to a few high-quality datasets that are widely adopted and encompass different math fields and complexity levels, such as GSM8K, MATH, AQuA, Camel, and TheoremQA. Fur- thermore, notice a lack of coverage for college-level math knowledge, such abstract algebra and formal logic, in existing datasets. To rectify this, we use GPT-4 to synthesize CoT rationales for questions in TheoremQA and create question-CofT pairs through Self-Instruct (Wang al. 2023h utilizing a few seed exemplars found online.
Broad Coverage of Different Math Fields and Complexity Levels:
Hybrid CoT and PoT Rationales: Contrary to previous work (Yuan et al. 2023 Luo et al. 2023
Lee et al., 2023) Wang et al., 2023g) that focus on CoT, our dataset strategically combines both. This integration enhances the dataset's versatility, catering varying mathematical problem-solving
3

---

Published as a conference paper at ICLR 2024
approaches. However, most existing datasets provide limited program rationales, leading to im- balance between CoT and PoT rationales. To fill the gap, we utilize GPT-4 to supplement the PoT rationales for selected datasets, including MATH, AQuA. GSM8K, and TheoremQA. We further enhance the dataset quality by meticulously removing near-duplicated solutions. These GPT-4 syn- thesized programs then validated by comparing their executed results with ground truth, ensuring the high quality and reliability of the rationales. The validation subet ratio is shown in Table 6
Following these guidelines, our instruction dataset, detailed in Table 1, encompasses 260K (instruc- tion, response) pairs, covering a wide range of core mathematical fields (arithmetic, algebra, proba- bility, calculus, and geometry, etc.), including hybrid CoT and PoT rationales, and offering diversity in both language and difficulty levels. This attests to its high quality and unique characteristics.
2.2 TRAINING SETUP
We unify all the subsets in our MathInstruct to conform to the structure of Alpaca-like instruction dataset (Taori et al. 2023). This standardization ensures that the fine-tuned models can process data consistently, regardless of the original dataset formats. We choose the open-source models Llama-2 (Touvron et al.) 2023b) and Code Llama (Rozière et al. 2023) as our base models. We fine-tune these models including 7B, 13B, 34B, and 70B on Math I inntrnutttt which allows us to validate our Math I instruct at multiple scales. We fine-tune all the models with Huggingface transformers library (Wolf et al.) 2019). We use a learning rate of 2e-5 for the 7B and 13B models, and le-5 for the 34B and 70B models. We set the batch size at 128 and used cosine scheduler with a 3% warm-up period for three epochs. To efficiently train the computationally intensive 34B and 70B models, employ DeepSpeed training with ZeRO-3 stage (Rajbhandari et al. 2020).
2.3 EVALUATION SETUP
Our hybrid training enables models to solve problems using either the CoT or PoT approach. By default, the model provides the CoT solution. To switch to the PoT approach, one can add the trigger phrase "Let's write program to solve the problem" following the question.
Our preliminary evaluation reveals that PoT generally outperforms CoT, notably in open open-form ques- tions like GSM8K and MATH, as programmable solutions are better at solving complex mathemati- cal and algorithmic reasoning procedures. However, PoT struggles with abstract reasoning scenarios such as commonsense reasoning, formal logic, and abstract algebra, particularly in the absence of built-in APIs. To further combine the power of both approaches, coding strategy: The model first attempts PoT prompting. If the program is not executable, we fall back to CoT prompting. This heuristic can further enhance our model's overall performance (see more discussions in Table 3.4). We also report the performance of ceff-consistency decoding method
introduce simple hybrid de-
[Wang et al. 2023f in Table 8
3 EXPERIMENTS
3.1 EVALUATION DATASETS
We have selected diverse evaluation datasets (Table 2. encompassing variety of in-domain and out-of-domain samples across diverse fields of mathematics, to assess the models' capabilities in general mathematical reasoning.
For the in-domain datasets, we consider GSM8K (Cobbe et al. 2021I. MATH (Hendrycks al.] (021b), AQuA-RAT (Ling al.] 2017). and NumGLUF Mishraa al. 2022b). For the out-of-domain datasets, we choose SVAMP Patel et al. 202T). Mathematics (Davies et al.) 2021), SimulEq (Koncel-Kedee Kedziorski et al. 2016). SAT-Math (Zhong et all. 2023). and MMLU- Math (Hendrycks et al. 2021a). The wide selection of evaluation datasets includes math problems from elementary, high school, and college levels. Some of the datasets even include formal logic and commonsense reasoning. The choice of these datasets is to ensure comprehensive evaluation of the models' capabilities to generalize to unfamiliar situations and different math fields. The chosen evaluation datasets consist of both forn-formed questions and multi-choice questions.
4

---

Published as a conference paper at ICLR 2024
Eval Dataset
Samples
In-Domain?
Answer Form
Fields
GSM8K Cobbe et al. 2021
1319
YES
Open- Open-formed
MATH (llendrycks al.
2021b
5000
YES
Open-formed
AQuA-RAL QQA Ling ct al.
2017
254
YES
Multi-choice
NumGLUE Mishra ct 2022b
1042
YES
Open nen-formed
SVAMP Patel et al. 2021
1000
NO
Open-formed formed Open- cormforred
Mathematics (Davies et al.
2021
1000
NO
SimulEq [Koncel-
al.
2016
514
NO
Open- formef
SAT-Math (Zhong al. 2023
220
Multi-choice
MMI.U-Math (Hendrycks
2021a)
974
Multi-choice
Table 2: Comprehensive overview of our evaluation datasets, featuring variety of in-domain and out-of-domain problems across diverse fields of mathematics. Different colored squares repre- sent different fields in mathematics: Pre-Algebra; Inter-Algebra; Algebra; Probability; NumTheory; Calculus; Geometry.
3.2 BASELINES
We partition our baselines into the following four categories:
Closed-source LLMs: We consider 4 closed-source LLMs including GPT-4 (OpenA 2023
GPT-4 (Code Interpreter), PaL.M2 Unicorn (Anil al. 2023), Claude-2 (Bai et al. 2022) and Codex (Chen et al. (02T). GPT-4, PaI.M22, and Claude-2 use CoT prompting while GPT-4 (Code Interpreter) and Codex use PoT prompting.
-llama Base: For the base models, we consider Llama-1/2 1/2 TTovrro et al. 2023a
Llama-2-
Chat (Touvron et al. 2023b).
Coder Model: To compare with different coder models, choose Code-Llama Rozière et al.
2023). CodeT5+ (Wang et al. 2023i) and CodeGen (Nijkamp et al. 2023).
STEM Pre-training: We cover Galactica (Taylor al. 2022 mainly to understand the perfor-
mance of models specialized in STEM knowledge.
Instruction Tuning: We include Orca-Platypus (Mukherjee et al. 2023 Vicuna-1.5 (Zheng
2023b). Tulu (Wang et al. 2023g). Platypus-2 (Lee et al.) 2023) and Guanaco (Dettmers et al.
et al.
2023). We cover a wide spectrum of models trained with different types of datasets.
Dataset-Speeific Tuning: We include both RFT (uaan et al. 2023) and WizardMath (Luo et al. 2023). which specifically tune the models to adapt GSM8K and MATH datasets. We include them to understand their generalization.
For most baselines, choose CoT prompting maximize their performance due to their incom- petence in program generation. All the 'Code Model' use PoT prompting. For GSM8K, MATH, AQuA, and NumGLUE, will evaluate both 8-shot in-context-leaming and zero-shot setups to report the highest score. For SVAMP, Mathematics, SimulEg. SAT, and MMLU, we use 5-shot in-context-lcarning maintain consistency with prior work (Wei al. 2022b) Chen et al. 2023). Our few-shot exemplars are mostly taken from PHI"] (Zheng et al.) 2023a). For MAmmo TH and MAmmoTH-Code r, we always evaluate under 0-shot setting. For all models, sequence length of 2048 tokens for decoding. For multiple-choice questions, if the generated answer lacks an option, we map by re-prompting the model: "Please find the closest option to [generated answer]. The options are [options]".
allow maximum
3.3 MAIN RESULTS
We report our in-domain and out-of-domain results in Table and Table 4 respectively. Overall, we see that MAmmоTH and MAmmoTH-Codk
able outperform the SoTA model different scales. In general, the performance gain for OOD datasets is more significant than IND datasets. These results show us the potential our models as mathematical generalist. On several datasets. MAmmoTH--Code--A4 and MAmmoTH-70B are even surpassing closed-source LLMs.
5

---

Published as a conference paper at ICLR 2024
Model
Base
Math-SFT7
GSMSK
MATH
AQuA
NumGLUE
Avg
Closed-source Model
GPT-4
Unknown
92.0'
42.5'
72.6
74.7
70.5
GPT-4 (Code-Interpreter)
Unknown
97.0*
69.71
PaI.M22
Unknown
80.77
34.31
64.1
Claude-2
Unknown
85.2
32.51
60.9
Codex (PoT)
No
71.6'
36.8'
54.1"
ART (InstrucGPPT)
Unknown
71.0
54,2
Parameter Model
IJama-1
No
10.7'
2.9
22.6
24.7
15.5
Llama-2
No
14.6'
2.5
30.3
29.9
19.3
Galactica-6.7B
GAL
GAL-Instruct
10.2
2.2
25.6
25.8
15,9
Code-Llama (PoT)
No
25.2
13.0
24.0
26.8
22.2
AQuA-SFT
Llma--2
AQuA
11.2
3.6
35.6
12.2
15.6
Llama-1 RFT
Llama-1
GSM8K
46.5'
5.2
18.8
21.1
22,9
WizardMath
Llama-2
GSMBK+MATH
54.9*
10.7!
26.3
32.0
MAmmoTH
Llama-2
MathIastruct
53.6
31,5
44.5
61.2
47.7
MAmmoTih (immoTIl-ooo TllCoder
Code-Llama
MathInstruct
59.4
33.4
47.2
66.4
51.6
+5
+21
+12
+30
+20
13-15B Parameter Model
Llama-1
No
17.8"
3.9
26.0
24.8
18.1
Llama-2
No
28.7'
3.9*
25.1
8.8
16.6
Code-Llama (Pol)
No
36.1
16.4
28.7
29.2
27.6
CodeT5+ (PoT)
No
12.5
2.4
20.5
19.4
13.7
CodeGen+ (PoT)
No
12.7
3.4
24.5
22.5
15.7
Vicuna-1.5
Llama-2
No
28.4'
5.8
24.8
36.9
23,9
Llama-1 RFT
Llama-1
GSM8K
52.1*
16.1
24.5
24.4
Orea-Platypus
Llama-2
Platypus
38.4
3.0
18.9
35.3
23.9
Platypus
1ama-22
Platypus
25.7
2.5
33.4
42.3
25.9
WizardMath
1laa-
GSM8K+MATH
63.9'
14.0'
40.8
34,9
MAmmoTH
Llama-2
MathInstruct
62.0
34.2
51.6
68.7
54.1
MAmmo Toemooll-Codee Coder
Code-Llama
MathInstruct
64.7
36.3
46.9
53.7
+1
+20
+18
+26
+19
30-34B Parameter Model
Llama-1 Code-Llama (PoT)
No
35.6*
7.17
33.4
28.4
26.1
No
44.0
23.1
29.3
30.4
Llama-1 RFT
Llama-1
GSM8K
56.5
7.4'
18.5
24.3
26.6
Galactica-30B
GAL
GAL- FAL-Inntruct
41.7
12.7
28.7
34.7
29.4
Platypus
Llama-I
Platypus
37.8
9.3
27.9
40.5
28.8
Tulu
Llama-2
Tulu
51.0
10.8
25.5
32.6
WAmmoTH--Com
Code-Llama
Mathlastruct
72.7
43.6
54.7
71.6
60.7
+16
+21
+21
+28
+28
65-70B Parameter Model
Llama-1
No
50.9
10.6*
35.0
50.2
36.6
I.lama-2
No
56.8*
13.5*
40.9
50.4
40.4
I.lmm-2-Chat
1ama-22
No
54.9
18.6
37.0
51.6
40.5
Guanaco
Llama-2
No
59.2
4.1
45.2
53.5
40.5
WizardMath
Llama-2
GSM8K+MATH
81.6*
22.71
20.0
48.9
43.3
Platypus
Llama-2
Platypus
70.6
15.6
51.2
55.4
48.1
MAmmoTH
Llama-2
Mathinstruct
76.9
41.8
74.4
64.5
Table 3: The table compiles all the in-domain evaluation results. Results marked as t are copied from other papers, which be found on paperswitheode leaderboards. Math-SFT? means whether the model has been instruction-tuned on any math reasoning datasets. Pink numbers highlight the highest number within the corresponding scale and dataset. Note that there does not exist 30B+ version for Llama-2 or a 70B version for Code-Llama.
From Table 3]
we can observe that our main competitors for IND datasets are WizardMath (Luo
al. 2023) and Platypus (Lee al. 2023) WizardMath's training is heavily rooted in GSM8K

---

Published as a conference paper at ICLR 2024
Model
SVAMP
Mathematics
SimulEq
SAT-Math
MMLU-Math
Avg
Closed-source Model
GPT-4
97.0*
60.8
83.1
951
74.9
82.1
Codex (PoT)
85.2
68'
7B Parameter Model
Llama-1
24.5
6.2
22.7
30.6
17.7
Llama-2
34.5
6.0
5.0
26.8
29.8
20.4
Code-Llama (PoT)
49.4
21.7
3.5
28.6
26.9
26.0
Llama-1 RFT
21.1
5.1
11.0
12.5
21.7
14.3
Galactica-6.78
25.6
4.6
4.2
17.5
28.0
16.0
WizardMath
36.1
9.3
12.8
25.4
31.1
28.6
Toolformer
29.4
МAmmоTH
67.7
46.3
41.2
42.7
42.6
48.1
MAm To-Ceder
71.4
55.4
45.9
40.5
48.3
52.3
+22
+34
+33
+14
+17
13B Parameter Model
Llama-1
34.7
6.9
5.4
27.7
30.7
21.0
Llama-2
35.1
11.5
5.8
32.7
34.4
23.9
Code-Llama (PoT)
60.0
21.3
3.8
25.9
27.7
27.7
Vicuna-1.5 1.5
55.7
10
6.6
34.0
34.1
28.1
Llama-1 RFT
46.5
6.7
10.1
13.2
21.6
19.6
WizardMath
51.9
14.1
14.9
24.5
32.1
27.5
Platypus
55.4
11.4
36.8
35.5
Drea-Platypus
56.8
12.6
7.9
29.5
41.6
29.7
MAmmоTH
72.4
49.2
43.2
46.8
47.6
MAmmo TaatnoTT--Coee
73.7
61.5
47.1
48.6
48.3
55.8
A
+14
+40
+33
+12
+7
+26
30-34B
Parameter Model
Llama-1
48.8
12.8
33.4
39.0
29.0
Code-Llama (PoT)
69.1
34,5
6.8
26.8
21.6
31.7
Llama-1 RFT
55.4
7.6
12.8
20.4
37.9
26.8
Galactica-30B
41.6
11.8
13.2
37.7
37.9
28.4
Tulu
59.0
10.7
31.3
39.8
Platypus
51.7
13.8
13.6
38.6
41.0
31.7
MAmmoTH Tamereea-CCeer
84.3
65.4
51.8
60.9
53.8
63.2
+15
+31
+38
422
+13
+32
65-70B Parameter Model
Llama-1
55.3
14.2
37.4
44.1
33.2
Llama-2
63.8
20.5
14.0
51.3
47.1
39.3
1lama-2-Chirn
71.5
19.2
21.7
44.1
46.9
40.6
WizardMath
71.8
17.1
37.9
13.2
27.4
33.4
Guanaco 66.8
17.8
20.2
50.0
47.3
40.4
Platypus
51.8
26.3
21.7
55.9
52.5
41.6
MAMооTH
82.4
55.6
51.4
66.4
56.7
62.5
+11
+29
+14
+11
+4
+21
Table 4: The table compiles all the out-of-domait evaluation results. Results marked as & are copied from other papers, which can be found on paperswitheode) leaderboards.
and MATH datasets. Therefore, WizardMath's results are highly competitive on these two datasets. However, the dataset-specific training can be detrimental to OOD datasets like AQuA. In contrast, Platypus fine-tunes LLMs wide range of text and math reasoning datasets. it improves the open- source SoTA on several datasets. Similarly, MAmmo TH can achieve universal improvement across the board. A major observation is that MAmmoTH is particularly strong at solving more complex math problems MATH, where the gain of our model over WizardMath (open-source SoTA on MATH) exceed 25% different scales.

---

Published as a conference paper at ICLR 2024
60.0
48.1
47.9
39.9
42.6
41.6
41.0
Accuracy
40.0
32.8
29.6
29.8
27.0
32.0
20.4
22,9
19.9
20.0
8.6
0.0
GSM MATH
Out-of-domain
Overall
Llama-2 Base
WizardMath (GSM MATH CoT)
MAmmoTTH (Mathinstruct- CoT)
MAmmoTH (Mathlnstructt PoT)
MAmmoTH (Mathlnstruct- Hybrid)
Figure 2: Investigation of the influence of CoT PoT hybrid training on the 7B Llama-2 model. "Out-of-domain" refers to the five datasets detailed in Table 2 Key insights include: 1) The SoTA model, utilizing dataset-specific CoT fine-tuning on GSM and MATH, displays strong performance within its domains but struggles in OOD scenarios; 2) Diverse data sources in MathInst enable better math generalist model; 3) Fine-tuning on the PoT subsets generally outperforms fine- tuning the CoT subsets; 4) Hybrid training yields the best-performing model. The breakdown results on each dataset can be found in Appendix Table 9
From Table 2023). Similar to in-domain results, Platypus is able to yield gains over the baseline models univer- sally across the board, especially on the MMLU-Math dataset, which is tied with MAmmoTH-70B. is worth noting that the performance gains of our model on OOD datasets are even more significant than on in-domain datasets. This demonstrates our models' remarkable generalizability to unseen math problems. Notably, MA-MoTT--7B also boosts the CoT performance of WizardMath-78 greatly on MMLU-Math by 9%, which contains substantial number of questions beyond the subjects we covered in our training dataset.
we observe that main competitor for OOD datasets is Platypus (Lee al.
Comparison between Different Base Models. our experiments, we experimented with both Llama-2 and Code-Llama as the base models. From the two tables, we can observe that Code- Ilama consistently better than Llama-2, especially on OOD datasets. The gap between MAmmo TH and MAmmo TH- TH-Coder OOD datasets of MAmmo TH-Coder (34B) is actually higher than MAmmoTH (70B). We believe МA - Coder benefits greatly from the continuous code training of Code-Llama, which not only enhances the PoT capabilities but also improves Llama's 's general reasoning skills.
reach up to 5%. Surprisingly, the average performance
3.4 ABLATION STUDY ON DATA SOURCE
Ablation of the Data Source. In order to better understand what factors contribute to the great gain
of MAMmo TH over existing baselines, we set up a group of control experiments in Figure 2 We study
the following setups:
(1) MAmmoTH (MathInstruch CoT): This experiment aims to understand how much our curated CoT data could improve the generalization over the SoTA model WizardMath (Luo al. 2023) trained specifically on GSM + MATH. As can be seen, while sacrificing accuracy on GSM + MATH by 3%, CoT subset fine-tuning improves the overall nine-dataset accuracy from 27% 32%.
PoT): This experiment aims to understand the advantage of our PoT subset. As can be observed, our PoT subset fine-tuning can significantly improve the overall curacy from 27% to 41% %. This ablation reflects the importance of unlocking the program generation capabilities of model.
(2) MAmmoTH (MathInstrur
(3) MAmnoTH (Math Inst Hybrid): We further combine CoT and PoT as the hybrid training data achieve the best overall performance of 47.9%. This combined gain comes from two aspects:
The CoT subset helps maintain generic language-based reasoning skills to handle scenarios where PoT cannot handle well, e.g., abstract reasoning multi-choice questions in AQuA and MMLU.
The PoT subset can teach the model how to utilize Python APIs to solve complex math problems with high precision, e.g., the MATH problems requiring complex computation.
8

---

Published as a conference paper at ICLR 2024
Training Data
GSM
MATH
AQuA
NumG
SVA
Mat
Sim
SAT
MMLU
AVG
14.6
2.5
30.3
29.9
34.5
6.0
26.8
29.8
-25.3
G
56.6
9.2
24.4
32.1
65.4
20.5
12.3
27.2
25.2
30.3
M
27.1
25.5
27.8
32.1
47
49.4
10.5
26.4
27.4
30.4
A
15.3
5.8
39.7
15.5
15.3
6.3
7.2
32.7
36.6
19.4
G+M
58.1
28.2
26.0
34.7
64.8
50.1
17.1
28.6
28.4
37.3
G T
56.5
26.5
27.4
35.5
64.4
50.6
18.8
29.1
29.1
37.5
G+M+C
57,4
28.5
26.2
37.5
65.3
50.4
17.7
29.3
28.7
37.9
G+M+A
56.1
27.1
37.8
37.2
64.8
48.2
19.8
35.4
39.8
40.7
G+M+C+A
57.5
29.1
46.9
42.2
65.8
49.6
32.7
42.3
43.1
45,5
M+C+A+N
24.7
26.1
39.4
59.7
61.6
36.4
42.3
G+M+C+N
50.8
26.2
20.9
65.8
48.5
26.4
41.1
G+C+A+N
51.3
14.8
41.7
58.8
66.3
31.0
42.2
34.1
40.5
42.3
G+M+C+A+T
55,4
28.6
42.5
44,9
65.4
50.8
34.9
41.3
42.5
45.1
++M+C+A+N
56.5
28.9
38.2
63.7
64.1
47.9
40.8
38.6
44.5
47.0
G+M+C+A+N+F
53.8
27.0
38.2
60.8
65.9
50.8
41.8
42.5
42.7
47.1
G M+M+C+A+N MQA
55.7
28.8
42.5
62.1
64.6
45.9
38.9
41.3
45.0
47.2
MathInstruet
53.6
31.5
44.5
67.7
42.7
42.6
47.9
Table 5: Influence of different major subsets Instrrruutt based on Llama-2 7B. G: GSM8K, M: MATH, C: Camel, A: AQuA, N: NumGLUE, MQA: MathQA. "Existing data": the subset of MathInstruch in Table bJy excluding all the NEW rationales curated by us. We shorten Math- ematics Mat, SimulEq as Sim, NumGLUE as NumG, and SVAMP SVA save space.
We put some case studies in Appendix B to demonstrate the respective advantages of PoT and CoT solving different types of math problems. To summarize, we attribute our substantial gain to: 1) diverse data sources covering different math fields and complexity levels and 2) a hybrid of CoT & PoT instruction tuning and decoding strategy.
Influence of Major Subsets. Given the diverse sources of MathInstruch used in training MAmmOTH, important to understand how each dataset contributes to the overall performance of the model. We focus on five significant subsets: GSM8K, MATH, Camel, AQuA, and NumGLUE. We conduct an experiment gradually adding each dataset into training and compare the performance with the one fine-tuned the whole Mat Instruct. These results underscore the significant im- pact of diverse data sources on MAmmo TH performance, a core aspect making MAmmоTH a math generalist. The results also provide valuable insights for future data curation and collection efforts (e.g., we should always collect diverse data and avoid collecting only specific types of data).
To help understand the contribution of the 6 newly curated datasets shown in Table them from Math h Inst ruct, and train model on the existing data. As shown in the last two rows of Table 5) our curated data substantially improves the performance many datasets and leads to 9% overall increase, which reflects the importance of the NEWLY curated dataset.
remove
Influence of Hybrid Decoding. To demonstrate the effectiveness of the hybrid decoding method, we conduct an experiment as outlined in (susecttoon 2.3 By default, we initially attempt the PoT decoding method for given question. If fails generate an executable query, then transition to the CoT decoding method. The performance of different decoding methods (CoT, PoT, and Hybrid) shown Table 10 This hybrid decoding improves performance on every test set, showcasing that our model can effectively leverage the strengths of both CoT and PoT decoding strategies.
CONCLUSION
In this paper, we propose a novel math instruction tuning approach to activate open-source LLMs' mathematical reasoning capabilities. Through comprehensive study, we show that our models can outperform the SoTA performance at different scales by a huge margin. Our models benefit massively from: 1) the broad coverage of different math fields and complexity levels, and 2) a hybrid of CoT and PoT training. Our instruction tuning dataset contains 260K samples, which makes fine-tuning highly affordable even for academic labs. Our work paves the road for future studies activate L.MMs' core capabilities in specialized domains.

---

Published as a conference paper at ICLR 2024
REFERENCES
Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik
Yejin Choi, and Hannanch Hajishirzi. MathQA: Towards interpretable math word problem solving with operation-bant bbased formalisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Pupers), pp. 2357-2367, 2019.
doi: 10. 10.18653/11/N19-11 1245. URL https://
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. ArXiv preprint, abs/2305. 10403, 2023. URL https: //axxiv org/ abs 2305 10403
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harm- lessness from ai feedback. ArXiv preprint, abs/2212.08073, 2022. URL https://arxiv.
org/ htr//bs//ae 2212. 08073
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. ArXiv preprint, abs/2107.03374, 2021. URL https::/
arxiv.org/abss 20t.//117..00 03374
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompt- ing: Disentangling computation from reasoning for numerical reasoning tasks. ArXiv preprint, abs/2211.12588, 2022. URL httthh:tt///taatt www.iiiv.rrg/b/////eeee
12588
Wenhu Chen, Ming Yin. Max Ku, Elaine Wan, Xueguang Ma. Jianyu Xu, Tony Xia, Xinyi Wang, and Pan Lu. Theoremqa: theorem-driven question answering dataset. ArXiv preprint, abs/2305. 12524 2023. URL http:pp:///xxx..or////////
12524
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Debghani, Siddhartha Brahma, et al. Scaling instructioo--iincuueed language mod- cls. ArXiv preprint, abs/2210.11411 2022.
wwtp://.rggaabb///////// 11416
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. ArXiv preprint, abs/2110.14166 2021. URL https:///axix:
org/abs/20i 2et.:/2111..111.....
Alex Davies, Petar Veličković, Lars Buesing, Sam Blackwell, Daniel Zheng, Nenad Tomašev, Richard Tanburn, Peter Battaglia, Charles Blundell, András Juhász, et al. Advancing mathe- matics by guiding human intuition with ai. Nature, 600(7887):70-74, 2021. URL https: 021-04086-1
//www.nature.. nature............ com/ particlee/s4155
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized 1lms. ArXiv preprint, 0oo//305514314,, 2023. URL pattts://arxivv /bbs/
005.14311
Andrew Drozdov, Nathanael Schärli, Ekin Akyürek, Nathan Scales, Xinying Song, Xinyun Chen, Olivier Bousquet, and Denny Zhou. Compositional semantic parsing with large language mod- els. International Conference on Learning Representations (ICLR), 2023. URL https: 77
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang. Jamie Callan, and Graham Neubig. Pal: Frogram-aided language models. In International Conference on Machine Learning, pp. 10764-10799. PMLR, 2023. URL
ml .tw..mg..mmm...........
v202/ 3f htt223ff/aa23ffooo/ .df
Zhibin Gou, Zhihong Shao. Yeyun Gong. Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. Critic: Large language models can self-correct with tool-interective critiquing. ArXiv preprint, abs/2305.1 1738, 2023. /
11738
10

---

Published as a conference paper at ICLR 2024
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In 9th International Confer- URL Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021a.
net fert/fformm? wdt.ummmmddddkB.. jm jm 3Gm@
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solv- ing with the math dataset. In Thirty-fifth Conference Neural Information Pro- cessing Systems Datasets and Benchmarks Track (Round 2). 2021b. URL https:
wwwp::ewwneccaaaaaaaeeaeeeeeeeee www.eediiogg.....o..o........... neurips round
bet (ec3300eed0dd7773222666ll1110110
Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman. Learning to solve arithmetic word problems with verb categorization. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 523-533 2014. doi: 10. 1058
3115/v1/D14- 1058. URL https: http://///claatholooy..ooooe
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. NeurIPS, 2022.
Rik Joncel-Kedziorski Hannanch Hajishirzi, Ashish Sabharwal, Oren Etzioni, and Siena Du- Ang. Parsing algebraic word problems into equations. Transactions of the Association for Computational Linguistics, 3:585-597, 2015. doi: 20.1162/tacll URL https:
//aclanthology.coo org
Rik Koncel-Kedziorsee Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. MAWPS: A math word problem repository. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo-
gies, pp. 1152-1157, 2016. doi: 10.18653/v1/N16-111 1136. URL https:
orq BGg/N16-113
Ariel N Lee, Cole J Hunter, and Nataniel Ruiz. Platypus: Quick, cheap, and powerful refinement of llms. ArXiv preprint, 0bs/2308.07311 2023. URL http:::://arr....o.go//bbs////ee
0731
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances Neural Information Processing Systems, 35:3843-3857, 2022. /tLs:tts::/ http:::://///eeeeeeeeee wttp://a///a-----.IT--.---.
Guohao Li, Hasan Abed AI Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for" mind" exploration of large scale language model society. ArXiv preprint, abs/2303.17760 2023a. Itttpss::://atx
org/ abs / /2033 17760
Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. Making language models better reasoners with step-aware verifier. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5315- 5333, 2023b.
haattps:/accaattoooooo
org. 201//222..aa acc-loogg 291..91.. pddf
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale gener- ation: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Anmual Meeting of the Association for Computational Linguisties (Volume 1: Long Pupers), pp. 158-167, 2017. doi: (0.18653/v 1017 URL WoaLhttps: ac canthology.coo ootttholloy..rrr/117-000000
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung. Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective in- struction tuning. ICML, 2023. URL httl:attt//ppezeeeees
net 2teee//peE2--2224855555
Haipeng Luo. Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qing- wei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. ArXiv preprint, Abs/2308.09588 2023. 09583
htltptt::: /tap::://axxie org

---

Published as a conference paper at ICLR 2024
Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang. and Graham Neubig. Language mod- els of code are shot commonsense learners. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 1384-1403, 2022. h
//aclanthol wwwpchhoooyooogggg....o.o org.o/rg/22222 emn emnlp 1lppaain..... n. 90. pd.
Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tan- may Rajpurohit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, and Ashwin Kalyan. LILA: A unified benchmark for mathematical reasoning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. pp. 5807-5832, 2022a. URL https: //aclanthology.orgt 2022. emnlp-main.
Swaroop Mishra, Arindam Mitra, Neeraj Varshney, Bhavdeep Sachdeva, Peter Clark, Chitta Baral, and Ashwin Kalyan. NumGLUE: A suite of fundamental yet challenging mathematical reasoning tasks. In Proceedings of the 60th Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), pp. 3505 3523, 2022b. doi: 10. 11111/////2222.aalllaae
URL
atttps:/aace hatp:////llatthoooyoyy. wwwp://gg22....... acl acc-accllloooo. long. 246
Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. Orca: Progressive learning from complex explanation traces of gpt-4, ArXiv preprint, ab/22366.02070777777 2023.
org/abs /2306. 02707
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. International Conference on Learning Representations (ICLR), 2023. URL https:
//openreviee ew. net:pddffi==iaycJkga
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with language models. In Deep Learning for Code Workshop, 2022. httpttss: /ttp:: http://wwwww....////22222. 00114
2023. OpenAL. Gpt-4 technical report. ArXiv preprint, abs/2303.08774, 2023. URL htlhhtts://arxt org/abs/2303..0 08774
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2080- 2094, 2021, doi: 10. 18653/v1/2021 http:///22n..aall--e1 maacl-main.16.. URL 2021 naac - ma in n. 168
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon Ilm: outperforming curated corpora with web data, and web data only. ArXiv preprint, abs/2306.01 16, 2023. URL https://axxxi wwwpsiwwwwwirrrooooooooooooooooe abs 2
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. ArXiv preprint, abs/2304.03277, 2023.
03277
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory opti- mizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1-16. IEEE, 2020. URL
https: //d1. acm. http://///11.555553333.7731 3433727
Subhro Roy and Dan Roth. Solving general arithmetic word problems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1743-1752, 2015. doi: 10.186533v1/DD1--- 1202. URL
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi. Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code llama: Open foundation models for code. ArXiv preprint, abs/2308. 12950, 2023. URL https: / wwt.://oggg/bb///ee 12950
12

---

Published as a conference paper at ICLR 2024
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong. Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Févry, Jason Alan Fries, Ryan Techan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. Multitask prompted training enables zero-shot task generalization. In The Tenth International Confer- ence on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, 2022. URL
net tww/forumm 1yre/fooouum 9wrb9D0R1...
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe V Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether thhougt can solve them. ArXiv preprint, abs/2210.09261, 2022. URL
https: atttp://aaxiis org/ abs 2210.221022 09261
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang. Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model.
https:///ithui. com/ tat su-lab/ watllaabssaaafooo canfordd_apacc. 2023.
Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science. ArXiv preprint, abs/2211.09085. 2022. URL hattp:://axiv..ottt httpa//www..rrbbb//21111111 09085
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. ArXiv preprint, abs/2302.13971, 2023a. URL https:
//axx/..orr//abs/233222 13971
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foun- dation and fine-tuned chat models. ArXiv preprint, abs/2307.09288, 2023b. URL https:
//arxiv.org/aaa /00ps///237.000 09288
Boshi Wang, Xiang Deng, and Huan Sun. Iteratively prompt pre-trained language models for chain of thought. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 2714-2730. Association for Computational Linguistics, 2022a. URL https:
http:/looyy.rg/2222200 //aclanthology. orq/2022 emnlp-main. 174
Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun. Towards understanding chain-of-thought prompting: An empirical study of what matters. In Pro- ceedings of the s1 Annual Meeting of the Association for Computational Linguistics ( (Volume 1; Long Papers). pp. 2717-2739. Association for Computational Linguistics, 2023a. doi: 10. 18653/
v1/2023.acl- aal.23..cl-oon.155 URL httttt:s: / hatp::/s///llaattholo awwaathoooooooooo..ooooooooooola hwwphhhwooooo..o///o/eee wwtp:oyw.......o...oo.o 2023.
Boshi Wang. Xiang Yue, and Huan Sun. Can chatgpt defend the truth? automatic dialectical eval- uation elicits Ilms' deficiencies in reasoning, ArXiv preprint, abs/2305.13166 2023b. URL
13160
Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Re-Peng Lim. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. ArXiv preprint, abs/2305.0409) 2023c. URL httthtttt///xxiiin www.wiiv.rrrr...o. abs 22055.
04091
Peiyi Wang, Lei Li, Liang Chen, Feifan Song, Binghuai Lin, Yunbo Cao, Tianyu Liu, and Zhi- fang Sui. Making large language models better reasoners with alignment. ArXiv preprint, abs/2309.02144, 2023d. UtRt:htttss//axxiii
org hblgg//b//23/ /309 (..09.00114)
Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. Scibench: Evaluating college-leve scien- tific problem-solving abilities of large language models. ArXiv preprint, 0bs/2307.11033, 2023e. U htt.t::t////aatta org/ abs 2307.. 10635
13

---

Published as a conference paper at ICLR 2024
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. Self- consistency improves chain of thought reasoning in language models. International Conference Learning Representations (ICLR), 2023f. URL hattps://cpenreviee http:///ceeeneeeeeeeeeeaaaaaaaaa
PPL1 M..INTMMr
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Es- haan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob An- derson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravse- haj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen. Super-Naturallnssrutiion alization via declarative instructions ference on Empirical Methods in Natural Language Processing, pp. 5085-5109, 2022b. URL https: http::://aaaantthh.oom
Gener-
1600+ NLP tasks. In Proceedings of the 2022 Con-
org/ 2022. emnlp caannppmaatn main.annn...............aaam.m.m 340
Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. How far camels go? exploring the state of instruction tuning on open resources. ArXiv preprint, abs/2306.04751, 2023g. URL http::::////./ //ap:::////xiii www.ariwwwv...vvoaaaaaooeeeaaaae /tobs///3336 04751
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannanch Hajishirzi. Self-instruct: Aligning language model with self generated instructions. The 61st Annual Meeting of the Association for Computational Linguistics (ACL 2023), 2023h.
URL https: pestps://aclaaee /ataaaaaannnttoooooooooama org/ 2023 acl -oongnn 754...5... pdf
Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi DQ Bui, Junnan Li, and Steven CH Hoi. Codet5+: Open code large language models for code understanding and generation. ArXiv preprint, 0s5/2335.07922, 2023i. URL
abs 2305. 07922
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, 2022a. URLhttps:: /tsttps:/openzeeieer wtwp://wwwww..eee/.non.....m wtt.://eorrrrooeooe22ooooo.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. Chain-of-thouu prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837, 2022b. URL https:
//operevii ew. wwt.//fftff pest//ddfid=__0 ViQ.MeS0_I.....
Jerry Wei, Da Huang, Yifeng Lu, Denny Zhou, and Quoe V Le. Simple synthetic data reduces sycophancy in large language models. ArXiv preprint, abs/2308.03955, 2023. URL. https:
03958
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's transform- ers: State-of-the-ar natural language processing. ArXiv preprint, abs/1910.03771, 2019. URL
https::/arxivv org/ htpp///ss11900 03771
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in- context learning as implicit bayesian inference. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, 2022. URL https: //openreview. wwt.:tettt//// d-RJJVFCH
Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, Xu Zhao, Min-Yen Kan, Junxian He, and Qizhe Xie. Decomposition enhances reasoning via self-evaluation guided decoding. ArXiv preprint, abs/2305.00633, 2023. URL httll:ttt/:/aaaat
org/ ab//bb///23 00633
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. ArXiv preprint, abs/2304. 12244, 2023. htll//tttss///rxiv...tm http:///wwww..r///////////./.
14

---

Published as a conference paper at ICLR 2024
Zhen Yang, Ming Ding, Qingsong Lv, Zhihuan Jiang, Zehai He, Yuyi Jinfeng Bai, and Jie Tang- problel URL httpss:///axx org/ abs /bs/2309........... 03241
Gpt can solve mathematical problems without a calculator. ArXiv preprint, abs/2309.03241, 2023.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. Re- act: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR). 2023. URL
net wttp:eeeeeeepefedde.....
VIuYUL-X
Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren. CrossFit: A few-shot learning challenge for cross- task generalization NLP. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7163-7189 2021. doi: 10.18653/v 100/5///1/222111 .emnlp-main.577. 572..
URL settp::/aclanthologge hwtphhhwwwoooooooooooooooo wwtp:/yy.rrr./......ooo
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. ArXiv preprint, abs/2309. 12284, 2023. URL https:
//arxiv.erg/aaaaae..ee abs /200s/230990 12284
Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Chuanqi Tan, and Chang Zhou. Scaling relationship on learning mathematical reasoning with large language models. ArXiv preprint, abs/2308.01825, 2023. URL httptpt://aaattti org /bs//bbb////./ 01825
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, al. Opt: Open pre-trained transformer language models. ArXiv preprint, abs/2205.00068, 2022. URL
22C5. 01068
Chuanyang Zheng, Zhengying Liu, Enze Xie, Zhenguo Li, and Yu Li. Progressive- prompting improves reasoning in large language models. ArXiv preprint, abs/2304.09797, 2023a. URL https://arxx www.orgg//oooooo abs 2304.0979
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. ArXiv preprint, abs/2306.05685,, 2023b. URL /tttps://arxiv...
abs/
2306. 05685
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models. ArXiv preprint, Abs/2304.00364, 2023. URL /tttps://arxiv. http://xxivv..org/////22000
06364
Aojun Zhou, Ke Wang, Zimu Lu, Weikang Shi, Sichun Luo, Zipeng Qin, Shaoqing Lu, Anya Jia, Linqi Song, Mingjie Zhan, et al. Solving challenging math word problems using gpt-4 code interpreter with based self-verification. ArXiv preprint, hts/2308.07921, 2023a. URL https: ptttss://arxiie www.aawwivvrgg.ooeeeooooooo abs 2200233000000 07921
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. ArXiv preprint, abs/2305.1 1206, 2023b.
URL https::/arxivv org/ abs/ 2305 11206
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuur- mans, Olivier Bousquet, Quoc Le. and Ed Chi. Least-to-most prompting enables complex rea- soning in large language models. International Conference on Learning Representations (ICLR), 2023c. https:::: htttthtsss//eeeeeiiiieeeee net pttppp/dffoooow/1709090000
15

---

Published as a conference paper at ICLR 2024
A RELATED WORK
A.1 MATHEMATICAL REASONING DATASETS
Our work builds upon the existing mathematical reasoning literature. Early on, mathematical rea- soning is mostly focused on solving synthetic basic math problems like AddSub (Hosseini et al.
2 and other arithmetic reasoning datasets Kkoncel-Kedziones
et al.) 2015 Roy & Roth 2015
Patel et al. 2021). Later on, more difficult math word problem datasets (Cobbe et al.) 2001 Amini et al. 2019 Ling et al. 2017 Hendrycks et al., 2021b) have been proposed to focus on addressing realistic math word problems. NumGLUE (Mishra et al., 2022b) and LiLA (Mishra et al. 2022a) compile the existing literature to build a more diversified dataset collection. However, these datasets mostly focused on grade school math problems. To further test LLMs' limits in addressing more complex math problems, MMLU (Hendrycks et al. (021a) includes college math problems in its evaluation suite. More recently, (Chen et al.) [023] Wang et al., (023e) have proposed to tackle more challenging college-level science and math problems. Our instruction tuning dataset is built upon existing work include diversified collection of math problems from different subfields.
A.2 REASONING WITH LARGE LANGUAGE MODELS
LLMs have demonstrated great capabilities reason with the help of Chain-of Thought prompt- ing (Wei et al. 2022b; Kojima et al. 2022 Wang et al. 2023f.. Suzgun et al. (2022) have shown that CoT can already surpass human performance challenging BIG-Bench tasks. Later on,
eral other works (Drozdov et al., 2023 Zhou et al. 2023c Nye et al,, 2022 Wang et al. 2022a 2023b; Wang et al. 2023d Yu et al. (022) also propose different approaches to
2023a Li al.
utilize L.MMs to solve reasoning tasks by allowing intermediate steps. ReAct Yao et al. (2023) pro-
poses leverage external tools like search engines to enhance LLM reasoning skills. Another trend
is to enable LLMs' capabilities to use programs as thought processes like PoT (Chen et al. 2022). Some follow-up works include self-critic (Gou al., 2023), self-eval (Xie al. 2023), plan-and- solve (Wang et al. 2023c). These methods propose to enhance LLMs capabilities to solve math problems with PoT. Self-critic (Gou et al., (023) and self-eval (Xie et al., (022) both adopt self- evaluation to enhance the robustness of the generated program. Plan-and-solve (Wang et al. (023c) instead adopts more detailed planning instructions to help LLMs create high-level reasoning plan. These methods all prove to bring decent improvements over POT.
A.3 INSTRUCTION TUNING IN LANGUAGE MODELS
Instruction tuning is part of a line of work designed to "align" language models with more use- ful objectives and human preferences. The instruction tuning step is seen major step acti- vate LLMs' certain capabilities to respond to human instructions. Previously, instruction tuning is mainly focused on enhancing LLMs' general-purpose instruction following abilities. Since 2021, CrossFit Ye et al., 2021) and Naturalnnttruction (Wang et al., 2022b). FLAN (Wei et al.) 2022a)
and TO (Sanh et al.J 2022) are amongst the first wave of instruction tuning effort to understand ILMS' generalization capabilities. Later on, FLAN-v2 (Chung et al.) 2022] Longpre et al., (023) have been proposed to understand the effect of scaling up the instruction datasets to understand its impact model performance. These approaches mainly adopt cumaan-nnnotated datasets to build the instruction following dataset. More recently, multiple works (Wang et al. 2023h Xu et al., 2023 Peng et al., 2023) Zhou et al. [023bb Wang al. (023g) propose to utilize synthetic instruc- tion following data distilled from GPT-3/4 to align open-source L.Mss. The most similar effort to ours is Platypus (Lee et al. 2023) which aims to utilize domain-specialized dataset to construct small-scale instruction following dataset to enhance LLMs' reasoning capabilities.
16

---

Published as a conference paper at ICLR 2024
B CASE STUDY
We conduct a comparison between our PoT results vs. CoT results in Figure 3 Figure 4 and Figure 5] In the first example, even though PoT and CoT can both solve the problem, CoT gives a very ledious solution to derive the answer. Such solution is not only slow but also unstable. In the second and third case, we can further see the advantages of PoT over CoT by utilizing external tools and Python packages to greatly simplify the solution. [Figure shswws some types of questions (especially the formal logic question) that are not easily handled by programs. In order address these types of questions, CoT is a better choice.
What is the value (3))
PoT
CoT
First, we need to understand the given expression: GXX 3
from sympy import factorial
Step I: Simplity the expression inside the parentheses. 31 3 x2 x1 6
value factooial(factoriia((3) factorial(3) print(value)
So, expression becomes:
Step 2: Recall the property of factorials that states (nt))* = n! for any positive integer & and any positive integer n k. our case, we have and
120
Using this property. we can rewrite the expression as:
3: Simplify expression further by applying the property mentioned in Step 6l = 6 655 x4 x3 x2 x1 720
So, expression becomes:
Step 4: Divide 720 by 6. 720 + 6 120
So, simplified expression is:
Therefore, the value of [3]] is 120
The answer is: 120
Figure 3: Example 1: PoT and CoT can both solve the problem, however, CoT gives a very tedious solution derive the answer.
Determine the number of ways arrange the letters of the word TEPEE.
PoT
CoT
from sympy import factorial
First count the arrangements all the letters are unique, which 4!. Then since the E's are not unique, we divide by 2! for the arrangements of the two F's, for answer of 60
word "TEEPE" a word)
d wvonntcoun((""
ways factorial print(ways)
The answer is: 60
>>> 20
X
Figure 4: Example 2: PoT generates the correct solution while CoT fails.
17

---

Published as a conference paper at ICLR 2024
Question: is chosen from the set (1,2,3,4), and b is chosen from the (5,6,7,8,9). What is probability that the product a prime number?
PoT
CoT
from sympy import isprime
First, we need to Eind the total number of possible combinations of choosing number and 5 numbers the sccond set. So, there are 5 20 possiblc combinations.
define (1,2,3,4]
numbers
b_set (5.6.788,9)
lenía
Now, let's consider the prime numbers. A prime number is number greater than that has no divisors other than and itself. We need find the pairs (a, bl where cither a or b prime number. or both.
n_b-lenth scl)
prime count
prime count
prime a_couut prime count
I. is prime number. there are prime numbers in first set: 5, and So, there possible combinations where number.
prime count prime
iteate over all possible pairs for for in b set:
If prime number, there 5 prime numbers in the second
3, 7, 11. and 13, So, there are possible combinations where b prime number.
isprime(a) isprimetb): if (bppime(="b):
3. both and are prime numbers, we need to find the pairs where the prime numbers 2, 3, 5, There pairs this (2. 3), (2, 5), (2, 7) and (3, 5).
increment the counters prime count prime count
Now, let's add up the number of combinations where either or is prime number:
prime)
prime) (both prime)-
chock if prime if (epriim((a:: prime a count if (sprime(b)):
there 13 combinations where Since there are 20 possible combinations total, the probability that the product ab is prime number is
prime number.
15
20
prime count probsbility prime ab count (i_ b_b) printprobabilitt)
Therefore, the probability is
The answer is:
20
0.1
X
Figure 5: Example PoT generates the correct solution while CoT fails.
if is such & then (2 16)/ 4.f7 4) satisfies. Answer Choices: (A) 20
54 (8)
58
28 60 (E) None of these
The expression in the numerator can be rewritten as (n 2nm + 4) and so the expression is equal to n + 2./ + 4. We substitute the smallest value of which is 36, and get 42. We substitute the largest value nf,. which is 72, and get 88, So the answer is (E) None of these
Co'T
Which following could
the sides isosceles triangle?
Choices: 2.2.3 (B) 2,2,4 (C) 4.12,12 5,5,5
This is because in an isosceles triangle, the lengths of two sides are equal, and the length of the third side must be less than the sum of the lengths of the other two sides. In this case, 4 is equal to the sum of the other two sides (2+2), not less. Therefore, answer (B) 2,2,4.
CoT
Select the best translation into predicate logic. Some CEO is weulthier thun everyone. (Cx: CEO; Px: is person; Wxy: wealthier than
Answer Choices: (A) (3x)[Cx (Py) Wxy)] (B) (ax) (My)(Py Wxy]]
(C) [Cx)
(Py))Py Wxy)] (D) (vx) = (3y) = Wxy)l
3x: There exists some Cx: is CEO, and
Vy: For all Py: y is person, and Wxy: x is wealthier than So, in plain English, this expression says: "There exists some such that x a CEO all y, y is person x is wealthier than y." Which matches the original statement. Therefore, the answer (A) (3x)[Cx (Py) Wxy)]
CoT
Figure 6: Example 4: Some types of questions (e.g., formal logic) are hard to be solved by PoT but could be handled by COT.
18

---

Published as a conference paper at ICLR 2024
C KEY INSIGHTS
We show the results in Table 5
And our key findings are summarized as follows:
When the model is trained individual datasets (GSM8K, MATH, AQuA), demonstrates strong performance in tasks closely related to the respective training dataset. However, it generally struggles to generalize this performance other tasks.
The datasets GSM8K, MATH, and AQuA emerge as crucial components in the training process. Training the model on combination of these datasets leads to marked improvement in its overall performance, indicative of the complementary nature these datasets. Omitting any one of these datasets results in noticeable decline in overall performance.
TheoremQA, despite its smaller scale, exerts to enhance the model's 's capabilities on datasets like Mathematics and SimulEq, yet it may slightly impede performance on GSM8K and MATH. Nevertheless, its inclusion in the training regimen is favored, as it introduces broader variety of questions and could potentially aid the model in generalizing to other complex problems that require advanced computational skills.
influence the model's performance. It tends
Incorporating a wider range of datasets into the training consistently improves the model's perfor- mance across most tasks. This suggests that broad and diverse knowledge base is instrumental enhancing the model's ability to generalize.
19

---

Published as a conference paper at ICLR 2024
D ADDITIONAL RESULTS
We added some additional results here. The results provides how much of GPT-4 generated data are being filtered (see Table 6). The results also include the generalization benefits of MathInstruct only training on GSM (see Table 7). We also include different self-consistency setup to show how far our method can reach with ensemble decoding methods.
GPT-4 Rationales Validated Ratio
AQuA-PoT
75% (9772 /13000)
MATH-PoT
51% (7088 14000)
TheoremQA-Pe
44% (703 1600)
TheoremQA-Co ToT
37% (592/ 1600)
GSM8K PoT
81% (14591 / 18000)
Table The validated ratio of GPT-4 distilled rationales newly curated subsets in Table
GSM
MATH
AQUA
NumG
SVA
Mat
Sim
SAT
MMLU
AVG
CodeL 7B GSM8K
63.2
18.4
25.4
32,9
67.6
40.1
16.3
26.4
26.1 48.3
35.2
CodeLlama MathInstruct
59.4
33.4
47.2
66.4
71.4
55.4
45,9
40.5
52.0
Table 7: CodeLLama 7B trained on GSMSK and MathTnstt ruct. Training exclusively on a single dataset like GSM8K leads to challenges in generalizing to other datasets
Decoding Method
GSM
MATH
1 PoT
58.8
32.1
PoT + 1 CoT backup
59.4
33.4
5 PoT + CoT seff-consistency
66.4
37.0
10 PoT self-consistency
67.0
37.8
10
CoT self-consistency
69.6
38.2
20 PoT self-consistency
70.1
38.6
Table 8: Exploration of different decoding methods. Though the self-consistency Wang al. (2023)) decoding can further improve the performance, introduces significantly larger inference. Our chosen strategy, while straightforward, strikes an optimal balance between complexity, performance, and practical applicability.
20

---

Published as a conference paper at ICLR 2024
Model
GSM
MATH
AQuA
NumG
SVA
Mat
Sim
SAT
MMLU
Overall
Base
14.6
2.5
30.3
29.9
34.5
5.0
26.8
29.8
19.9
WizzardMath
54.9
10.7
26.3
36.1
12.8
25.4
31.1
27.0
MA (MathInstruct- CuT)
49.2
9.9
42.2
37.1
48.5
17.3
34.1
39.8
MAmmoTE TH (MathInstructt PoT)
50.8
28.9
28.6
52.7
65.0
42.0
28.3
41.0
MAmmoTH (MathTnstruct)
53.6
31.5
44.5
61.2
67.7
46.3
41.2
42.7
42.6
47.9
Table 9: Breakdown results of Figure on the 7B Llama-2 model.
Investigation of the influence of CoT & PoT hybrid training
Model
Decoding
GSM
MATH
AQuA
NumG
SVA
Mat
Sim
SAT
MMLU
AVG
CoT
50.5
10.4
47.3
9.2
18.9
39,9
33,0
MAmmoTI1-BB
PoT
51.6
28.7
43.3
$2.3
65.1
41.9
48.2
39.1
44.6
46.1
Hybrid
53.6
31.5
44.5
61.2
67.7
46.3
41.2
42.7
42.6
47.9
CoT
22.4
7.9
36.2
36.0
37.0
8.2
7.2
32.7
34.6
24.7
MAmmoTH-Coder::
PoT
58.8
32.1
47.2
57.1
71.1
53.9
44.6
40.0
47.8
50.3
Hybrid
59.4
33.4
47.2
66.4
71.4
55.4
45,9
40.5
48.3
52.0
CoT
56.3
12.9
45.3
45.6
53.8
11.7
22.4
43.6
42.3
37.1
MAmmoTH- 13B
PoT
61.3
32.6
48.8
59.6
72.2
48.5
40.3
46.8
45.4
50.6
Hybrid
62.0
34,2
51.6
68.7
72.4
49.2
43.2
46.8
47.6
52.9
CoT
32.1
10.2
40.6
36.2
43.0
9.6
10.1
40.9
36.6
28.8
H-mmoTTHCCodert 13B
PoT
64.3
35.2
46.8
54.2
73.2
60.0
44.2
48.2
48.2
52.7
Hybrid
64.7
48.3
54.9
CoT
34.3
39.0
36.2
44.6
10.8
10.9
46.4
42.9
30.7
MAmmoTH-CCa HomderH--Codrr-33B
PaT
72.3
42.8
50.6
$2.7
Hybrid
72.7
43.6
54.7
71.6
84.3
65.4
51.8
60.9
53.8
62.1
CoT
72.4
21.1
58.9
71.6
20.0
31.9
$2.1
49,2
MAmmoTTS TH-70B
PoT
76.7
40.1
60.2
81.7
55.3
45.3
64.1
53.5
60.1
Hybrid
76.9
41.8
65.0
74.4
82.4
55.6
51.4
66.4
56.7
63.4
Table 10: Influence of different decoding methods each dataset.
21

---

Published as a conference paper at ICLR 2024
E
LIMITATIONS
Despite their training on a diverse set of mathematical rationale datasets, the MAmmo TH models might exhibit limitations when faced with problems outside their primary domain of expertise like mathematical analysis, complex analysis, graph theory, numerical analysis, etc. Thus, our models are not suitable for solving more complex problems in these fields. Also, they have not been trained with proof-type problems, thus their theorem proving capability is also limited. In the future, we would like to expand the models' skill to cover more fields and theorem provving problems.
There is also risk of the MAMmOTH models generating potentially harmful, offensive, or biased content, especially they are asked to answer questions beyond math. The MAmmoTH series could be misused for malicious purposes, such as spreading misinformation or probing sensitive topics. Developers should conduct safety testing and tuning tailored to their specific applications before deploying any МAmmоTH model. While we have made every effort to ensure the cleanliness and purity of our training data, we cannot guarantee absolute perfection. It is unlikely but not impossible that some inappropriate questions slipped through the curation process.
Future work may continue to explore how to further improve the robustness and generalizability of MAmmOTH in mathematical reasoning. For example, recent work identifies "sycophancy" and "Clever Hans effect" in reasoning: LLMs cannot maintain truthful solutions to reasoning tasks when challenged by the user's absurdly invalid arguments and critiques (Wang et al. 2023b). Potential methods to improve the models' reasoning robustness could involve the exploration of synthetic data intervention methods as explored in
(Wei et al. 2023).
22


Reviews:
[{"review_id": "5zaTPp4deQ", "reviewer": "Reviewer_Dja9", "summary": "This paper proposed a MathInstruct dataset to improve the general performance of all types of math problems.\nSpecifically, they annotate public datasets with CoT and program-of-thought (PoT) annotations with GPT-4.\nThrough experiments on many in-domain and out-of-domain datasets, they demonstrate the effectiveness of this dataset.", "questions": "1. GSM8K has about 7k training data, how do you get 14k examples?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Expensive efforts in creating the dataset, very valuable if provided to the research community. \n2. Performance improvements over the baseline approaches without the MathInstruct dataset", "weaknesses": "1. I actually train a 7B-CodeLLaMA with GSM8K0PoT training set prompted from GPT-3.5-turbo myself, the performance can definitely achieve 62% (>59.4 in Table 3). I'm not sure what would be the quality of this dataset. I don't mind sharing my GSM8k training set for the authors to reproduce. \n2. In other words, I think the author should have more experiments to justify the datasets. For example, GSM8K PoT, we need to compare with a CodeLLaMA that trained on GSM8K PoT training set.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Mk4NEl8Gjr", "reviewer": "Reviewer_z3BX", "summary": "This paper describes MathInstruct, a mathematical reasoning-focused instruction tuning dataset assembled from 13 constituent datasets, 6 of which the authors supplement with curated synthetic rationales from GPT4. The authors use MathInstruct to train MAmmoTH, a series of Llama-based fine-tuned language models for general math reasoning. The MathInstruct dataset contains a balance of Chain-of-Thought and Program-of-Thought rationales, allowing MAmmoTH models to predict intermediate outputs in either format. The authors exploit this ability by defaulting to Program-of-Thought prediction, then backing off to CoT prompting in the event a predicted PoT program fails to execute.\n\nThe authors conduct a comprehensive evaluation of a number of open-source language models on multiple datasets, both in-domain (held out splits of datasets appearing in their MathInstruct corpus) and out-of-domain. The proposed MAmmoTH models outperform the vast majority of open LLMs, and even manage to outperform most closed LLMs (aside from GPT4) on the MATH and AQuA datasets.", "questions": "1) You mention that PoT is \"activated\" by using a particular prompt trigger. Does this mean the model won't generate PoT rationales without specifying this trigger, or will it generate a mix of both by default? (Insight on this question could address part of my second bullet in the Weaknesses section)\n2) Connected to the question above, it would be nice to add an overall ratio of PoT/CoT to Table 1. Summing and dividing the listed sizes gives 27% PoT, which explain the need for the PoT trigger prompt, but I couldn't find this number in the paper - did I miss it somewhere?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- Including both CoT and PoT intermediate reasoning in MathInstruct is a great move, as the strengths and weaknesses of the two techniques are largely complementary; the hybrid inference strategy exploiting both modes is both intuitive and effective, which is very satisfying.\n- Focusing on the diversity of the corpus seems to have paid off, as Table 5 shows that MathInstruct fine-tuning outperforms the sum of its largest parts on the more challenging out-of-domain test sets like the SAT questions.\n- It's always nice to see performance gaps between proprietary and open models closing.", "weaknesses": "- It would be useful to have numbers for some of the closed models (that have APIs) for NumGlue, Mathematics, SimulEq and MMLU-Math. I realize this is a slightly annoying request as it incurs monetary costs and the scientific value of comparing to systems with unknown training distributions and architectures is dubious, but knowing which of these datasets are the most challenging for widely-used \"flagship\" models still has value as a heuristic for contextualizing the contribution.\n- The authors only experiment with one approach to hybrid prompting - it seems like a number of approaches could be viable, e.g. self-consistency across samples from both CoT+PoT, or letting the model pick which mode to operate in. If other approaches were tried but weren't effective, it would be good to see results (or at least a remark) indicating what was tried and justifying the chosen approach as the best one empirically.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v6V6a4OwzK", "reviewer": "Reviewer_tS94", "summary": "The paper proposed a newly curated instruction-tunning dataset for tunning open-source models for general math problems. The MathInstruct dataset contains 13 datasets and intermediate steps to arrive at the solution. They explored a hybrid of CoT and PoT rationales. As a result, the open-source models tunned by MathInstruct beat the current best open-source models finetuned for math problems as well as GPT-4 on a portion of the math datasets.", "questions": "1. Is there a baseline that just finetunes all of the 13 datasets into 1 model? I think this would give a better idea of how to access MathInstruct. If not that's fine.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper demonstrated that hybrid rationales sourced from both Chain-of-Thought reasoning steps and Program-of-Thought coding capability achieved better performance than using just CoT or just PoT for math problems. \n2. Comprehensive results that compare models that are 1. differently sized and 2. differently instruction-tuned. This can be a very useful resource for anyone interested in studying the math reasoning capabilities of LLMs.\n3. OOD scenarios are also considered and shown.", "weaknesses": "1. It is known that doing SFT on the dataset, especially with intermediate reasoning can do better. So even though MathInsutruct is already better than its competitors at generalizing, it can still remain a problem for even harder or unseen math problems.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HKuz1bsZQD", "reviewer": "Reviewer_x7c7", "summary": "This paper presents MAmmoTH, a set of publicly available large language models specifically created for solving a wide range of mathematical problems. These models undergo training using a meticulously assembled dataset called MathInstruct, which is compiled from 13 different math datasets and includes detailed intermediate reasoning steps. MathInstruct is unique in that it incorporates a combination of two reasoning methods, chain-of-thought (CoT) and program-of-thought (PoT), spanning diverse mathematical domains. The use of both CoT and PoT enables these models to employ distinct problem-solving approaches for different types of math challenges. This study underscores the importance of embracing diverse problem types and employing hybrid reasoning techniques to enhance the development of highly capable mathematical generalist models.", "questions": "1. How much of the training data would be filtered after the validation? It would be good to know the data utilization rate for the data generation process.\n\n2. I'm curious about whether the generated annotation converted from TheoremQA can be successfully executed, because the problems there would require many advanced calculation like integral and derivative computation.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This paper provides a very simple method to synthesize useful annotations to equip small LLMs with maths reasoning ability. The authors also conduct fairly comprehensive comparison against many existing maths models with a wide range of model sizes, regardless of in-domain/out-of-domain tasks. The ablation studies help us to better understand the influence of the subparts in the training annotations.", "weaknesses": "Despite the commendable performance exhibited by MAmmoTH, several notable weaknesses should be acknowledged:\n\n**Limited Technical Novelty**: The approach employed in developing MAmmoTH bears resemblance to previous works, such as Orca (Mukherjee et al., 2023). In situations where an adequate reservoir of mathematical Chain-of-Thought (CoT) or Program-of-Thought (PoT) data is unavailable, the model resorts to generating content from scratch. While this approach serves as a valuable resource and addresses the scarcity of specialized mathematics models, it may not be considered a groundbreaking contribution without a more rigorous evaluation and innovative techniques.\n\n**Absence of an In-Depth Analysis of Training Data Distribution**: It is evident that each dataset contributes unevenly to the final training annotations. For instance, TheoremQA contains a mere 600 samples, raising questions about the true impact of incorporating such a small dataset. While it is reasonable to assume an enhancement in TheoremQA performance, it remains unclear whether this addition adversely affects the performance of other in-domain and out-of-domain tasks. Additionally, there is a concern that the model's focus on other mathematics datasets may hinder its proficiency in learning TheoremQA.\n\n**Insufficient Ablation Studies on Training Dataset Influence**: The majority of ablation studies appear to be centered around GSM8K, with the authors incrementally augmenting the training data size atop GSM8K. However, there is a notable absence of alternative strategies, such as initially training the model on MATH and subsequently introducing other datasets in a stepwise manner. Another informative ablation study could involve the removal of individual training datasets, such as M + C + A + N, and G + M + C + N, to elucidate the specific impact of each dataset on model performance.\n\n**Lack of Comprehensive Error Analysis**: While MAmmoTH has demonstrated impressive performance, a detailed error analysis would provide valuable insights into areas where the model can further improve. The inclusion of illustrative examples could enhance our understanding of the model's strengths and limitations, aiding in the refinement of its capabilities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fJdvN6FLMC", "reviewer": "Reviewer_JzMS", "summary": "This paper proposes MMmmoTH, which consists of two main contributions. First, it combines many different math datasets together, to the MathInstruct dataset with 260k data samples. Secondly, it uses GPT-4 to geenerate hybrid CoT and PoT solutions for the problems in the dataset. After fine tuning Llama with math instruct, their method can achieve much better results compared with the existing methods.", "questions": "I think the authors did not provide enough details about how CoT and PoT are mixed together. It seems that both CoT and PoT are simply treated as natural languages, and feed into the model for fine tuning? Are there any special tokens used? It seems that you only used \"let's write a program to solve the problem\" as the prompt, right?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "I think the main strength of this paper is it provides better results on many math datasets, including GSM8K and Math.\n\nOriginality: the main originality of this paper is combining multiple math dataset together, and use hybird CoT and PoT as the solution for the problems in the dataset. However, the use of CoT and PoT is a common idea used in many math papers.\n\nQuality: Good. This paper provides a clear pipeline of the algorithm, with detailed comparison with other methods.\n\nClarity: Good, it is easy to follow.\n\nSignificance: Mild, as stated below.", "weaknesses": "This paper has limited novelty, because it seems that it mainly combines all the math datasets together, and use GPT-4 to label the dataset, and then fine tune Llama with the new labels. This is a fairly standard pipeline, and it seems that the main improvement comes from the intelligence of GPT-4. \n\nMoreover, the idea of hybird instruction tuning is a bit confusing. According to Sec 2.4, the authors will first run PoT, and if the program cannot execute, they will switch to CoT. It seems to be a very preliminary way to combing CoT and PoT together. I was thinking a better way could be interleaving CoT and PoT in the solution. \n\nThe fine tuning part of LLama is kind of straightforward, and there are many existing work using similar ideas. So I will not say it is an important contribution.\n\nOverall speaking, I think the main contribution of this paper is \"using GPT-4 to create a new dataset (which is a combination of many existing datasets), and fine tune Llama using the created dataset\". Therefore, I will say this paper has limited significance. I give weak accept mainly because I feel this is an important problem, and the authors provide a reasonably good solution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)