MAmmoTH2: Scaling Instructions from the Web
Authors: Xiang Yue
Tianyu Zheng
Ge Zhang
Wenhu Chen
Venue: NeurIPS 2024 poster (2024)
Keywords: large language models
instruction tuning
reasoning
Primary area: natural_language_processing
TL;DR: We introduce a scalable approach to harvest 10M high-quality instruction data from web corpus for fine-tuning language models, significantly boosting their reasoning performance without costly human annotation or GPT-4 distillation.

Abstract: Instruction tuning improves the reasoning abilities of large language models (LLMs), with data quality and scalability being the crucial factors. Most instruction tuning data come from human crowd-sourcing or GPT-4 distillation. We propose a paradigm to efficiently harvest 10 million naturally existing instruction data from the pre-training web corpus to enhance LLM reasoning. Our approach involves (1) recalling relevant documents, (2) extracting instruction-response pairs, and (3) refining the extracted pairs using open-source LLMs. Fine-tuning base LLMs on this dataset, we build MAmmoTH2 models, which significantly boost performance on reasoning benchmarks. Notably, MAmmoTH2-7B’s (Mistral) performance increases from 11% to 36.7% on MATH and from 36% to 68.4% on GSM8K without training on any in-domain data. Further training MAmmoTH2 on public instruction tuning datasets yields MAmmoTH2-Plus, achieving state-of-the-art performance on several reasoning and chatbot benchmarks. Our work demonstrates how to harvest large-scale, high-quality instruction data without costly human annotation or GPT-4 distillation, providing a new paradigm for building better instruction tuning data.

MAmmoTH2: Scaling Instructions from the Web
Xiang Yue", Tuney Zheng', Ge Zhang', Wenhu Chen'
Careegi Mellon University, University of Waterloo xyue20andrew http:/aaanrew..m..c.mm cmu, edu
venluchen@@uraterrlo..cc
https://tiger-at ai-lab. cithub.io/MAnnnn
Abstract
Instruction tuning improves the reasoning abilities of large language models (LLMs), with data quality and scalability being the crucial factors. Most instruction tniing data come from human crowd-sourcing GPT-4 distillation. We propose paradigm to efficiently harvest 10 million naturally existing instruction data from the pre-training web corpus enhance L.M reasoning. Our approach involves (1) recalling relevant documents, (2) extracting instruction-response pairs, and (3) refining the extracted pairs using open-source LLMs. Fine-tuning base LLMs on this dataset, we build MHmmoTH2 models, which significantly boost performance on reasoning benchmarks. Notably, MAm TH2-7B's (Mistral) performance increases from 1% to 36.7% % on MATH and from 36% to 68 4% on GSM8K without training on any in-domain data. Further training MAmmoTH2 on public instruction tuning datasets yields TAm achieving state-of-the-as performance on several reasoning and chathot benchmarks. Our work demonstrates how harvest large- scale, high-quality instruction data without costly human annotation or GPT-4 distillation, providing new paradigm for building better instruction tuning data.
lnttral-ex7-Innstruut MAmmoTH22 TH2-78-Plus (Ours)
Qwen- 15--100
Mixtral 8x7B-Inntruce Instruct
HA4mmoTH2-8x78 (Ours)
MAAmmoTH2-xx76.o
85
68.7
80
73
74
60
63
33.8
32.6
23
35
34
32
36
20
TheoremQA
MATH
GSMBK
GPQA
MMLU-STEM
BBH
MBPP
AlpacaEval2 ArenaHard
Reasoning Benchmarks
Additional Benchmarks
Figure 1: Overview of MAmmo TH2-Plus results. The MHmmoTH2-8x2 111e--xxB-PPPPe variant outperforms Mixtral-Instruct on reasoning benchmarks, matching Owen-1.5-110B with only 13B active parameters. It also surpasses Hiixtral-Intrree by around 10 points on general code and chatbot benchmarks.
Introduction
Reasoning is fundamental aspect of human cognition and mroblem-solving Clark et al. 2018
Hendrycks et al. 2021a Cobbe al. 2021 Rein al., 2023 Yue al., 2023a Profccieccy in
38th Conference on Neural Information Processing Systems (NcurIPS 2024).
All of the authors contributors the project.

---

Previous Methods
Our Method
Recall
Extract
Refine
Anmotators
Annotated
Seed
Synthetic
Web
Raw Doc
Extracted QA
Refined QA
Human annotation
Distillation from GPT-4
WebInstruct: 10M instruction data from the web
Costly, usually small-scale
Prone to bies hallucinations
Diverse, high quality and large-scale
Figure 2: Comparison between our dataset curation method and previous studies.
reasoning is essential for advancing scientific knowledge, developing new technologies, and making informed decisions in various contexts. Recently, large language models (LLMs) [Brown et al. 20200
Ouyang et al. 2022 Touvron et al. [023a]bb b Achiam et al.) 2023 Team et al. 2023] have shown
remarkable progress in various NLP tasks. However, their ability to perform complex reasoning
tasks [in et al. 2024 in the domains of mathematics, science, and engineering is still limited.
Recent studies have extensively explored how to enhance base LLMs' reasoning abilities. The two main approaches are continued training and instruction tuning. Continued training trains LLMs on large-scale filtered documents [Lewkowyez et al. 2022 Taylor et al., [022 Azerbayev et al.) 2023 Shao et al.) [024] Ying et al. 2024]. Instruction tuning seeks employ supervised fine-tuning loss on, usually small-scale, high-quality instruction-response pairs [Ouyang et al.] 2022, |022|Chung Chung et al. 2024
human-annotated annotated instruction datasets [Cobbe et al. [202] Hendrycks et al. 202 Amini While human-
cct al. 2019) are often limited in scale, recent studies (Yu et al. 2023] Yue et al. [023b Toshniwal
al. 2024 Li ct al. 2024a Tang et al. [024] attempt to prompt GPT-4 with seed data to increase the scalability. However, the synthesized instruction data becomes highly biased, not diverse, and
prone to high degree of hallucination.
To address these limitations, we propose to discover naturally existing instruction data from the web (Figure 2). We argue that the pre-training corpus (e.g.. Common Crawl) already contains vast amount of high-quality instruction data for LLM reasoning. For example, the web corpus contains a large amount of educational materials in the form of instruction-followinn pairs. These documents range across various domains like math, science, engincering, and humanitics. Such readily available instruction data is not only diverse but also of high quality. However, such instruction data is highly dispersed across the corpus, which makes it particularly challenging to discover.
In this paper, we aim to mine these instruction-respone (1) Recall step: We create diverse seed dataset by crawling several quiz websites. We use this seed data to train a fastText model [Joulin et al.. [016] and employ it to recall documents from Common Crawl [Computer] 2023). GPT-4 is used to trim down the recalled documents by their root URL. We obtain 18M documents through this step. (2) Extract step: We utilize open-source LLMs like Mixtral (Dingg et al. 2024) to extract Q-A pairs from these documents, producing roughly 5M candidate Q-A pairs. (3) Refine step: After extraction, we further employ Mixtral-8 8x 7B [Jiang et al., 2024) and Qwen-72B (Bai et al. 2023) to refine [Zheng et al. 2024b) these candidate Q-A pairs. This refinement operation aims to remove unrelated content, fix formality, and add missing explanations to the candidate Q-A pairs. This refinement operation is pivotal to maintaining the quality of the mined Q-A pairs. Eventually, we harvest total of 10M instruction steps. Unlike existing the Web without any human crowdsourcing GPT-4 distillation.
pairs from the web using shree-step pipeline.
pairs through these sonection-tunnng datasets, our dataset WEBINSTRUCT is purely mined from
We validate the effectiveness of WEBINSTRUCT by training MAmmo TH2 on various base models (Figure Tp, including Mistral-7B [Jiang et al.
2023], Llama3-8B [Meta] 2024). Mixtral-8 8 7B [Tiagg NAmmoTH2 significantly outperforms the base models seven held-out reasoning benchmarks: TheoremQA [Chen et al. 2023b), GSM8K [Cobbe et al. 2021], MATH [Hendrycks et al., 2021
cct al. [2024], and Yi-34B YYonng et al. 2024
ARC-C [Clakk et al. 2018]] MMLU-STEM [HeudSrTMMeeeddrkk (Hendrycks ct al. 2021b), GPQA Rein et al. 2023 and BBH Suzgun al. 2002). MAmmoTH2-77 improves p Mistral- 7B'ss performance by an average of 14 absolute points, while MAmmo TH2 TH2-343 -4B enhances Yi-
34B's performance by an average of 5.8 absolute points. Notably, Mistral-7B's MATH accuracy can rise from 1 11.2% to 66.7% after training on WEBINSTRUCT. As our dataset contains no in-domain data from our evaluation benchmarks, this highlights the models' strong generalization ability.
We further enhance MAmmoTH2 performance on code generation, math reasoning, and instruction- following tasks by tuning it open-source instruction datasets, including OpenHermes2.5 [Teknium

---

2023].
Code-Feedback (Zheng et al. 2024c), and Math-plu math.plus. The resulting model,
excels on seven reasoning benchmarks and other general tasks. MAmmoTH2-78 P-ls and MAmmoTH2-8B-PPP achieve state-of-the-art performance on TheoremQA, ARC-C, MMLU-STEM, GPQA, and BBH, and competitive results on MATH (45%) and GSM8K (85%). - also performs well on general tasks, with TH2-oTH2-777 7B-Plus showing promising results on Hu- manEval and MBPP, and MAmmoTH2--0 TH- leading the AlpacaEval 2.0 and Arena Hard leaderboards.
both tuned from Llama-3-base using datasets of the same size (10M), provide an apple-to-au to-apple comparison. The only distinction that Mamma-3-8B-Instruue is trained on 1OM muman-annnoateed dataset while we do not require any human annotation. MAmmo TB2-8B--------- Plus outperforms Llama-3-Instenn by 6 points on reasoning tasks while matching its performance on general tasks, reflecting WEBINSTRUCT'S cost-effectiveness advantage. MHm consistently surpasses official instruction models like Mixtral-Instruct on chat benchmarks. These results demonstrate the effectiveness of our approach to scale up instruction data from the web and offer new perspective for future instruction tuning studies.
Interestingly. AHmmoTH2-8BBP1u and Laamma---8B-Istteee
WEBINSTRUCT
In this section, we outline the process of constructing INBINSTUUCT. Specifically, we divide the data collection pipeline into three stages: (1) relevant document recall from the web corpus, (2) Q-A pair extraction from recalled document, and (3) Q-A pair refinement. The full pipeline depicted in Figure 3 and an example for extraction and refinement is provided in Figure 4
2.1 Recall from Common Crawl
In contrast to previous math-centric approaches [Paster et al. [023 Wang et al. 2023c Shao et al.
2024]. aim for broad coverage of disciplines such math, science, engineering, etc. Therefore, careful balancing of the seed data is necessary to ensure diversity. However, publicly available training datasets mostly limited to mathematics. To address this issue, we propose crawl new
Recall from Web
URL Select
Seed
Fast-Text
URL Filter
Fast-Text
Q-A Pair Extraction
Q-A Pair Refinement
Pre-process
Extraction
Decontaminate
LLM Refinement
Figure 3: Step 1: Recall relevant documents from Common Crawl. Step 2: Extracting Q-A pairs. Step 3: Refine with the extracted Q-A pairs.
Rew Docs
Unformatted Site Information, Ads
Rewritten Formatted QA augonented with detailed solutions
Topice SelenceloosstomyPPniiilee http:/iilloog/ntttoommmma/m
InBlologg/nCheemistyy Socratic hoteonommmmytetrrpyyyysiees
Question:
How do you simplify (u* o3/(u² n-1 y4y write it using only
How do you simpiity #yj""vv" Av" (yy"///2y"11"//// 40 and write it usinng only positive exponents?
exponents?
Answer:
NickTheTurtle
To simplify the expression (u* //t((u² pm1 yty" and rewrite evaluating expression from the innermost operation outward.
Axglannation:/kknyhtti miled
(о*(ny)) power simply
using only positive exponents, we
ReinnRelated Questionsin What is the quotient of powers pruperty? simplity
inininnn
quotient
rule? Inimpect questionin
consider exponent of Any expressian raised to the power of zero equal to Iprovided the base zero). Therefore, (u² v²/(u² y-
entire traction:
QA lacking deteried solutions only positive exponents?
Extracted QA
This simplification because raising any non-2aro expression to the power of zero will always result
unnecessary
Question: How you simplity v² ( yty² using
So, the simplified expression simply. 1
Answer: Explanation: Anything the oth power is simply 1.
Figure 4: An illustrating example from WEBINSTRUCT for the extraction and refinement step.

---

exam problems from several educational websites. These sites contain diverse problems from various disciplines, helping to ensure diversity. We crawled 100K seed data as positive training examples and randomly selected 100K negative documents from CC [Computer 2023] to train a fastText model [Joulin et al. 2016]. The trained fastText model used to recall relevant documents. We employ the open-source fastText library with a vector dimension of 2556 to train the model for 3 epochs, with a learning rate of 0.1, a maximum n-gram length of 3, and a maximum number of word occurrences of 3. We recalled 100B tokens using the trained fasttext model from an internal CC. These raw web documents are further grouped by their domains (root URL) and only domains with more than 1000 documents are retained. We extracted roughly 600K domains from the recalled documents. We then prompt GPT-3.5 to scan through the domains and automatically select those that might contain instruction data. Around 50K domains are further labeled as positive samples by GPT-3.5. Note that all the recalled documents in the first round are not kept for further usage in Q-A Pair Extraction and Refinement. Next, we sample documents from the selected domains as positive examples, and documents from the non-selected domains and general Common Crawl as negative examples re-train improved fastText classifier. The newly trained fastTText classifier is used to recall documents. We recalled 40B tokens using the newly trained fastText model. We prompt GPT-4 to sift through the recalled domains again, ultimately leading 18M raw documents, primarily originating from the desired websites.
2.2 Q-A Pair Extraction
We observe that significant number of naturally existing Q-A pairs are present in the 18M documents. However, these Q-A pairs are interspersed with a high volume of noise such ads, markups, boilerplate, etc. Our preliminary training on these raw documents only yields limited gains.
First, we carefully pre-process the HTML to pre-extract useful content from the recalled documents. This is mostly rule-based filtering to clean site information, ads, HTML boilerplate, etc. This step significantly reduces the document length for the next stage. We then prompt Qwen-72B [Bai et al. 2023) to identify the question and answer pairs from the preprocessed documents. Specifically, we provide a few in-context examples to help the model understand what to extract. We also allow the model to return void if no natural question-ansse -answer pairs exist. In this stage, only 30% of the recalled documents were identified as containing naturally existing Q-A pairs, resulting in roughly 5M Q-A pairs as our candidates for the next step. However, these candidates still contain a substantial amount of unrelated content and formality issues. Besides that, large portion of the extracted Q-A pairs also lack explanations for how the answer is derived. Therefore, we propose to perform another round of refinement to increase the data quality.
To avoid contamination, we follow previous work (Shao et al., 2024] and filter out web pages containing questions or answers to all of our evaluation benchmarks. Specifically, we filter out all web pages that contain n-grams (n 10) string matches with either the questions or answers.
2.3 Q-A Pair Refinement
To further improve the extracted Q-A pair candidates, propose refining them using LLMs. In this step, we prompt Mixtral-291 x8 (Jiang et al. [024] and Qwen-72B [Bai et al.] 2023] to reformat the extracted Q-A pairs. If the answer does not contain any explanation, these two LLMs will attempt to complete the intermediate reasoning steps leading to the given answer. We adopt two models to increase the diversity of our dataset. Eventually, we harvest 10M Q-A pairs as final instruction-tuniing dataset WEBINSTRUCT.
2.4 Dataset Statistics
To better distinguish our dataset from the existing ones, we include summarization table in Table It can be observed that most SFT datasets contain less than IM samples but are of high quality. XwinMath (Li et al., (024a] is the largest dataset, scaling up to over 1M samples through GPT4 synthesis. while Matssuutnnttruut [Toshniwal et al. [024] has not been generated using GPT-4 but instead uses Mixtral-8x77 Diang et al. [2024). However, the seed data for both datasets is only based on GSM and MATH, leading to narrow domain coverage. In contrast, continue-training (CT) datasets are normally filtered from the web with much larger size, often exceeding 10B tokens and even rising to 120B tokens. However, continued pre-training these massive datasets can be not only expensive

---

Table 1: The list of existing supervise-fing fine- fine-tuning (SFT) and datasets are primarily from academic NLP sources or synthesized by GPT-3.5/4 using seed data. CT datasets are larger but nosier. Our dataset falls between these two types.
(CT) datasets. SFT
Dutaset
#Pairs
Domain
Format
Dataset Source
FLAN V2 Chung et
2024
100K
General
SFT
NLP data Human CoT
Self-Instruct Wang al
2023b
82K
General
SFT
Generated by GPT3
GPT4-Alpaca Taori of 202
52K
General
SFT
Generated by GPT4
SuperNT Wang al. 2022
96K
General
SFT
NLP Datasels
Tora Gou ct al. 2025
16K
Math
SFT
GSM+MATH Synthesis by GPT4
WizardMali LDb
96K
Math
SFT
GSM+MATH Synthesis by GPT4
MathInstruct
262K
Math
SFT
Math datasets Synthesis by GPT4
MetaMathOA Yū
2023
395K
Math
SFT
GSM+MATH Synthesis by GPT3.5
XwinMath
Li crai. 2024a
1.4M
Math
SFT
GSM+MATH Synthesis by GPT4
OpenMathlnstrnn Toshmwall et
2024
1.8M
Math
SFT
GSM+MATH Symthesi by Mixtral
Dulasel
#Tokens
Domain
Format
Datasel Source
OpttWebMath Puster al.
2023
12B
Math
LM
Filtered from Web
MathPile Wang el al. 20225c
10B
Math
LM
Filtered from Weh
Cosmopenda Ben Allal ml
2024
25B
General
LM
Synthesized by Mixtral
MINERVA Lewkowycz ct al.
2022
38B
Math
LM
Filtered from Web
Proof-Pile-22
Azerbayey ct al.
2023
55B
Math
LM
OpenWebMath-AAxiv--e
Galactica Taylor el al. 2022
106B
Math &
LM
Filtered from Web
DeepseekMatth Shao al.
2024
120B
Math
LM
Recalled from Web
WEBINSTRUCT
(10M) 5B
Math & Sci.
SFT
Rocall Extracted from
but also ineffective due to the high noise ratio. WEBINSTRUCT, with roughly 5B tokens, strikes good balance between scalability and quality. It approaches the scalability of common CT datasets while maintaining high quality through the shree-step construction pipeline. This makes dataset unique compared to other alternatives.
2.5
Additional Public Instruction Datasets
To further enhance the diversity and quality of our dataset, fine-tune MAmmoTH2 on several open- source instruction tuning datasets. These datasets are carefully selected based on their relevance to different reasoning subjects. Additionally, we consider some chat datasets balance reasoning ability and general chat ability. The open-source datasets we incorporate are OpenHermes 2.5 (Teknium) 2023). Code-Feedback [Zheng et (202cc and our Math-Plus which an augmented version of MetaMathQA (395K) (Yu et al. [023] and Orca-Math (200K) [Mitra et al., 2024]. More details of the public datasets can be found Appendix
Experimental Setup
3.1 Training Setup
We unify all the samples our instruction dataset to conform to the structure of a multi-turn instruction tuning dataset. This standardization ensures that the fine-tuned models can process data consistently, regardless of the original dataset formats. We select the open-source models Mistral 7B [Jiang et al. 2023), Mixtral 7B (Jiang et al. 2024). Llama-3 8B [Meta] 2024), and Yi-34B (Young al. 2024 as our base models. We fine-tune these models to validate our WEBINSTRUCE at multiple scales using the LLaMA-Factory (Zheng al. (024d] library. We use learning rate of 5e-6 for Mistral 7B and le-5 for Mixtral, Llama-3 8B, and Yi 34B. The global batch size is set to 512 with maximum sequence length of 4096. We employ cosine scheduler with a 3% warm-up period for 2 epochs. To efficiently train the models, we utilize DeepSpeed [Rasley et al. 2020 with the ZeRO-3 stage. All the models are trained with 32 110 GPUs.
3.2 Evaluation Datasets
To rigorously assess the capabilities of models in reasoning abilities across different domains, we utilize several widely used datasets, GSM8K [Cobbe et al. 202T], MATH [Hendrycks et al., 202
TheoremQA (Chen et al. 2023b), BIG-Bench Hard (BBH) [Suzgun al. 2022), ARC-C CCark

---

Table 2:
Main results on reasoning datasets. Models without the '-Instruct" suffix refer the released base models. Results are taken from official papers or blogs when available; otherwise, we use our own evaluation script. Underscored results represent the best baseline scores under the size constraint, All models are inferred with few-shot CoT: TheoremQA (5-shot), MATH (4-shot), GSM8K (4-shot), GPQA (5-shot), MMLU-STEM (5-shot), BBH (3-shot), and ARC-C (8-shot).
Model
TheoremQA
MATH
GSMSK
GPQA
MMLU-ST
BBH
ARC-C
AVG
GPT-4-Turbo-0409
48.4
69.2
94.5
46.2
76.5
86.7
93.6
73.6
Parameter Size between 20B and 110B
Qwen-1.5-110B
34.9
49.6
35.9
74.8
91.6
63.6
Qwen-1.5-72B
46.8
36.3
68.0
92.2
Deepscek-LM-67B
25.3
15.9
31.8
57.4
71.7
86.8
50.7
Yi-34B
23.2
15.9
29.7
62.6
66.4
89.5
50.7
Llemma-34B
21.1
25.0
71.9
29.2
54.7
48.4
69.5
45.7
Mixtral-8x7B
23.2
28.4
74.4
29.7
59.7
66.8
84.7
52.4
Mixtral-&x7BIIman
25.3
22.1
71.7
32.4
61.4
57.3
84.7
50.7
Intem-Math-20B
17.1
37.7
82.9
28.9
50.1
39.3
68.6
46.4
Trained only with WEBINSTRUCT (All evaluations are held-out)
XAmmoTH2-344
30.4
35.0
75.6
31.8
64.5
68.0
90.0
56.4
A over Yi
+7.2
+19.1
+7.7
+2.1
+2.9
+1.2
+0.5
+5.8
XAmmoTH2-B8 TH2-8x7B
32.2
39.0
75.4
36.8
67.4
71.1
87.5
58.9
A over Mixtral
+9.2
+10.6
+1.0
+7.1
+7.4
+3.3
+2.8
+6.5
Continue trained with additional instruction datasets (All held-out except MATH and GSM8K)
XAmmoTH2-8x7B. PxP
34.1
47.0
86.4
37.8
72.4
74.1
88.4
62.9
A over Qwen-1.55 1.5---1.-1100
-0.8
-2.6
+1.0
+1.5
-1.0
-0.7
-.00
-0.7
Parameter Size
7B SB
Deepseek-7B
15.7
6.4
17.4
25.7
43.1
42.8
47.8
28.4
Owen-1.5-7BB
14.2
13.3
54.1
26.7
45.4
45.2
75.6
39.2
Mistral-7B
19.2
11.2
36.2
24.7
50.1
55.7
74.2
38.8
Gemma-7B
21.5
24.3
46.4
25.7
53.3
57.4
72.5
43.0
Llemma-7B
17.2
18.0
36.4
23.2
44.9
50.5
33.6
WizardMath-7B-l
11.7
33.0
83.2
28.7
52.7
56.7
76.9
49.0
Abcl-7B-002
19.3
29.5
83.2
30.3
29.7
32.7
72.5
42.5
Intem-Math-7B
13.2
34.6
48.1
59.8
42.5
Rho-1-Math-7B
21.0
31.0
66.9
29.2
57.7
72.7
47.3
Deepseek-Math-7R
25.3
34.0
29.2
56.4
59.5
67.8
48.0
Deepscck-Math-Instuue
23.7
44.3
82,9
31.8
59.3
55.4
70.1
52.5
Llama-3-8B
20.1
21.3
54.8
27.2
55.6
61.1
78.6
45.5
Llama-3-8B--Inse
22.8
30.0
79.5
34.5
60.2
66.0
80.8
53.4
Trained only with WEHINSTRUCT (All evaluations are held-out)
MAmmoTH2-78
29.0
36.7
68.4
32.4
62.4
58.6
81.7
52.8
A over Mistral
+9.8
+25.5
+32.2
+7.7
+12.3
+2.9
+7.5
+14.0
AAmmoTH2-880 8BB A over Llama3
32.2
35.8
70.4
35.2
64.2
62.1
82.2
54.3
+12.2
+14.5
+15.6
+8.0
+8.6
+1.0
+3.6
+8.8
Continue trained with additional instruction datasets (All held-out except MATH and GSM8K)
3AmmoTH2-7BBPPP 7B-Plus TH2-8- 88-Plus
29.2
45.0
84.7
36.8
64.5
63.1
83.0
58.0
32.5
42.8
84.1
37.3
65.7
67.8
83.4
59.1
A over best baseline
+7.2
+0.7
+1.5
+2.8
+5.5
+1.8
+2.6
+5.7
ct al. 2 GPQA RRinn et al. 2023], MMLU-STEM [Hendrycks et al. 2021a These datasets
collectively enable a comprehensive assessment of language models' reasoning prowess across spectrum of complexity and realism. The details of the evaluation datasets are in Appendix B
We further evaluate the models on additional code generation tasks (including HumanEval [Chen
et al. 2021]. MBPP [Austin et al. [2021] and their augmented version [Liu et al.) 2024)) )) general
LLM benchmarks like MMLUU Henddrykks al. 2021a] and its recent robust and challenging version
MML MMLU-Pro (TIGER-Lab 2024]. We also consider chat benchmarks like MTT-Bench (Zheng et al. 2024a], AlaacaEval 2.0 Li et al. 20236), and Arena Hard [Li et al. 2024b to demonstrate the
generalizability of WEBINSTRUCT and TEBBNSTTRUCTPPLL
more general LLM benchmarks.

---

Main Results
4.1
Experimental Results on Reasoning Benchmarks
presents main results, with existing models partitioned into two tracks based their parameter size. For 7B parameter models, we observe that our model trained solely with WEBIN STRUCT achieves significant improvements over the base models. For instance, MAmmoTH2-7T 7B boosts the performance of Mistral-7B by an average of 14 points. Notably, WEBINSTRUCT does not contain any training data from these evaluation benchmarks, making all evaluations essentially held-out. The substantial performance gains demonstrate the strong generalization capabilities MAmmo WH2-7T Similarly, MHmmoTH2-8B boosts the performance of Lllam-3-8B--al
Table 2
by an average of 8.8 points. We also experiment with larger models like Yi-34B and Mixtral to show that the performance gains are consistent across the board. Notably, Yi-34B's performance on MATH also increases by 19% after training on WEBINSTRUCT.
Further tuning several additional public datasets also significantly enhances performance. The MAmmo TH2-PP-2-P-- PPlus model family achieves state-of-the-art results across the board. For example, THM-nooHHPPPU performance TheoremQA, GPQA, and ARC-C represents the best-known results for any model under 10B parameters. MAmmo TH2-7B-Pluse performance on MATH and GSM is also close the best-known results. We also show the results of the models solely trained the additional public datasets in Appendix
An interesting comparison is between THm 8-0---8B-PP-1 Plus and Llama3-Instruct, both models are trained from the Llama3-base. Llama-3-instruet was trained on a 10M auman-annotater in- struction dataset along with public datasets, similar to WEBINSTRUCT combined with additional public datasets. Therefore, these two models are highly comparable. Our experiments show that outperforms Llama3-Instruct by an average of 6% across the benchmarks. This substantial gain indicates that WEBINSTRUCH is highly cost-effective. For larger models, we found that TH2-872 11122-xxBBPPIun can match the performance of Qwen-1.5- 1.0 with only 13B active parameters. These results demonstrate the effectiveness of our scalable instruction tuning approach.
4.2 Additional Experimental Results
To further demonstrate the capabilities of our models beyond the reasoning benchmarks presented in Table 2, conduct additional experiments to evaluate their performance code generation, general language understanding, and cestruction-folloimnn tasks. [able showcases the results of var- ious models on code generation tasks. The TH2-7B 7---7BBPPu model exhibits strong performance, achieving the highest average scores of 66.1 and 58.2 on HumanEval(()) and MBPP(+) datasets, respectively. It outperforms the official instruct counterparts like Distral-7B-Instruut--00 metrics, indicating its superior code generation abilities.
these
To assess the general language understanding and tnolruction-followiigg capabilities of our models, we evaluate them on a range benchmarks, as shown Table 3 The MAmmoTH2-PP models exhibit strong performance across these tasks, showcasing their versatility and robustness. For example, 1 achieves the highest scores on AlpacaEval 2.0 and Arena Hard leaderboards, surpassing competitive models like GPT-3.5-Turbo and Tulu-2-DPO-70B
[vison et al. 2023
MATH
TheoremQA
ARC-C
30
80.0
Accurscy (%)
Accuracy (%)
25
20
Accurssy
77.5
75.0
20
15
72.5
2M 4M 6M 8M 10M
2M
4M
6M 8M
10M
2M
8M 10M # of Instructions
4M 6M
# of Instructions
of Instructions
Extracted QA (LM Loss)
Refined QA (LM Loss)
Refined QSFT Loss)
Figure 5: Mistral-7B model reasoning performance improves with scaling instructions. Additionally, SFT Loss is more effective learning approach compared to LM Loss.
7

---

Table 3: Evaluation of code generation, cestructioo-followiin and MMLU(-Pro) performance for various models. We report the average of HumanEval(+) and MBPP (+) accuracy as the code generation performance (breakdown results are Appendix D). Baseline scores are sourced from the original papers or the EvalPlus, MT-BBench, AlpacaEval 2.0, Arena Hard and MMLU-Pro leaderboards. (*-") indicates that the score not available from the sources. TH2 THu-Plus exhibits strong general conversational ability and excels at multitask language understanding across wide range of domains compared to their official instruct counterparts and larger models.
Code Generation
Alpaca Eval 2.0
Arena Hard
MT-Bench
MMLU
MMIUUPro
GPT-4-1106-preveew
85.6 (77.5)
9.32
50.0
GPT-3.5-Turbo-1100
79.7 (70.2)
8.32
19.3
18.9
GPT-3.5-Turbo-030
7.94
18.1
18.1
70.0
Tulu-2-DPO-70B
51.2 (43.0)
7.89
21.2
15.0
67.8
40.5
Llama-2-70b-chaa
31.4 (26.5)
6.86
14.7
11.6
63.0
33.6
Yi-34B-Chat
38.7 (32.6)
7.86
27.2
23.1
73.5
42.1
Mistral-7B- 1B-nndtttctttt
43.4 (36.5)
7,60
17.1
12.6
60.8
30.8
Llama-3-8B Instruct
65.8 (58.0)
8.02
22.9
20.6
67.2
40,9
Mixtral-8 8annnsx----tnnnttnnnnn
52.3 (44.7)
8.30
23.7
23.4
70.6
41.0
MAmmo T---n---12---P-
66.1 (58.2)
7.88
23.4
14.6
63.3
40.9
61.9 (53.3)
7.95
18.5
16.6
64.6
43.4
MAmmo H1HHH228-888PP-u
63.3 (55.3)
8.20
33.8
32.6
68.3
50.4
The strong performance of MAmmoTH2 code generation and general language understanding tasks, as evidenced by Table demonstrates that our method does not overfit to the reasoning benchmarks. Instead, it shows the models' ability generalize well to wide range of tasks, highlighting their versatility and robustness. These additional experiments further validate the effectiveness of our WEBINSTRUCT in developing powerful and flexible language models.
Ablation Study
5.1
Scaling Effeet of Instructions
first investigate the impact of model scaling and loss functions on the performance language models across three representative tasks: MATH, TheoremQA, and ARC-C. We train models with varying training samples (1M to 10M) using extracted QA and refined QA data, and compare the effectiveness of two training losses: LM loss and SFT loss. Figure 5) showw that increasing model size and using SFT Loss with synthetic data consistently improves accuracy across all tasks. These findings demonstrate the importance of model scaling and supervised fine-tuning with synthetic data for enhancing language model performance in various domains.
5.2 Comparison of Two Refined Models
To assess the effectiveness of the Q- A pair refinement process by different LLMs, we conducted experiments by training three Mistral-7B models: one on the data refined by Mixtral-22B x8, an- other on the data refined by Qwen-72B, and a third on the merged samples fined by both models. For fair compari- son, we trained the models with the same 9000 steps and global batch size of 512. Our results show that the model trained on Mixtral-22B: x 8 refined data achieves comparable performance to the one trained on Qwen-72B refined data. The model trained on the merged samples consistently outperforms the models trained on data refined by individual L.Mss. This demonstrates that combining data refined by different models mitigates potential biases that may arise from using single model. By leveraging the strengths of both
Table 4: Comparison the two data-refining LLMs. We train the three models with the same steps.
Data
GSM
MATH
MMLU-
Theo.
ARC.
Mixtral
62.9
29.1
56.5
26.1
78.3
Qwen
65.4
28.9
60.6
23.5
80.8
Merged
65.6
31.0
60.5
24.8
81.8

---

Mixtral and Qwen, we obtain a more balanced and comprehensive dataset, preserving the diversity of naturally occurring web instructions while enhancing data quality. Unlike traditional distillation methods, our approach does not directly inherit knowledge from other models but instead utilizes LLMSs for data extraction and refinement. This distinction helps preserve the original structure and content of instructions from diverse web sources, potentially reducing model-introduced biases.
5.3 Comparison of Different Domains and Sources.
To understand how each domain (c.g., math, science, others) and data source (c.g., forum websites and education websites) contribute to the training. train Mistral 7B on the subsets of different domains and data sources. Details how we obtain domain labels can be found in Appendix G
As shown in Table 5] training on differ- ent domains and data sources leads varied performance across the evaluation benchmarks. The education website data source consistently outperforms the fo- rum data source, indicating the higher quality of educational questions. Inter- estingly. while the math domain excels on MATH. it does not lead significant improvements on GSM8K, another math- focused dataset, suggesting that training on a single math dataset may not gen- eralize well to other math benchmarks. Furthermore, training solely on the math domain does yield substantial gains on science and STEM benchmarks, highlighting the need for diverse training. In contrast, the "Other" domain, which includes diverse range of subjects, achieves the highest score on GSM8K, emphasizing the importance of diversity in the training data.
Table 5: Impact of different data domains and sources on model performance. All models are trained with identical steps; Base denotes the base model's performance.
Data Source
GSM
MATH
MMLU-S
Theo.
ARC.
Base
47.4
15.7
51.4
17.3
77.6
Forum
$1.0
24.0
54.7
21.0
78.2
Education
58.0
24.8
54.3
23.2
79.5
Math
52.9
27.3
51.6
21.7
74.1
Science
54.4
23.7
58.9
21.0
83.6
Other
59.4
20.8
55.3
21.1
79.4
5.4 Effectiveness of Extract and Refine Steps
Our ablation studies highlight the essential role of the "Extract" and "Refine" steps in our data pipeline, demonstrating that these steps contribute significantly to model performance on broad range of reasoning tasks.
Effectiveness of Extract and Refine Steps. study, define each setup follows:
To clarify the experimental settings used our ablation
(a) Recall: Training 18M recalled documents, serving as the baseline dataset.
(a') DeepSeek Math Recall Shao et al. 2024 Deepseek Math 500B pre-training tokens.
(b) Extract: Filtering the recalled documents to retain only high-quality question answer pairs.
(c) Refine: Further refining the extracted pairs by adding reasoning steps and clarifying responses.
(d) Public SFT: Supervised fine-tuning using additional public fine-tuning data.
(d') DeepSeek Math SFT: The SFT set from the DeepSeek Math.
The results, shown in Table indicate that training solely on recalled documents with supervised fine- tuning. i.c., Recall + Public SFT (a + d), achieves only modest gains, highlighting the importance of filtering and refining the initial corpus. Adding the "Extract" step, or Recall + Extract Public SFT (a + d), yields notable improvements by narrowing the training data to high-quality question- answer pairs, thus reducing token usage while preserving strong performance across benchmarks.
Incorporating the "Refine" step, or Recall Extract Refine Public SFT (a b c + d), further improves model accuracy by enhancing the rationale and clarity of responses, which is particularly beneficial for maasoning-inteniie tasks. Additionally, comparisons with the DeepSeek Math Corpus (a") and DeepSeek Math SFT (d') demonstrate that while domain data can produce high scores on targeted benchmarks, our generalized pipeline yields comparable or even superior results across a broader array of reasoning tasks.

---

Table 6: Summary of ablation study results using different training data. The Base Model is DeepSeek
Coder VI.5 7B Guo et al. 2024].
Setting
Model
#Tokens
TheorQA
MATH
GSM8K MMLU-S
BBH ARC-C
AVG
evalnations are held-out
Base Model
DS Coder v1.5
18.3
22.3
47,9
47.0
53.5
62.4
41.9
+ (a)
28B
23.5
30.3
60.3
53.3
55.5
69.4
48.7
+ +(a) + (b) + (c) (a')
MAmmoTH2-DS
10B
27.8
33.8
64.0
$6,9
58.5
72.8
$2.3
Math Base
500B
held-out except GSM and MATH (d)
2B
77.5
66.9
52.8
+a) + (d)
30B
27.2
39.2
79.2
55.6
60.3
71.5
55.5
+(a) (b) (d)
9B
27.3
38.6
78.5
54.2
60.5
70.4
54.9
+a) (b) (c) MAmnoTH2 THa-S-HH2---220 -
12B
43.8
80.1
61.0
73.2
58.0
+aa)
DS Math Instruct
501B
82.9
55.4
70.1
56.0
conclusion, these findings validate necessity both the "Extract" and "Refine" steps in building an effective instruction dataset from web-sourced data. This approach not only provides scalable and economical alternative to human-annotated data but also enhances the model's generalization capability wide array of reasoning tasks.
5.5
Case Study
We further conduct a case study examining the quality of extracted and refined QA pairs from the dataset. We showcase some good and bad cases in Appendix J We observe that the rreestion/answer pairs extracted from well- formed exam and education websites are of high quality, The common issue that large portion of extracted answers do not contain intermediate rationale (chain-of- thought). This issue could lead worse generalization.
4%
4%
2%
Wrong Answer
Wrong Question
Errors: Correst
Wrong CoT
10%
Unckanged: 12%
Therefore, we prompt Mixtral and Qwen-72B to complete the intermediate steps. We observe that the success of such completion is relatively high. However, there are cases where the extracted question/answer pairs contain serious formatting issues, which pose challenges for the following refinement step. Besides these issues, we also observe that LLMs can sometimes modify the intention of the originally extracted content, causing hallucinations.
Fully Correct Enbanced: 78%
Figure 6: Quality distribution of 50 sam- pled refined QA examples.
To quantify the error percentages, we randomly sample 50 refined QA examples and ask the human annotators compare whether the refined examples are correct and significantly better than the extracted ones in terms of format and intermediate solutions. As we can see from Figure 6) 78% examples have been improved after refinement and only 10% examples introduce hallucinatioon after refinement. Overall, our case study reveals that the harvested instruction tuning dataset is generally accurate with low error rate.
Conclusion
In this paper, we argue that the web corpus contains vast amount of high-quality instruction data across various domains. To mine this data, we develop three-step pipeline consisting of recall, extraction, and refinement steps. Through this pipeline, we harvest WEBINSTRUCT. a total of 10M diverse, high-quality instructionnresponne pairs and train language models. Our experiments demonstrate that MAmmoTH2 exhibits significantly enhanced science reasoning abilities compared to baseline models. Our work showcases the potential of harnessing the vast amount of instruction data in the web corpus democratize the development of LLMs with enhanced reasoning capabilities.
10

---

References
Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, Altman, S. Anadkat, et al. Gpt-4 technical report. ArXiv preprint, abs/2303.087744, 2023. URL
https://arxiv. org/ abs/ wtabs//33..om 2330308777
A. Amini, S. Gabriel, S. Lin. R. Koncel- Koizel-kedziorski, Y. Choi, and H. Hajishirzi. MathQA: Towards interpretable math word problem solving with operation operation-based formalisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: 2019. doi:
Human Language Technologies. Volume (Long and Short Papers), pages
10.18653/v 1/N19-122211 1245. URL pttttp://aacaanntoolos
org/ N19- 1245
J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. ArXiv preprint, abs/2108.07732, 2021.
URL https: http:p:://arxiv.oorte abs//a/......//////// 07732
Z. Azerbayev, H. Schoelkopf, K. Paster, M. Dos Santos, S. M. McAleer, Q. Jiang, Deng. S. Biderman, and S. Welleck. Llemma: An open language model for mathematics. In The Twelfth International Conference on Learning Representations, 2023. URL Sttttttts///peereeeiie net/ forum? forum/id=4WnRRR915]
J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang. X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, al. Qwen technical report. ArXiv preprint, abs/2309. 16609, 2023. URL htttt://t/::/iiv .rg wtrg/bbs/2399 16609]
L. Ben Allal, A. Lozhkov, G. Penedo, T. Wolf, and L. von Werra. Cosmopedia, 2024. URL https://huggingfacce wtt.////to///ttiiiecooe Batgggginnaaeee//osnnoieee
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems 33; Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL htttts::/pprceedinee neurips wwtppcpr//ccpppppccccccccee.eeat http://////naaa/150000000aa000a0 05770101100111117770100000000000 1aa.11444444---aaaaarreaae Abstract html
M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. ArXiv preprint, //bs/2117.03777 2021. URL httpss/////ar
httg/bbs/2100/ 20000//00733374
W. Chen, X. Ma, X. Wang, and W. W. Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. Transactions on Machine Learning Research, 2023a. URL httts:://pperrerr/eee wttp:/iwwwweeeeeeeeeeeeeeeeeeee wwt.:e//ooormm..dxe/4222....
W. Chen, M. Yin, M. Ku, P. Lu, Y. Wan, X. Ma, J. Xu, X. Wang, and Xia. Theoremqa: A theorem- driven question answering dataset. In The 2023 Conference Empirical Methods in Natural Language Processing, 2023b. URL https: aalaanoolotgyyyo org/ 2023. emnlp- main. 489/
H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, Y. Li, X. Wang, M. Dehghani, S. Brahma, al. Scaling finteteeon-iinetuneed language models. Journal of Machine Learning Research, 25(70):1-53, 2024. URL //axxi
/ 2210 11416
P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge. ArXiv preprint, abs/1803.05457) 2018. httpss//s///a
org/ abs/ 15000.08510057
K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. ArXiv preprint, 10.1111. 2021. URL /ntps://arxiy, org/ abaaaaaa/2110 14168
Computer. Redpajama: an open dataset for training large language models, 2023. URL https //github.com http://www.o.ggggeeeeeooprrttaaa

---

L. Gao, A. Madaan, S. Zhou, U. Alon, P. Liu, Y. Yang, J. Callan, and G. Neubig, Pal: Program-aide language models. In International Conference on Machine Learning, pages 10764-10799. PMLR, 2023. URI.httpa /tatps///ariv. org/ pdf 22111 10435
org/pdf
Z. Gou, Z. Shao, Y. Gong, Y. Yang, M. Huang. N. Duan, W. Chen, et al. Tora: A tool-integrated reasoning agent for mathematical problem solving. ArXiv preprint, abs/2309. 17452, 2023. URL https://arxiv.cm www...ogggaaoo abs// 2209. 17452
D. Guo, Q. Zhu, Yang. Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y. Wu, Y. Li, et al. Deepseek- coder: When the large language model meets programming-the rise of code intelligence. arXiv preprint arXiv:2401 1402444114496... 14196, 2024.
D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measur- ing massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021a. URL https wetaarrvieew.ne/forree jut.:/orrmmmiddddiiii/aaa Im3mml
D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and Steinhardt. Measuring mathematical problem solving with the math dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). 2021b. URL
H. Ivison, Wang, V. Pyatkin, N. Lambert, M. Peters, Dasigi, Jang, D. Wadden, N. A. Smith, I. Beltagy, et al. Camels in changing climate: Enhancing Im adaptation with tulu 2. ArXiv preprint, abs/2311.10702, 2023, URL htttps///aa http:////axxvvo.//pfff.. /2111 10702
A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. I. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, et al. Mistral 7b. ArXiv preprint, abs/2310.066 2023, URL
https://arxiv.org//445 org. hbtrg//bb/233000 06825
A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. I. Casas, E. B. Hanna, F. Bressand, et al. Mixtral of experts. ArXiv preprint, abs///b/.//////e 2024. URL /tttps://arxi...oo wrw. wttp/r//ggb//..... 24000000888000 04088
A. Joulin, E. Grave, Bojanowski, M. Douze, H. Jégou, and T. Mikolov. Fasttext. zip: Compressing text classification models. ArXiv preprint, abs/1612.03651,, 2016. URL Utthhtts::// httpss///pxxxii org/
abs/1612.0365.
A. Lewkowycz, A. Andreassen, Dohan, E. Dyer, H. Michalewski, V. Ramasesh, Slone, C. Anil, I. Schlag, T. Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 15:3843-3857 2022. URL URL https //openreview. wwwpl/eww.ee.ooooeeeeeeeeeeeee nww/ffrumm dim
C. Li, W. Wang. J. Hu, Y. Wei, N. Zheng, H. Hu, Z. Zhang, and H. Peng. Common 7b language models already possess strong math capabilities. ArXiv preprint, abs/2403.04706, 2024a. URL btopps://ariiy org/ abs/ 240324433000 04706
T. Li, W.-L. Chiang, L. D. Evan Frick, B. Zhu, E. Gonzalez, and Stoica. From live data to high-quality benchmarks: The arena arena-hard hard pipeline, April 2024b. URL https://mmsys org/ blog/2028 04 19 weweena-harddrr
X. Li, P. Yu, C. Zhou, T. Schick, O. Levy, L. Zettlemoyer, J. Weston, and M. Lewis. Self-alignment with instruction backtranslation. arXiv preprint 1.Xiv:23388.625599 2023a.
X. Li, T. Zhang, Y. Dubois, R. Taori, I. Gulrajani, C. Guestrin, P. Liang, and T. B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models. https://githe: com/ tatsu 1ab/ httb//lpaca__ette evall 2023b.
Z. Lin, Z. Gou, T. Liang, R. Luo, H. Liu, and Y. Yang. Criticbench: Benchmarking llms for critique- correct reasoning. ArXiv preprint, abs/2402. 14809, 2024. URL htttps://aaai http:///axvvv.////.a/// wwtp//s////////0/// 14809
12

---

J. Liu, C. S. Xia. Y. Wang, and L. Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36, 2024. httl:hhtps:///pppeeee
net. wttp://eww..eepdddd///g/oooooo..
H. Luo, Q. Sun, C. Xu, P. Zhao, J. Lou, C. Tao, X. Geng, Q Lin, S. Chen, and D. Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. ArXiv preprint, abs/2308.000 09583, 2023. URL https:t:t:// http::://arxivv org/
Meta. Introducing meta llama 3: The most capable openly available Ilm to date. https: /ai meta com/blog/meta-- 11ama-3/ April 2024.
A. Mitra, H. Khanpour, C. Rosset, and A. Awadallah. Orca-math: Unlocking the potential of slms in grade school math. ArXiv preprint, http//2222.188000. 2024. UtLhttss:://aat //axxiv org/ abs/
2402. 14830
M. Nye, A. J. Andreassen, G. Gur-Ari, H. Michalewski, J. Austin, D. Bieber, D. Dohan, A. Lewkowyez. M. Bosma, D. Luan, et al. Show your work: Scratchpads for intermedi- ate computation with language models. In Deep Learning for Code Workshop, 2022. URL https://openrevier net/ forum?ddriedY.Jmm2oo.
L. Ouyang. J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agar- wal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022. URL https: https::/pprocoddnngs neurips cc/ cc/paper_...pp.p httppppprrellleppppppe//oeppppee piles//pper/2022//iii. (brtee600B447794488660000777---3
Conference...ccccoomeo pdf
K. Paster, M. Dos Santos, Z. Azerbayev, and J. Ba. Openwebmath: An open dataset of high-quality mathematical web text. In The Tweifth International Conference on Learning Representations,
2023. URL
B. Peng, C. Li, P. He, M. Galley, and J. Gao. Instruction tuming with gpt-4. ArXiv preprint, abs/2304.03277. 2023. URL https: /tttpps///rriivy wtw.aaiwww.....aaaaaooooaoooo abbs/2044 03277
Rasley, S. Rajbhandari, O. Ruwase, and Y. He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In R. Gupta, Y. Liu, J. Tang, and B. A. Prakash, editors, KDD '20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020, pages 3505-3506, 2020. URL https: :/al acm. org/ wttgg//////oo/// 11111114//344883 32494486633666033333333333333333 3406703
D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman. Gpqa: A graduate-level google proof q&a benchmark. ArXiv preprint, abs/2311. 122222,, 2023. URL http:tt//axxxi..oot org
12022
V. Sanh, A. Webson, C. Raffel, S. H. Bach, L.. Sutawika, Z. Alyafeai, A. Chaffin, A. Stiegler, A. Raja, M. Dey, M. S. Bari, C. Xu, U. Thakker, S. S. Sharma, F. Szczechla, T. Kim, G. Chhablani, N. V. Nayak, D. Datta, J. Chang, M. T. Jiang, H. Wang, M. Manica, Shen, Z. X. Yong, H. Pandey, R. Bawden, T. Wang, T. Neeraj, J. Rozen, A. Sharma, A. Santilli, T. Févry, J. A. Fries, R. Techan, T. L. Scao, S. Biderman, L. Gao, T. Wolf, and A. M. Rush. Multitask prompted training enables zero-shot task generalization. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, 2022. URL/attss:////// / http://ppenrrvieeeeeeeeeoo/oooaa id-9Vrb9DowI4
Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, M. Zhang, Y. Li, Y. Wu, and D. Guo. Deepseekmath: Push- ing the limits of mathematical reasoning in open language models. ArXiv preprint, abs/2402.03300, 2024. URL http:tttt///ai..oooo org/ abs/ 2402. 03300
A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, A. Gupta, A. Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023. URL https://openreviee nett/forunn.d-uyyISBBooo
L. Sun, Y. Han, 7. Zhao, D. Ma, Z. Shen, B. Chen, L. Chen, and K. Yu. Scieval: A multi-level large language model evaluation benchmark for scientific research. ArXiv preprint, abs/2308.131 13149, 2023. URLpttps:: // arxiv. org/ abs/2308 13149
13

---

M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay. H. W. Chung. A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou,, and Wei. Challenging big-bench tasks and whether chain of of can solve them. ArXiv preprini, abs/2210.09261 2022. URL. httts:://arxiie org 2210. 092661
Z. Tang, X. Zhang, B. Wan, and F. Wei. Mathscale: Scaling instruction tuning for mathematical reasoning. ArXiv preprint, abs/2403. 028844, 2024. URL hRtpss/////ax org/ abs/ 2403. 02884
R. Taori, Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and I. B. Hashimoto. Stanford alpaca: An instruction-following llama model. https:///githb. cow www.o//ttt/oo//////oooeeoeeeee lab/ stanford_alpaca 2023.
R. Taylor, M. Kardas, G. Cucurull, T. Scialom, A. Hartshorn, E. Saravia, A. Poulton, V. Kerkez, and R. Stojnic. Galactica: A large language model for science. ArXiv preprint, abs/2211.09085, 2022.
URL atttp://arxivv org/ abs/ 2211.09085
G. Team, R. Anil, S. Borgeaud, Y. Dai, A. Hauth, et al. Gemini: a family of highly capable multimodal models. ArXiv preprint, abs/2312.1180h 2023.
Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M.
http:/::aaaax.o/////.aa/ org/ abwg/22../333..... 11805
Teknium. Openhermes 2.5: An open dataset of synthetic data for generalist Ilm assistants, 2023.
URL http::///uuggiingfaac co/
Lab, Mmlu professional dataset. Hugging Face Dataset Hub, 2024. URL https://
huggingface..co/aataassaaa wot.aaaaatt////iiia.. LaM/MMMU-PMMMMMM
S. Toshniwal, I. Moshkov, S. Narenthiran, D. Gitman, F. Jia, and I. Gitman. Openmathinstruct--l: l:
A 1.8 million math instruction tuning dataset. ArXiv preprint, abs/2402.10176, 2024. URL
https://arxiv... org/ /btp///2...... 10176
H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. ArXiv preprint, abs/2302. 13971, 2023a. (tRLhttps:/ //axxiv./// hts//bbb/23022 13971.
H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. ArXiv preprint, ////2377.09283 2023b. URL https: org/ aa//abaa/22200 .09288
X. Wang. 7. Hu, P. Lu, Y. Zhu, J. Zhang, S. Subramaniam, A. Loomba, S. Zhang, Y. Sun, and W. Wang. Scibench: Evaluating college-level scientific mroblem-sovvin abilities of large language models. In The 3rd Workshop on Mathematical Reasoning and Al at NeurlPS 2023a. URL https://openrest
Y. Wang, S. Mishra, P. Alipoormolabashi, Y. Kordi, A. Mirzaei, A. Naik, A. Ashok, A. S. Dhanasekaran, A. Arunkumar, D. Stap, F. Pathak, G. Karamanolakis, H. Lai, I. Purohit, Mon- dal, J. Anderson, K. Kuznia, K. Doshi, K. Pal, M. Patel, M. Moradshahi, M. Parmar, M. Purohit, N. Varshney, P. R. Kaza, P. Verma, R. S. Puri, R. Karia, S. Doshi, S. K. Sampat, S. Mishra, S. Reddy A, S. Patro, T. Dixit, and X. Shen. Super-Natuuallnsttuutiio
General-
ization via declarative instructions on 1600+ NLP tasks. In Proceedings of the 2022 Confer- ence on Empirical Methods in Natural Language Processing, pages 5085-5109, 2022. URL httpps:/acclanthollgs org/2022 emnlp wommp-main.... 340
Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi. Self-instruct: Aligning language models with self generreed instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 13484- 13508, 2023b. URL httptttttt///aatta/
org/ abs/ 2212.2222221112 10560
Z. Wang, R. Xia, and Liu, Generative ai for math: Part i-mathpile: billion-token--calle pretraining corpus for math. ArXiv preprint, abs/2312. 2023c. URI. htt.tttsss//aaiii.orrtttt www.oorgggbbbbbsbeeoeee 17120
Wei, M. Bosma, V. Y. Zhao, K. Guu, A. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le. Finetuned language models are zero-shot learners. In The Tenth International Conference on Learning Representutions, ICLR 2022, Virtual Event, April 25-29, 2022, 2022a. URL https //openreview.net/eeeeeeeeeeeeeee
14

---

J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, V. Le, D. Zhou, et al. Chain-of of-thuught prompting elicits reasoning large language models. Advances in neural information processing systems, 35:24824-24485
2022b. httpstts::// htttttss::/pppreeeieeeee hetpaarwwieweeeee/ppeeee bet/pdf?idd ?id=_ _ijQMMeSB__
C. Xu, Q Sun, K. Zheng. X. Geng. P. Zhao, J. Feng, C. Tao, and D. Jiang. Wizardlm: Empowering large language models to follow complex instructions. ArXiv preprint, abs/2304.12244 2023. URL https: /ttaas////xxiii org/ 2304. 12244
H. Ying, S. Zhang, L. Li, Z. Zhou, Y. Shao, Z. Fei, Y. Ma, J. Hong, K. Liu, Z. Wang, al. Internlm-math: Open math large language models toward verifiable reasoning. ArXiv preprint, abs/2402.06332, 2024. URL httph::tt////c httts:t//aaaiiii org/ abs/2402. 06332
URL/https:
A. Young, B. Chen, C. Li, C. Huang, G. Zhang, G. Zhang, H. Li, J. Zhu, Chen, J. Chang, et al. Yi: Open foundation models by 01. ai. ArXiv preprint, abs/2403.0460 2024. URL
attps://arxxs org/ abs abs/2403. 04652
L. Yu, W. Jiang, H. Shi, Jincheng, Z. Liu, Zhang, Kwok, Z. Li, A. Weller, and W. Liu. Metamath: Bootstrap your own mathematical questions for large language models. In The Twelfth International Conference on Learning Representations, 2023. URL https:://peerrwwe nnt/ forum? Corum'ic-NNNNOhghiDRt
L. Yuan, G. Cui, H. Wang, N. Ding, X. Wang. J. Deng, B. Shan, H. Chen, R. Xie, Y. Lin, al. Advancing Ilm reasoning generalists with preference trees. ArXiv preprint, abs/2404.02078, 2024. URL https:://arx: orgg//ii...oooooooo/e. ab0/ 02078
X. Yue, Y. Ni, K. Zhang. Zheng, R. Liu, G. Zhang, Stevens, D. Jiang, W. Ren, Y. Sun, al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. ArXiv preprint, abs/2311 I. 16502, 2023a. URL htttp:///axx:
wwg.abs/////ooooooeo..../eaae/ abs 2311. 16502
X. Yue, Qu, G. Zhang. Fu, W. Huang, Sun, Y. Su, and Chen, Mammoth: Building math generalist models through hybrid instruction tuning. In The Twelfth International Conference on Learning Representations, 2023b. / www..neeooommmddyllletet
Y. Zhang, Y. Luo, Y. Yuan, and A. C.-C. Yao. Automathtext: Autonomous data selection with language models for mathematical texts. ArXiv preprint, abs/2402.07625, 2024. URL https // //arxiv.org abs abs/2402. 07625
L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al. Judging Ilm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems. 36, 2024a. URL htthtt:s::// //oberr:////prreeee htt//ppervviiieee eview net
T. Zheng, S. Guo, X. Qu, J. Guo, W. Zhang, X. Du, C. Lin, W. Huang, W. Chen, J. Fu, et al. Kun: Answer polishment for chinese self alignment with instruction bock-translation. ArXiv preprint, 2024b. httpttttt////iiiie owtrxxiivi.orr//eoo absp/401.... 06477
T. Zheng, G. Zhang. T. Shen, X. Liu, B. Y. Lin, J. Fu, W. Chen, and X. Yue. Opencodeiinleppreter:: Integrating code generation with execution and refinement. ArXiv preprint, abs/2402. 146588 2024c.
URL https::/aarx:: org./ii.orr///oooooo 2222 14658
Y. Zheng, R. Zhang, J. Zhang, Y. Ye, Z. Luo, and Y. Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. ArXiv preprint, abs/2403. 13372, 2024d. Utttttss:///axi org/ abs/2403. 13372
K. Zhou, B. Zhang, J. Wang. Z. Chen, W. X. Zhao, Sha, Z. Sheng, S. Wang, and J.-R. Wen. Jiuzhang3.0r Efficiently improving mathematical reasoning by training small data synthesis models. arXiv preprint arXiv:2405. /4365, 2024.
15

---

A Details of Additional Public Instruction Tuning Datasets
OpenHermes 2.5 [Tknium 2023
The OpenHermes-2255 dataset is comprehensive collection of diverse data sources for instruction tuning, including 1M examples from datasets in math, science, and coding, alongside synthetic and chat-based data. It incorporates diverse sources such as Airoboros 2.2, CamelAl Domain Expert Datasets, ChatBot Arena, Collective Cogmition, Evol Instruct, Glaive Code Assistant, GPT4-LLM, GPTeacher, Medical Tasks, MetaMath, SlimOrea, Platyuus, ShareGPT, and Unnatural Instructions GPT4, We remove TheoremQA from Platypus as is one of our test sets.
Code-Feedback [Zheng et al. 2024c The Code-Feedbacl dataset is a multi-turn code generation and refinement dataset, containing 68,000 multi-turn interactions between users, code generation models, and compiler systems. It includes initial user instructions followed by compiler and user feedback. This dataset significantly enhances the model's multi-turn interaction coding ability.
Math-Plus: This dataset combines public datasets such MetaMathQA (395K) Yuu et al. 2023 and Orca-Math (200K) [Mitra et al., 2024]. Both of these datasets are generated by GP1-3.5/4
a using GSM/MATH and other math datasets as the seed data. To further augment the dataset, prompt GPT-4 to rewrite Q-A pairs from MATH training sets, adding an additional 300K examples enhance the challenging problems. The total size of the Math-Ple Plus dataset is 894K examples.
To ensure consisteney and compatibility, we carefully align the format and structure of these additional datasets with WEBINSTRUCT. These supplementary datasets provide rich resource for training models to answer questions and provide explanations across a wide range of topics, enhancing their versatility and applicability in real-world scenarios.
B Details of Evaluation Datasets
To rigorously assess the capabilities of models in reasoning abilities across different domains, we utilize several widely used datasets. Each of these datasets is designed to challenge the models in different aspects of reasoning.
GSM8K [Cobbe et al. 2021]: This test dataset contains 1.32K diverse grade school math problems, intended to test basic arithmetic and reasoning ability in educational context.
MATH (Hendrycks et al.] 2021b]: Comprising 5000 intricate competition-lever problems to evaluate the models' ability to perform complex mathematical reasoning.
TheoremQA [Chen et al.) 2023b]: Focused on applying mathematical theorems to solve advanced problems in fields such as mathematics, physics, and engineering, TheoremQA includes 800 questions that test the theoretical reasoning capabilities.
BIG-Bench Hard (BBH) [Suzgun et al. 2022]: Consisting of 23 tasks previously found challeng- ing for language models from BIG-Bench [Srivastava et al. 2023), BBH contains total of 6511 challenging problems examining the capability of LLMs to solve them.
ARC-C [Clark et al. 2018): : ARC includes questions derived from various geade-level science
exams, testing models' ability to handle both straightforward and complex scientific queries. We
the challenge subset, which contains 1,172 test questions.
GPQA [Rein et al,, 2023]: This dataset provides "Google-proot questions in biology, physics, and chemistry, designed test deep domain expertise and reasoning under challenging conditions. We use the diamond subset containing 198 hard problems.
MMLU-STEM [Hendrycks et al., 2021a]: Spanning 57 subjects across multiple disciplines, MMLU evaluates the breadth and depth of model's knowledge in manner akin to academic and
professional testing environments. We select the STEM subset of MMLU with 3.1 problems.
These datasets collectively enable a comprehensive assessment of language models' reasoning prowess across spectrum of complexity and realism. We further evaluate the models on additional
code generation tasks (including HumanEval [Chen et al. 2021]. MBPP [Austin et al. [221] and
their augmented version [Liu et al. 2024 D, general LLM benchmarks like MMLU (endryycks) et al. 2021a). and chat benchmarks like M'T-Bench [Zheng et al. 2024a AlpacaEval 2.0 [Li et al.
2023b), J, and Arena Hard [Li et al. 2024b demonstrate the generalizability WEBINSTRUCT and
WEBINSTRUCT PLUS more general LLM benchmarks.
16

---

C Related Work
Instruction Tuning. Instruction tuning is crucial for aligning large language models (LLMs) with
end tasks. There are two main types of instruction tuning data: (1) human- written data, such as FLAN [Wei et al. 2022a], TO [Sanh et al. 2022]. J. and SuperNI [Wang et al. 2022]. which assemble
large instruction-uun tuning datasets from existing numan-labbled datasets; and (2) synthesized data, like
Self-Instruct [Wang et al. 2023b], WizardLM [Xu et al. 2023), and GPT4-Alpaca [Peng et al. 2023). J, which create instruction-tune tuning datasets by synthesizing from powerful LLMs like GPT-4 [Achiam jtt al. 2023]. Both types of instruction-tuninn
data have advantages and disadvantages. Human- written data is limited in size due to the high cost and in task diversity because existing numan-labeler datasets mostly focus on few NLP tasks. Although synthesized data can be generated at any scale, the high rate of hallucination can lead to significant quality degradation. Moreover, the diversity of synthesized data is heavily influenced by the seed data. Without a diverse seed dataset, the synthesized data will lack domain coverage.
Compared with Humpback (Li et al. 2023a], we focus on mining naturally existing instruction- response pairs from the web, rather than generating new instructions. This additional extraction step reduces corpus redundancy and enhances overall data quality.
Mathematics Reasoning. In recent years, there has been growing interest in enhancing the mathematical reasoning abilities of large language models (LLMs). Three main approaches have been proposed to improve LLMs' mathematical reasoning skills:
Prompting: Chain-of- www.ooootttoooooooooeeee
(CoT) Nye et al. 2022 Wei et al. 2022b] elicits LLMs'
inherent reasoning ability by demonstrating intermediate reasoning steps. Program-of- -oaimm--f--tuuug thoughts-
prompting (PoT) [Chen et al. 2023a Gao et al.] 2023] utilizes tools to further augment LLMs'
math reasoning abilities. Subsequent work Gou al. 2023 Toshniwal et al. 2024 Yuan et al.
2024 combines CoT and PoT to maximize LLMS' reasoning ability.
Continued Training: Enabling LLMs to solve mathematical problems has been a long-standing challenge. MINERVA [Lewkowycz et al. [022] and Galactica [Taylor et al.) 2022] were pioneers in continued training of I.I.Ms to adapt to scientific domains for math and science reasoning. Open-source models like Llemma (Azerbayev et al. (2023), DeepSeek-Mait (Shao al. [024]. and Intern-Mate Math [Ying et al. 22224 have surpassed MINERVA and Galactica on math benchmarks. These approaches mainly rely using efficient classifier to recall documents from Common Crawl to retrieve a massive high-quality math-related corpus (>100B tokens) to enhance math reasoning.
Instruction tuning aims to enhance LLMs' math reasoning skills by efficiently training on -anaat-n public datasets like GSMSK (Cobbe et al. 2021]. J. MATH [Hendrycks] et al. 202 and MathQA (Amini et al. 2019]. However, these datasets are often insufficient
Instruction Tuning:
size and diversity. Therefore, recent work (Yu et al. 2023 Yue et al., 2023b Toshniwal et al.
2024 Luo et al. 2023 Li et al. 2024a] proposes augmenting them with strong commercial LLMs like GPT-4 [Achiam et al. 2023]. J. These methods can significantly boost LLMs' performance on in-domain math benchmarks but may fall short of generalization. Concurrently, Zhou et al. 2024 explored an efficient way to train a small LLM for math problem synthesis.
Our work combines continued training with instruction tuning exploit the benefits of both ap-
proaches. Specifically, our dataset is recalled from Common Crawl like DeepSeekMath (Shao et al.) 2024). However, due to the significant level of noise in the raw corpus, we utilize a strong LLM to filter and clean the corpus to extract the instruction tuning pairs.
Science Reasoning. In addition to mathematical reasoning, there is growing interest in improving LLMs' general scientific reasoning ability in subjects like physics, biology, chemistry, computer science, etc. Several benchmarks, such as MMLU (Hendrycks al. 2021a], TheoremQA [Chen et al.) 2023b), Sci-Bench [Wang et al.) 2023a]. SciEval [Sun et al. 2023). J. and GPQA (Rein et al. 2023] have been developed to measure LLMS' reasoning ability on tasks beyond math. However, there has been less effort in curating high-quality training data for the science domain. Most datasets, like OpenWebMath [Paster al. 2023), Proof-Pile [Azerbayev et al.) [2023,, and MathPile [Zhang et al.) 2024]. are heavily biased towards mathematics. In this work, we aim to generalize the pre-training data to broader subjects through our newly curated science seed data.
17

---

D Code Generation Results
We report the code generation results of our models and baselines in Table 7
HvmanEval
HumanEval+
MBPP
MBPP+
Average
Average+
Mistral-7B
28.7
23.8
51.9
42.1
40.3
33.0
Gemma-78
26.8
20.1
52.6
43.4
39.7
31.8
Llama-3-88
33.5
29.3
61.4
51.6
47.5
40.5
Gemma-I. Commaa-1.1-71 1aitaa-..--B-Bie L.1-retInttruct
42.7
35.4
57.1
45.0
49.9
40.2
Eistral-BBIInstruee
75.0
70.1
44.7
37.0
59.9
53.6
Llama-3-8B 1ee-3---8B---ttt
61.6
56.7
70.1
59.3
65.9
58.0
Mixtral-8 7B-Instrcct--v..
45.1
39.6
59.5
49.7
52.3
44.7
MAmmo 7B-Plus
72.1
65.9
60.1
50.4
66.1
58.2
MAmmo TH2- 8B-Plus
57.9
48.6
61.9
53.3
MAmmo TH2- 8x -luu
57.9
53.7
68.7
56.9
63.3
55.3
Table 7: Code generation results of different models.
Baseline results copied from the
EvalPlus Liu et al. 2024] leaderboard.
E Impact of Additional Public Instruction Tuning Datasets
In Apppedddix A. we introduce additional public instruction tuning datasets to further boost the model's reasoning performance. Here, we show the three setups of models trained on: 1) WEBINSTRUCT only; 2) Additional Public Datasets only; 3) WEBINSTRUCT Additional Public Datasets (which we first trained on WEBINSTRUCT and then continued training on additional public datasets). The results are shown in Table g
TheoremQA
MATH GSM8K
GPQA
MMLU-ST
BBH
ARC-C
AVG
Mistral 7B Base
WEBINSTRUCT
29.0
36.7
68.4
32.4
62.4
58.6
81.7
52.8
PUBLIC DATASETS
22.6
37.9
83.5
29.3
57.6
62.7
79.9
53.4
WEBIIS.+PUBLLE
29.2
45.0
84.7
36,8
64.5
63.1
83.0
58.0
Mixtral 8x7B Base
WEBINSTRUCT
32.2
39.0
75.4
36.8
67.4
71.1
87.5
58.9
PUBLIC DATASETS
31.3
45.1
85.3
37.4
69.4
73.2
88.1
61.4
PURLIC.
34.1
47.0
86.4
37.8
72.4
74.1
88.4
62.9
Table Impact of additional public instruction tuning datasets.
F Distributions of Website Domains in WEBINSTRUCT
Figure show the distribution of the top websites in WEBINSTRUCT.
18

---

Top 25 Website Usage Distribution
blology stackexchaing. com chemisryy www.eryy.cccecccaag..... .omm www indiablk.com wwrochhngg.cc com socratic.org
physics physics.taccens wtw.ssscecceeeeee.ccoom
enotes.com preafviid.org trenetutors.co com sharemy esson com physe physissfaumm cam
Website
brainmass.com
giaatclubccm weeay.com
www com..o...cco com
studyypol.com com
zbmath-org
everydeycalculltion
study.com
waw brainiy
jiskha com
anowerss.com .com
meth stahssscckkhhhcc hange com coursehero.com
2
10
Percentage Tolal
Figure 7: The distribution of the top websites in our instruction dataset.
G Domain Distribution of WEBINSTRUCT
Figure 8 presents breakdown of the WEBINSTRUCT by subject domains and data sources, providing insights into the composition and diversity of the mined instruction-response pairs. The subject labels automatically annotated using the Allma-3-8BB-Insruuc model, while the distribution between education and forum data is obtained by analyzing the source URLs of the samples. The pie chart reveals that WEBINSTRUCT is predominantly composed of science-related subjects, with 81.69% of the data falling under the broad "Science" category. Within this category, Mathematics takes up the largest share at 68.36%, followed by Physics, Chemistry, and Biology. This highlights the cubblem-oolving and scientific reasoning. The remaining
dataset's strong emphasis on mathematical non-science categories, such Business, Art & Design, and Health & Medicine, contribute the diversity of the dataset. In terms of data sources, the vast majority (86.73%) of the instruction- response pairs come from exam-style questions, while forum discussions make up the remaining 13.27%. This source breakdown indicates that IEBTNST primarily consists well-structured, educational content, supplemented by real-world discussions and inquiries from forums. The diverse subject coverage and the combination of education and forum data enable WEBINSTRUCT to capture wide range of reasoning tasks and problem-solve scenarios.
Humanitics Social Science
Tech & Engineering
1.49%
Health & Medicine
4.72%
Design
Business
10.21% %
Science
Biology 2.17%
Forum: 13.27%
Physics 7..7%
Education: 86.73%
Chemistry 3.77%
Mathematics 68.36%
Science: 81.69%
Figure 8: Breakdown of WERINSTRUCT by subject domains and data sources.
19

---

H Limitations of WEBINSTRUCT
Despite employing a threc-step pipeline to ensure the quality of the mined instruction-respone response pairs, there may still be some noise and inaccuracies in the dataset, mentioned in Figure 6 The extraction and refinement steps rely on the performance of the LLMs used, which may introduce biases and errors. Future work could explore more advanced techniques for data cleaning and validation, such as human-in-the-loop approaches or training a data selection model for filtering. Furthermore, while WEBINSTRUCT covers wide range of subjects, including math, science, and engineering, there may specific subdomains or advanced topies that are underrepresented, such as humanities and other daily chat topics. Expanding the coverage of the seed data to include broader and more diverse resruction-response pairs could further enhance the reasoning capabilities of LLMs in different areas.
I Broader Social Impact
The development of MHmmoTH2 and the INSTRUTUUCT has the potential positively impact edu- cation by providing the community with access to a large-scale, diverse set of instruction-response pairs across various subjects, particularly in mathematics and science. MAmmoTH2 can assist students in their learning process by offering step-by-step solutions, explanations, and targeted feedback, thus enhancing their understanding of complex concepts and problem-solving skills. Furthermore, the dataset and models can serve as valuable resources for educators to create more effective and engaging educational content, assignments, and assessments. However, it important consider potential limitations and biases in the dataset and the model. The quality and accuracy of the instruction responns pairs WEBINSTRUCT may vary. and the model's ' outputs may reflect these inconsistencies. Therefore, it is crucial to employ rigorous data filtering and quality control measures ensure the reliability and trustworthiness of the dataset and the model.
20

---

J Case Study
In this section, we aim review some of the instances in WEBINSTRUCT. The data extracted from the website contains many issues. For example, the most common issue is that the answer is relatively short without any intermediate rationale, which could occur frequently from the quiz website with multi-choice questions. Another common issue is that the extraction destroys important formatting in the document, which leads to unrecoverable errors.
Good Case
Extracted:
21g Question: : Find the distance from the point 2 to the line
37xt 10
= 0.
Answer: : 61
903
Refined:
Question: Calculate the distance from the point ( )) to the line defined by the equation 37
10
21
10
=0 O.
Answer:
To find the distance 'd' from point (zo, yo) line given by Ar By
the formula:
d Amo + Byo Cl
A
B
For the point (2. 9) and the line
10
0, substitute the coordinates into the formula:
10a
37 3
21
d
37
)
Simplifying further, the exact simplified distance 'd' is 61 / 20s. Thus, the correct distance from the point to the line is 61,
90S
Figure A1: In this Refined version, the expression of both the question and the answer is clearer and more precise. employs standardized mathematical symbols and terms, such as representing the coordinates of the point and the equation the line as (No. yo) and Az + By + = 0, respectively. Moreover, it explicitly provides the formula for calculating the distance. Through these revisions, readers can more easily understand how compute the distance from a point to line, making the solution process more transparent and easier to follow.
Good Case 2
Extracted:
Question: : What are the two main ways that minerals are classified? They can be classified by luster, streak, breakage, and hardness.
Answer:
Refined:
Question: What are the primary criteria used for classifying minerals in mineralogy?
Answer: Minerals primarily classified based two main categories: physical properties and chemical properties. The physical properties include luster, streak, crystal form. hardness, density, and cleavage or breakage. Chemical properties involve mineral's chemical composition and reactions with other substances. These characteristies help identify and distinguish different minerals in mineralogy.
Figure A2: Refining the extracted response provides more comprehensive and accurate explanation of mineral classification in mineralogy. By expanding on the initial response to include both physical and chemical properties, the refined version offers a deeper understanding of how minerals are categorized and identified. This enriched content can enhance the model's ability to generate informative responses related to mineralogy.
21

---

Good Case 3
Extracted:
Question: How do you find theasymptotes for Q(z) (2x2)/2>> 5z -62
Answer: vertical asymptotes x=-1, x=6/n horizontal asymptote y=2
Refined:
Question:
How do you determine the vertical and horizontal asymptotes for the function Q(x)
2n
7-5x-6
Answer: To find the vertical asymptotes, we need to identify the values of æ that make the denominator
equal to zero:
- 5- ( 1))
This gives us the vertical asymptotes x For the horizontal asymptote, as æ approaches infinity or negative infinity, the dominant terms in the numerator and denominator are both =² Thus, we compare their coefficients:
x: 6.
Numerator's 's highest degree coefficient: Denominator's highest degree coefficient: 1
The horizontal asymptote given by ratio these cocfficients:
2
Therefore, the vertical asymptotes are at and 6, and the horizontal asymptote is 2.
Figure A3: The refined answer provides a more detailed explanation for finding the asymptotes. includes the steps for solving for the vertical asymptotes by factoring the denominator and determining where it equals zero. It also explains how find the horizontal asymptote by comparing the leading coefficients of the highest-degree terms in both the numerator and the denominator. The use of LaTeX formatting makes the mathematical notation clearer and casier to understand.
Good Case 4
Extracted:
Question: Which the following is not an example of biased language? a. "You are too short reach that." b. "Are you deaf?" c. "Move along, grandpa!" d. think I've been gypped.
Answer: : D. think I've been gypped.
Refined:
Question:
Which the following statements does not exhibit biased language?
a. "You too short reach that."
b. "Are you deaf?"
c. "Move along, grandpa!"
d. "I think I've been gypped."
Answer: The statement that does not exhibit biased language is:
d. "I think been gypped."
Explanation: Biased language refers words or phrases that stereotype, discriminate, show prejudice towards a particular group. In this case, options a, b, and e contain biases towards people's physical abilities (height), hearing ability, and age, respectively. However, "I think I've been gypped" is an informal expression meaning to be cheated or deceived, which, while potentially offensive, does not directly target specific group based attributes like race, gender, disabilities.
Figure A4: The refined version of the input presents the question and answers in clearer, more structured format that improves readability and comprehension. Specifically, using bulleted list for the options makes them distinct and casier to evaluate individually. Moreover, the addition of an explanation provides context and justification for the selected answer, enhancing the educational value of the content. This refined structure and the inclusion of an explanatory note make it superior training example for models, as helps in understanding not just the answer but the rationale behind it, promoting deeper learning and better pattern recognition in language processing tasks.
22

---

Good Case 5
Extracted:
stucked the following challenge: "If the line determined by distinct points (æ1, y1) and (222, y2) is not vertical, and therefore has slope (y2 //)/(22 -1), show that the point- slope form of its equation is same regardless which point used as the given point." Okay, we can separate (ro, yo) from the form to get:
Question:
0(02 ri) - z(y22 yi) = yo - 21) 1o y1)
But how exclude this point (mo, Uo) and leave only x. y/ 3, V1, 22, in the equation? UPDATE: There is solution for this challenge:
(y) y2)a + (x2 21yy 22//1 M22
From the answer found that
va(a - a+1) v1(x x2) = y mi)
but why this is true?
Answer: : Thanks saulspatz, the solution is to simply show that, whether are using (x1, y1) (22, y2) as the given point, equation not change.
So both equations:
- 1/1 = m(( x1)
m(x I2)
reduce to:
(g/1 - -2 + (22 3 yy f2 21y2.
Refined:
Question: Demonstrate that the point-slope form of the equation of line determined by two distinct points (I1, vI) and 82). where the line is not vertical, remains consistent regardless of which point is chosen the reference. Why does the form of the equation change when different reference points are used?
Answer: : To address this challenge, we need to confirm that the point-slope equation a line remains consistent regardless of the point chosen to derive it. Given two points (æ1, y1) and (Ir2, y2) with a non- non-vertical line passing through them, the slope m of the line 12-
22
The point-slope form of line's equation based on point (I11, v1)
y1 = m(( x1)
Similarly, using point (22, y2) gives:
1/2 = m(x - x2)
These can be rewritten and compared to show their equivalence. Expanding both equations: From the first equation: y æı- (x x1) From the second equation: 3/2 12=21 (x x2)
x2-81
z2-81
We simplify these to find common form. Multiplying both sides by (I'2 21) and rearranging terms, find:
(= x1) yı 42) = y(22 21)
This confirms that both forms collapse to the same linear equation, showing the consistency of the point-slope form irrespective of the chosen reference point. Thus, it is demonstrated that the structural form of the equation remains unchanged, ensuring that the mathematical properties of the dee-nefined its slope and points through which it passes-are invariant the choice of the reference point used expression. This uniformity stems from the linearity and the constant slope defined by any two points on a straight line.
The original answer contains an irrelevant introductory sentence, which makes less focused. In contrast, the refined version provides clearer question by explicitly stating the problem of demonstrating the invariance of the point-slope form. The explanation the answer is expanded to show siep-by-step equivalence between the two forms and includes detailed reasoning about why the consistency holds. The added explanation reinforces the understanding of linear equations and the concept of constant slope, making the solution comprehensive and pedagogically valuable.
Figure A5:
23

---

Bad Case 1
Extracted:
Question: : Sort
47 3 3T, 9,
v3 -4
17
v3
Answer:
3T, 4,
v3
Refined:
Question:
Sort the following numbers ascending order: -3 17
1,
47
3
3
V3
7
V5
Answer:
The sorted numbers in ascending order are: -3 -4, 0 V5
3
47
9,
17
7
v3"
The refined example does not enhance the clarity or quality of the information. The original formatting and question structure were already clear and concise, and the transformation into slightly more verbose question-and-ansee and-n-swerr format does not add significant value. Additionally, both versions use proper LaTeX formatting for mathematical expressions, so there is no improvement in readability or technical accuracy. Essentially, the refinement introduces additional words without enhancing comprchension. which could potentially dilute the effectiveness of the model training by focusing on unnecessary stylistic alterations rather than content clarity quality.
Figure A6:
Bad Case 2
Extracted:
Question: Which (AD SRAS) will be affected and which direction (leftward rightward)
will shift when Congress goes to war against Canada?
Answer: When Congress falls into a war with Canada, the aggregate demand curve will be affected by moving to the left. Suppose Congress fights against Canada, then, it is likely that Congress will pass laws that advocate for increased tariff's imports. Spending exports
element of aggregate demand. Thus, its curve shifts rightwards, when its components, which are, spending on consumptions, government spending, or spending on investment increase.
On the other hand, Canada will not do business with the USA as usual when they fall out with Congress. For example, the expenses exports Canada will reduce.
Thus, the aggregate demand will fail increase and shift on the there less consumption from houscholds, falls of investments from firms or when the demand for exports from a country like Canada falls.
Refined:
How does a war between Congress and Canada affect the Aggregate Demand (AD) curve, and in which direction will it shift? Ansver: When Congress declares war against Canada, the Aggregate Demand (AD) curve shifts to the left because Congress will implement increased tariff' on imports. Spending exports, which is component of aggregate demand, decreases due these tariffss. The Short Run Aggregate Supply (SRAS) curve, however, not directly affected by this situation. mainly refleets changes in production costs and the availability resources, which are not directly influenced by international confficts unless they lead to disruptions in the domestic production process. In summary, the AD curve shifts left due decreased exports and potentially lower consumer and business spending, while the SRAS curve remains relatively stable unless there indirect impacts domestic production.
Question:
Figure A7: The refined answer, while more structured and clearer in terms of the specific economic curves discussed, introduces inaccuracies that were not present in the original. Specifically, the refined version inaccurately states that spending on exports decreases due to tariffs, which contradicts the fundamental economic concept that tariff's affect imports rather than exports. Furthermore, the original answer provides a broader analysis of potential economic impacts, such as changes in household consumption and firm investment, which are omitted in the refined version. These omissions reduce the depth and accuracy of the economic analysis provided in response the effects of war on economic curves.

---

Bad Case 3
Extracted:
Question: Is there tool (LaTeX, JavaScript, Mathematica.)) that allows one to draw Venn diagram automatically from subsets relations, egg. C D, would yield:
C+D
(Sorry for the pitiful Ascii drawing)
saaaaangg.coo/qqusston////66/ complement-s-in-latee answers it latex but I couldn't 't get their answer to work in my answer, as it used tikzpicture ( seems unsupported here).
Answer :
to-draw-venn-deee venn diagrams-especialll especially-
Refined:
Question: Is there any software for generating diagrams to show relationships among set members, for example, A C A B?
Answer: One might try using LaTeX, which supports drawing with the TikZ pack- age, although some features may work in certain settings. For more details, check out this link: https//tcx.stackexehhie http://.....ecccee..c//cccstttao
com/qucstions/9688 /hhw-to-drawwween wewn--iiggrannsseeea diagrans--speccilll
complement---in-latee.. If problems persist, updating the software or consulting the community might
help.
Figure A8: The refined example's 's quality becomes bad because it omits most of the specific subset relations provided in the original query, making unclear which relationships need be visualized. The answer shifts the focus towards potential troubleshooting rather than directly addressing the user's need generate diagrams based on specific subset conditions. Moreover, the original question included sample ASCII diagram for clarity, which is entirely dropped in the refined version, removing a helpful virtual context.
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: The paper discusses the limitations of WEBINSTRUCT in Appendix H We acknowledge some noise and inaccuracies in the dataset due to the reliance on LLMs for extraction and refinement, which may introduce biases and errors. We also briefly discuss how address these issues for future work.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texx system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
26

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The paper provides extensive details on the experimental setups, including the datasets used, evaluation benchmarks, and the training details for the models. We describe the data collection pipeline, the training setup with specific parameters, and the evaluation methodology in clear manner. This level of detail is sufficient to allow for the reproduction of the experimental results, thus supporting the main claims and conclusions of the paper.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
27

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The paper submitted the data and code with the paper. We will also release the data and code to the public upon acceptance. Additionally, the paper includes detailed instructions for the experimental setup, dataset preparation, and model training processes.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurIPS code and data submission guidelines (https://nips. cc/
wwwlic/guides/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /nips.ccc/pullc//guu
Cooe/obbeeeeeeeee wetoommmsssonni ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if' applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The document provides detailed information about the training setup, including learning rate, global batch size, maximum sequence length, and the total number GPUs employed for the models. It also mentions the use of cosine scheduler for the learning rate adjustments and specifies the models trained with these settings.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
28

---

Justification: The paper provides enror bars and statistical significance information for experiments. Specifically, includes detailed tables and figures that showcase the performance of various models with appropriate error bars. These visualizations are used to compare model performance across different benchmarks, such as reasoning and code generation tasks, ensuring that the reported results are statistically significant and accurately reflect the models' capabilities
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
The paper provides detailed information about the computing resources used for the experiments. It specifies that all models were trained with 32 A100 GPUs and utilized DeepSpeed with the ZeRO-3 stage for efficient training. Additionally, the paper mentions the learning rates, global batch size, and maximum sequence length used during training. These details are sufficient for reproducing the experiments.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform. in every respect, with the NeurIPS Code of Ethics httppp/://aarrr...cooppo//aaaaaa
Answer: [Yes]
Justification: The paper explicitly addresses ethical considerations in several sections. It ensures that the data used for training models obtained from publicly available sources and does not include any private or sensitive information. The research follows guidelines
29

---

for responsible AI, including the use of diverse datasets to mitigate biases and detailed assessment of model performance to ensure fairnsss and accuracy. Moreover, the methods and results are presented transparently, allowing for reproducibility. Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations their jurisdiction).
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper addresses both the potential positive and negative societal impacts of the research in Appendix I
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
The paper details several safeguards implemented to ensure the responsible release of the models and datasets. For instance, the dataset creation process includes filtering out web pages containing evaluation benchmark data to prevent contamination. Additionally, the extraction and refinement steps involve using strong language models to clean and enhance the quality of the data, reducing the risk of generating biased or low-quality outputs. These measures demonstrate a commitment to data integrity.
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
30

---

Released models that have high risk for misuse or dual use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The paper explicitly mentions and respects the licenses and terms of use for the various datasets and models utilized. includes references to the original sources and MathPile, and Cosmopeida. Additionally, the datasets used for fine-tuning, like OpenHermes and Code-Feedback, are properly cited, ensuring that the terms of use are adhered to.
acknowledges the contributions of existing assets, such as Open
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, tatasstssppperssw/s paperswi thcode. com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The paper introduces new assets, such as the WEBINSTRUCT dataset, which consists of 10 million instruction response
ressrontion-rssoonse pairs harvested from the web. The documentation provided is thorough, detailing the steps for data collection, extraction, and refinement. Examples and case studies are included illustrate the quality and structure of the data. Additionally, the datasets used for fine-tuning, such as OpenHermes 2.5 and Feee-Fee,,ack, are clearly listed and described, ensuring comprehensive documentation. Furthermore, the models developed in the paper, including the MAmmoTH2 and MAmmo'TH T'2--ol models, are planned to be open-sourced, which will include the necessary documentation and resources to facilitate their use by the research community.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
31

---

The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper focuses on extracting and refining Instruction-respnns pairs from existing web data and does not conduct new crowdsourcing experiments or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
The paper does not involve research with human subjects. primarily focuses on the development and evaluation of language models using data harvested from web sources, without any indication of human participant involvement or potential risks related to human subjects.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
32


Reviews:
[{"review_id": "iebDbDTkUZ", "reviewer": "Reviewer_yEcJ", "summary": "This paper proposes a method to synthesize instruction tuning data at scale from the pretraining web corpus. The proposed method first recalls relevant documents from the corpus, and then extracts QA pairs, and finally refines the extracted QA pairs with an LLM. The synthesized instruction data proves to be helpful in enhancing the model’s reasoning abilities compared with instruction tuning data from other sources.", "questions": "LLMs are used in the “extract” and “refine” steps in the proposed pipeline for generating and editing instruction tuning data. Will the choice of LLMs introduce bias into the synthesized data (especially compared with distillation-based methods)?", "limitations": "The authors have discussed the limitations in Appendix H and societal impacts in Appendix I.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe proposed method is novel and effective.\n2.\tThe authors conduct extensive experiments to demonstrate that it’s possible to synthesize tuning data from unsupervised text corpus to build strong LLMs that outperform models trained with data collected in existing paradigms.\n3.\tThe paper is well-written and easy to follow. The code and data are released, which will serve as high-quality resources for research and building strong LLMs.", "weaknesses": "There lacks a discussion and comparison with a related work “Self-alignment with Instruction Backtranslation” (Li et al., ICLR'24) which also synthesizes instruction tuning data from unlabeled corpus.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback on our work's novelty, comprehensive experiments, and clear writing!\n\n> “Lack a discussion and comparison with Humpback [1]”\n\nThanks for the note!  Humpback does not release its implementation, data, and models, which makes the replication and head-to-head comparison difficult. Fundamentally, our approach is different from Humpback: \n- Humpback aims to synthesize instructions by backtranslating the existing documents. We focus on **mining naturally existing instruction-response pairs from the web rather than generating new instructions**. \n- Our additional extraction step significantly makes **the corpus less redundant and improves corpus quality** (see the newly added results in response to **Reviewer k2fV**).\n- The \"Refine\" step further enhances instruction quality. \n\nWe will add more detailed discussions of Humpback in the related work.\n\n> “Will the choice of LLMs introduce bias into the synthesized data (especially compared with distillation-based methods)?”\n\n- Thanks for the question! It's important to note that **our approach is not distillation in the traditional sense**. The LLMs in our pipeline are used solely for extraction and refinement of existing data, not for generating new instructions. MAmmoTH2 essentially learns from a cleaner version of raw web data rather than distilling knowledge from other models. \n- We acknowledge that the choice of LLM could influence the accuracy of extraction. To address this, we **chose two open-source models, Mixtral and Qwen**, known for their strong performance and different training approaches. This diversity helps to balance out potential biases from any single model.\n- Compared to distillation methods, **our approach potentially reduces bias** by **preserving naturally occurring instructions from diverse web sources**, while cleaning and structuring them for more effective learning.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K3uVbPJ6SW", "reviewer": "Reviewer_k2fV", "summary": "This paper proposes an approach to automatically harvest large-scale instruction data from pre-training corpora for reasoning tasks. The main steps include: (1) Recall: training a fastText model to recall relevant documents from the pre-training corpus, similar to DeepSeekMath; (2) Extract: using open-source models with few-shot prompting to extract question-answer pairs from the recalled documents; (3) Refine: prompting open-source models to remove noise, adjust formats, and complete the reasoning process for the extracted question-answer pairs.\n\nUsing this method, the authors harvested 10 million instruction data and trained MAmmoTH2 models. Without relying on closed-source models, MAmmoTH2 achieves excellent performance on various reasoning tasks.", "questions": "See weaknesses.", "limitations": "The authors have addressed most of the limitations. The limitations section can be further improved by referring to the weaknesses part.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- Method: The motivation is clear, and the idea of automatically extracting instruction data from web data is novel, simple, and scalable.\n- Experiments: The experiments and evaluations are comprehensive and achieve good results.\n- Well Written: The paper is very easy to understand.\n- Reproducibility: The authors have open-sourced part of the corpus, models, and evaluation scripts to ensure the reproducibility of the results.", "weaknesses": "1. Effectiveness: I wonder if the WebInstruct approach can further improve the performance of state-of-the-art domain models. For example, DeepSeekMath achieved good results by only training on recalled documents and fine-tuning on high-quality data (MATH: DeepSeekMath-7B-Instruct 46.8% vs. MAmmoTH2-7B-Plus's 45.0%). Moreover, since the models have already been trained on SFT data, comparing only the few-shot performance is not comprehensive enough. I suggest also comparing the performance of the Plus version trained with high-quality \"additional instruction datasets\" for most of the experiments. Consider supplementing the following results:\n   - Recall + Plus: Directly train on the 18M recalled documents and fine-tune a Plus version to verify if the \"extract + refine\" steps have significant benefits.\n   - Recall + Extract + Plus: Directly train on the extracted QA (Fig.5, Extracted QA) with LM/SFT loss and fine-tune a Plus version to verify the benefits of the refine step.\n   - In Fig.5, I also recommend reporting the performance after fine-tuning the Plus version for SFT loss vs. LM Loss.\n\n2. Lack of method details:\n  - For example, the code for the recall stage and the prompts used for extraction and refinement could be included in the repository or appendix.\n  - In Sec. 5.1, I suggest explicitly defining the SFT Loss to help more readers understand it clearly. By \"SFT Loss\", I understand the authors mean \"masking the loss of instruction input\", right?\n\n3. Scalability:\n   - The effectiveness of WebInstruct constructed using small models is unknown for larger models; moreover, this approach is difficult to apply to models with hundreds of billions of parameters due to high inference costs.\n   - During refinement, the model generate missing explanations. Have you observed and quantified the hallucination phenomenon? If present, such incorrect reasoning processes can negatively impact model training, such as increasing hallucination/bias, especially if the corpus is used for larger models.\n\n4. Minor points:\n   - Some citations are missing for baselines in Table 2, e.g., Gemma, Abel, and Rho-1.\n   - How can the WebInstruct approach be extended to more general domains? What other issues need to be addressed?\n   - A concurrent work, Jiuzhang3.0 [1], is quite similar in motivation and method. It would be better to discuss and compare with it. What are the advantages and issues of MAmmoTH2 compared to Jiuzhang3.0?\n \n---\n\n[1] Zhou, Kun, et al. \"JiuZhang3. 0: Efficiently Improving Mathematical Reasoning by Training Small Data Synthesis Models.\" arXiv preprint arXiv:2405.14365 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback on our work's clarity, novelty, and comprehensive experiments!\n\n> “Additional results for the effectiveness of WebInstruct”\n\nWe've included early-stage results using **Qwen-1.5-1.8B** to demonstrate **the usefulness of our \"extraction\" and \"refinement\" steps**:\n\n| Model                                | MATH  | TheoremQA | ARC-C |\n|------------------------------------|----------|-----------|-------|\n| Qwen-1.8B Base               | 10.1    | 11.1      | 50.11 |\n| Recall                                | 11.26  | 12.38     | 49.06 |\n| Recall + Extract                 | 14.82  | 13.25     | 51.19 |\n| Recall + Extract + Refine (WebInstruct) | **17.18** | **14.87** | **53.83** |\n\nThese results clearly show the **benefits of each step in our pipeline** and **align with the motivation of the suggested experiments**. We will include these results in the Appendix.\n\n> “Method details”\n- We'll add the extraction/refinement prompts to the repository and appendix for transparency.\n- We'll explicitly define SFT Loss in Sec. 5.1. Yes, it refers to masking the loss of instruction input.\n\n> “Scalability”\n\nThe core idea of WebInstruct is to **mine naturally existing high-quality instructions from the Web**. Compared with hundreds of billion tokens for continual training, our approach is more **cost-effective**. Compared with traditional SFT datasets with only hundreds of thousands of examples, **WebInstruct with 10M examples is more scalable without requiring any human annotations.\n\n> “Quantified Hallucination”\n\nWe quantified hallucination through **human error analysis in Figure 6**. Our case study reveals that the harvested instruction tuning dataset is generally accurate with a low error rate: **78% of examples improved** after refinement and only **10% introduced hallucinations**. Future work includes developing more advanced methods (e.g., training a filtering/reward model) to select the least hallucinated questions. We will strengthen this discussion.\n\n> “Minor points”\n- We'll add missing citations for baselines in Table 2 (e.g., Gemma, Abel, and Rho-1).\n- We'll discuss potential extensions to more general domains and associated challenges.\n- Thanks for pointing out the concurrent work Jiuzhang 3.0 and we'll discuss the Jiuzhang 3.0! Both methods leverage synthetic data generation to enhance reasoning capability, but we extract and refine naturally existing instructions on the web rather than synthesizing new ones. Our core claim is that high-quality SFT data for reasoning naturally exist on the web, and our contribution is developing a simple yet effective approach to harvesting this data.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GO6bsUWMPw", "reviewer": "Reviewer_Mpo3", "summary": "The paper introduces MAmmoTH2, a novel approach to instruction tuning for large language models (LLMs) by harvesting naturally existing instruction data from the web. The authors develop a three-step pipeline (recall, extract, refine) to collect 10 million high-quality instruction-response pairs without relying on costly human annotation or GPT-4 distillation. Fine-tuning LLMs with this dataset significantly improves performance on reasoning benchmarks. The MAmmoTH2-Plus model, further tuned on public instruction datasets, achieves state-of-the-art results on multiple benchmarks.", "questions": "-  How does MAmmoTH2 compare directly with other methods that use synthetic or human-annotated data in terms of data quality and model performance?\n- What measures were taken to ensure the quality and relevance of the extracted Q-A pairs from the web?\n- How does the model address potential biases in the web-sourced data, and what steps were taken to mitigate these biases?", "limitations": "The authors address some limitations of their approach, such as the dependency on web data quality and the challenges in maintaining the diversity and relevance of the instruction data. However, a more detailed discussion on potential biases introduced by web data and the ethical implications of using such data could strengthen the paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Demonstrates a cost-effective way to collect large-scale, high-quality instruction data from the web.\n- Significant performance gains on reasoning benchmarks, with MAmmoTH2 models outperforming existing models.\n- Comprehensive evaluation across multiple benchmarks, showing robust improvements.", "weaknesses": "- The approach primarily combines existing methods (data recall, extraction, refinement) rather than introducing fundamentally new concepts or techniques.\n- More explicit comparison with prior work is needed to highlight the unique contributions and differences of this approach.\n- The quality and diversity of the collected data heavily depend on the web sources, which may introduce biases or inconsistencies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for positive feedback on our cost-effective approach, significant performance gains, and comprehensive evaluation!\n\n> \"Novelty of approach\"\n\nOur method's novelty lies in **its unique pipeline to mine naturally existing instruction data at scale**, offering a new paradigm for creating large-scale, high-quality instruction datasets without costly human annotation or GPT-4 distillation. To the best of our knowledge, we are **the very first paper to formally understand the effect of automatically scaling up SFT data to see its impact**, especially on reasoning tasks.  Reviewer yEcJ, Reviewer k2fV, and Reviewer Jcxm have acknowledged our approach as novel, simple, and effective.\n\n> \"More explicit comparison with prior work\"\n\nIn Tables 2 and 3, we provide a comprehensive comparison with prior works, encompassing general pre-trained base LLMs, general instruction-tuned models, and reasoning-specific models such as Deepseek Math, Intern-Math, and Rho-1-Math. Notably, we also include a comparison with Llama-3-8B-Instruct, which utilizes 10 million human-annotated instructions. Our model demonstrates superior reasoning performance to these existing models.\n\n> \"Web sources have biases and inconsistencies. What steps were taken to mitigate these biases?\"\n\nWe've taken several steps to mitigate biases and ensure data quality:\n* Using diverse seed data across multiple domains\n* Employing multiple LLMs in the refinement stage\n* Implementing a three-step pipeline (recall, extract, refine) to improve data quality\n\n> \"How does MAmmoTH2 compare directly with other methods that use synthetic or human-annotated data?\"\n* We are pioneers in developing a method for scaling up instruction tuning via data synthesis.\n* Our approach outperforms models trained on human-annotated datasets of similar size (e.g., Llama-3-8B-Instruct) on reasoning tasks while matching performance on general tasks, suggesting comparable or higher quality for certain tasks.\n\n> \"What measures were taken to ensure the quality and relevance of the extracted Q-A pairs?\"\n\nWe use a multi-step process to ensure the quality and relevance of our data:\n* Careful selection of seed data and websites\n* LLM-based extraction of relevant Q-A pairs\n* Refinement step to improve formatting and add missing explanations\n* Human evaluation of a sample set (as shown in Figure 6)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2MgYi6x7tT", "reviewer": "Reviewer_Jcxm", "summary": "The paper proposes a 3-stage pipeline to harvest ex-large-scale instruction data from the pre-training web corpus to enhance LLM reasoning, which involves 1) recalling relevant documents, 2) extracting instruction-response pairs using LLM, and 3) refining the extracted pairs by completing the intermediate reasoning steps using LLM.\n\nThe paper\n\n- proposes an effective pipeline to synthesize large-scale high-quality instruction data, especially for reasonable prompts and reliable answers;\n- empirically validates the effectiveness of scaling up instruction data for reasoning tasks;\n- builds `MAmmoTH2-Plus` models, achieving performance superior to or comparable with previous SotA on various reasoning datasets;\n- provides a ex-large-scale instruction dataset for reasoning tasks, *WebInstruct*, as unique public data resource;\n- conducts extensive ablation studies, providing many insights like:\n  - SFT loss is better for LM loss (at least when evaluated on QA tasks);\n  - refining extracted instruction pairs by completing the intermediate reasoning steps is significantly helpful;\n  - using multiple LLMs to refine the instruction data is usually better than a single LLM;\n  - “Education” data (exam-style) are usually better than “Forum” data (discussion-style) (at least when evaluated on QA tasks);\n  - even benchmarks conventionally thought very relevant might conflict with each other (GSM & MATH in Table 5), implying limited generalization of LLMs.", "questions": "Suggestions:\n\n- The refinement step is important and the current setting can be seen as distillation from strong models (Mixtral-22B×8 and Qwen-72B). The method could be more promising if it could help self-improvement/weak-to-strong generalization. I highly recommend adding experiments of training Mixtral-22B×8 and Qwen-72B or stronger models in future versions.\n\nConfusions:\n\n- Are training data sizes in experiments for Table 5 controlled to be comparable?\n- What does the Data Source “Base“ mean in Table 5?", "limitations": "The limitations of this work are acceptable and the authors point out potential directions to address the limitations for future works.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The scaling effect of instruction data is an important empirical question. The paper is the first to scale instruction data to 10M pairs, showing the feasibility and effectiveness of scaling up instruction data (for reasoning tasks).\n- Synthesis of high-quality prompts and answers is important for further data augmentation but rather under-explored. The paper finds an effective method to synthesize reasonable prompt and relatively reliable answers by harvesting from web corpora.\n- `MAmmoTH2-Plus` models achieve performance superior to or comparable with previous SotA on various reasoning datasets.\n- Extensive experiments are conducted on various base models and especially diverse challenging reasoning benchmarks, instead of easy ones with limited scope (e.g. many benchmarks similar to GSM8K), convincingly validating the method's effectiveness.\n- Many insightful and useful observations in ablation studies (as mentioned in the summary).\n- The paper is generally well written to be clear and detailed.", "weaknesses": "- It might need further consideration about **whether training on *WebInstruct* is compatible with or necessary to be added to existing training pipelines to achieve the best final performance (for reasoning tasks)**. The paper achieves its best performances (`MAmmoTH2-Plus`) with a 2-stage instruction tuning on pre-trained models but doesn’t involve continual pre-training, which should be rather important for models’ reasoning abilities as proved by works like DeepSeek-Math. Pre-training and RL should be out of this work’s scope. But it would be better to further clarify the impacts of 1) continual pre-training, 2) training on *WebInstruct*, 3) final fine-tuning on additional instruction datasets and their combinations.\n  - Table 7 shows the performance on reasoning benchmarks of applying 2/3/2+3 on Mistral-7B/Mixtral-8x7B. But **the comparison might be a little unfair**: the domains of the “Public Datasets” are wider than those of the *WebInstruct* with the code generation dataset *Code-Feedback*, but the benchmarks only involve mathematical and scientific reasoning in natural language, which might underestimate the performance of “Public Datasets”, considering the possible confliction between code generation and reasoning in natural language. It might be better to remove *Code-Feedback* from the “Public Datasets” to compare with *WebInstruct*.\n  - **To consider 1) continual pre-training**, it is impossible to conduct by yourselves, but a possible workaround could be to make full use of the resources provided by DeepSeek-Math: DeepSeekMath-7B is continually pre-trained from DeepSeek-Coder-Base-v1.5. By comparing performances on reasoning benchmarks of applying 2/3/2+3 on DeepSeek-Coder-Base-v1.5/DeepSeekMath-7B and the two models themselves, a more comprehensive study on the impacts of these training stages can be done.\n- Table 7 shows that, for strong Mixtral-8x7B, the gains of adding *WebInstruct* to “Public Datasets” is marginal, implying that **the effect of *WebInstruct* for strong base models might be limited**.\n\n---\n\n# After rebuttal and discussion\n\nThe authors resolved most concerns and validated that MAmmoTH2 can efficiently substitute continual pre-training in the standard SotA pipeline. The limitation is that MAmmoTH2 fails to combine with continual pre-training to effectively push forward the upper limit.\n\nI decide to change my score to 8.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback on our effective pipeline, large-scale instruction dataset for reasoning tasks, many useful insights from extensive experiments.\n\n> \"Compatibility with existing continual-training pipelines and impact investigation\" \n\nWe appreciate this valuable suggestion though continual training is beyond our project's scope. Our focus was on improving reasoning performance through scaled instruction tuning. To address this point, we fine-tuned the Deepseek-Math-Base-7B on WebInstruct and additional public datasets. Results show that **WebInstruct can further significantly improve the DeepseekMath** (it’s already been continue-pretrained on math documents). After fine-tuning additional public SFT data, our model achieves comparable performance on math reasoning and higher performance on other reasoning benchmarks, demonstrating compatibility with existing continual training pipelines. We will add the additional results and discussions in the revision.\n\n|                           | TheoremQA | MATH  | GSM8K | GPQA  | MMLU-S | BBH   | ARC-C | AVG  |\n|---------------------------|-----------|-------|-------|-------|--------|-------|-------|------|\n| Deepseek Math 7B Base     | 25.3      | 34.0  | 64.2  | 29.2  | 56.4   | 59.5  | 67.8  | 48.1 |\n| + WebInstruct               | 30.1      | 38.2  | 70.5  | 33.3  | 59.5   | 61.8  | 76.1  | 52.8 |\n| + Additional SFT            | 31.5      | 45.2  | 80.2  | 35.2  | 60.5   | 62.0  | 76.4  | 55.8   |\n| Deepseek Math 7B Instruct | 23.7      | 44.3  | 82.9  | 31.8  | 59.3   | 55.4  | 70.1  | 52.5 |\n| Mistral 7B Base           | 19.2      | 11.2  | 36.2  | 24.7  | 50.1   | 55.7  | 74.2  | 38.8 |\n| + WebInstruct               | 29.0      | 36.7  | 68.4  | 32.4  | 62.4   | 58.6  | 81.7  | 52.8 |\n| + Additional SFT            | 29.2      | 45.0  | 84.7  | 36.8  | 64.5   | 63.1  | 83.0  | 58.0 |\n\n\n> \"Self-improvement and weak-to-strong generalization\" \n\nWe agree that exploring self-improvement and weak-to-strong generalization would be valuable. We'll consider experiments with Mixtral-22B×8, Qwen-72B, or stronger models in future work.\n\n> \"Public Datasets domains wider than WebInstruct\" \n\nThis is not accurate. We evaluated code generation (HumanEval, MBPP) and general chat benchmarks (MT-Bench, AlpacaEval 2.0, Arena Hard) in Table 3. The additional PLUS data training aims to make our models more general and capable of tasks beyond reasoning.\n\n> “Confusions”\n\nFor Table 5, we train all models with the same steps. We will clarify this.\n“Base\" in Table 5 refers to the base model's performance. We'll make this clearer in the table caption or legend.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors introduce MAmmoTH2, for instruction tuning LLMs by harvesting instruction-response pairs from naturally occurring web data. Their method follows a three-step pipeline with recall, extract, and refine and synthesizes a large dataset of 10 million high-quality instruction pairs without relying on costly human annotations or GPT-4 distillation. The results demonstrate that MAmmoTH2-trained models outperform other LLMs on reasoning benchmarks, achieving competitive performance with fewer resources.

The paper presents a novel, scalable, and cost-effective method for generating instruction tuning data that significantly improves performance in reasoning tasks. While the method combines existing techniques like document recall, QA extraction, and refinement, the integration of these steps in a pipeline to scale instruction data is innovative and addresses the costly reliance on human annotations or large closed-source models. The authors conduct  comprehensive experiments, ablations.

However, while the paper is generally strong, a few areas warrant improvement before publication. These issues do not detract from the paper’s overall value but addressing them would make the work stronger based on the results already presented in the author discussion, if these could be included in the paper to make it much stronger.

Strengths:
1. The pipeline introduced in this paper demonstrates a novel, cost-efficient method for generating large-scale instruction data from naturally existing web sources without human annotation or expensive closed-source model assistance.
  
2. The authors present extensive experiments and ablation studies, clearly demonstrating the effectiveness of each step with recall, extract, refine in the pipeline. The model shows impressive gains in reasoning tasks.

3. The authors implement multiple mechanisms to ensure the quality of the data, including using diverse web sources and performing human evaluations, which show a low hallucination rate (only 10%). The multi-step refinement process, employing different LLMs, also reduces the likelihood of bias.

Suggested Revisions (also discussed by the authors and reviewers):
Firstly, thanking all the reviewers for participating enthusiastically and constructively in suggesting additional experiments to make the paper better.
1. Several reviewers highlighted the need for more explicit comparisons with related works, such as Humpback and Jiuzhang 3.0. While the rebuttal addresses some of these concerns, it would be great to add them into the final draft, especially in terms of effectiveness and scalability.

2. As  asked in the review, further ablation studies comparing models trained only on recalled documents versus those that underwent the full extract and refine steps, adding them to the draft would be fantastic. The refinement step appears to be crucial, but further explanation of how the refined instructions improve performance (e.g., more detailed chains of thought, error correction) would enhance understanding.
Decision: Accept (poster)