MIND: Math Informed syNthetic Dialogues for Pretraining LLMs
Authors: Syeda Nahida Akter
Shrimai Prabhumoye
John Kamalu
Sanjeev Satheesh
Eric Nyberg
Mostofa Patwary
Mohammad Shoeybi
Bryan Catanzaro
Venue: ICLR 2025 Poster (2025)
Keywords: pretraining
mathematical reasoning
synthetic dialogue
LLM
reasoning
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose a novel large-scale and diverse Math Informed syNthetic Dialogue (MIND) generation method that improves the mathematical reasoning ability of LLMs during pretraining.

Abstract: The utility of synthetic data to enhance pretraining data quality and hence to improve downstream task accuracy has been widely explored in recent large language models (LLMs). Yet, these approaches fall inadequate in complex, multi-hop and mathematical reasoning tasks as the synthetic data typically fails to add complementary knowledge to the existing raw corpus. In this work, we propose a novel large-scale and diverse Math Informed syNthetic Dialogue (MIND) generation method that improves the mathematical reasoning ability of LLMs. Specifically, using MIND, we generate synthetic conversations based on OpenWebMath (OWM), resulting in a new math corpus, MIND-OWM. Our experiments with different conversational settings reveal that incorporating knowledge gaps between dialog participants is essential for generating high-quality math data. We further identify an effective way to format and integrate synthetic and raw data during pretraining to maximize the gain in mathematical reasoning, emphasizing the need to restructure raw data rather than use it as-is. Compared to pretraining just on raw data, a model pretrained on MIND-OWM shows significant boost in mathematical reasoning (GSM8K: +13.42%, MATH: +2.30%), including superior performance in specialized knowledge (MMLU: +4.55%, MMLU-STEM: +4.28%) and general
purpose reasoning tasks (GENERAL REASONING: +2.51%).

Published as a conference paper at ICLR 2025
MIND: MATH INFORMED SYNTHETIC DIALOGUES FOR PRETRAINING LLMS
Syeda Nahida Akter?; Shrimai Prabhumoye John Kamalu', Sanjeev Satheesh Eric Nyberg?, Mostofa Patwary', Mohammad Shoeybi', Bryan Catanzaro! NVIDIA', Carnegie Mellon University?, Boston University sa sakter@andrew. cmu.eduee sprabhumoe
ABSTRACT
The utility of synthetic data to enhance pretraining data quality and hence to im- prove downstream task accuracy has been widely explored in recent large lan- guage models (LLMS)) Yet, these approaches fall inadequate in complex, multi- hop and mathematical reasoning tasks as the synthetic data typically fails to add complementary knowledge to the existing raw corpus. In this work, we propose a novel large-scale and diverse Math Informed syNtthetic Dialogue (MIND) gener- ation method that improves the mathematical reasoning ability of LLMS. Specifi- cally, using MIND, we generate synthetic conversations based on Open (OWM), resulting in a new math corpus, MIND-OWM. Our experiments with dif- ferent conversational settings reveal that incorporating knowledge gaps between dialog participants is essential for generating high-quality math data. We further identify an effective way format and integrate synthetic and raw data during pre- training to maximize the gain in mathematical reasoning, emphasizing the need to restructure raw data rather than use it as-is. Compared to pretraining just on raw data, a model pretrained on MIND-OW shows significant boost in mathematical reasoning (GSM8K: +1.442% MATH: +2.30%), including superior performance in specialized knowledge (MMLU: +4.55%, MMI.USTEM: +4.28%% and general purpose reasoning tasks (GENERAL REASONING: +2.51%).
INTRODUCTION
The ability to reason is fundamental element of human cogmition, encompassing our ability to think logically, draw conclusions, and make decisions based on available information (Gendron et al., 2024). Large Language Models (LLMS) have demonstrated remarkable performance across wide range of general reasoning and specialized knowledge tasks. In particular, the improvement of LLMS in solving complex mathematical reasoning tasks (Hendrycks et al., 2021b; Cobbe et al., 202 la) has been significant recent years (Gemini, 2024; Nvidia et al., 2024; OpenAI, 2024).
Strong mathematical reasoning ability heavily relies on the abundance of high-quality, compos- ite, and structured pretraining corpora. An effective mathematical corpus should not only contain relevant content but also be formatted to guide models break down complex problems into smaller sub-pre problemo and solve each part ttep-by-step. ---sstt--ehhacinng the model's ability to process and reason about complex problems (Wei et al., 2022). Prior studies show that structured and well-formatted corpora play a crucial role in enhancing multi-hop and logical reasoning abilities (Cobbe et al., 202 Li al., 2023; Gunasckar et al., 2023), underscoring the importance of well-organized math- ematical datasets in pretraining LLMS.
Curating complex, high-quality structured mathematical data is costly and resource-intensive, largely due to the uneven distribution of high-quality sources. Most advanced models (OpenAI, 2024: Gemini, 2024) are not publicly accessible, and it is unclear how their approach is enhancing math reasoning. To mitigate this challenge, synthetic data generation has emerged cost-effective alternative for creating more balanced and diverse training corpus for pretraining LLMS (Maini et al., 2024; Eldan & Li, 2023; Gunasekar et al., 2023; Shah et al., 2024). However,
scalable, and
Work done during internship NVIDIA

---

Published as a conference paper at ICLR 2025
while these techniques have shown promise in improving general reasoning tasks, their data often lack the step-by-step problem solving structure crucial for multi-hop reasoning and complex math- ematical tasks (Maini et al., 2024), making them sub-optimal for such reasoning.
To address these challenges, we propose MIND, novel approach to generate Math Informed syNthetic Dialogue data at scale. In MIND, we provide a pretrained LLM with web document and explicitly prompt it in a zero-shot manner to generate conversation that (a) decomposes the original context step-bg step into multi-turn conversations and (b) explores each step depth within single turn. As illustrated in Figure 2, MIND generates conversation from a raw text by prompting open-source LLM on seven diverse conversational styles. The generated conversa- tions are refined using heuristic filters and then can be used to pretrain a language model.
CAHA-I
CVM-119
MND-CAM--11 Conoenationa
50.20
99.21
26:38
7:02
0.50
MATH
GSMSK
MMLU-STEM
MAUU
General Reasoning
Figure 1: Continuous pretraining with all styles of conversations MIND-OWM-4B) derived from a small subset (OWM-4B) and a 3.6x large raw corpus (OWM- 14B) reveals that model trained with conversations outperforms the one trained with larger corpus in GSM8K, MMLU and gen- eral reasoning-
MIND demonstrates that transforming raw web text into structured conversations using an off- the-shelf open-source LLM significantly enhances the mathematical and logical reasoning abilities of LLMS compared unstructured raw or rephrased web text. Additionally, MIND provides the flexibility to preserve the diversity of the web corpora and leverage knowledge imbalances between participants for further expansion of the corpora as they either educate each other or collaboratively bridge their shared knowledge gaps through explanation and analysis in a conversation. Moreover, MIND enables the continuous generation of synthetic data from a single document by employing infinite conversational styles, further enriching the diversity. Unlike static text rephrasing (Maini et al., 2024), conversations encourage dynamic reasoning, where participants build on each other's ideas, ask questions, and offer clarifications. This quality makes conversations particularly effective for complex reasoning tasks, as they not only preserve the original information but also expand with new layers of understanding and explanation.
the significance of high-quality structured data over quantity.
In summary, the key contributions of this work are as follows:
We propose novel approach, MIND, to generate structured conversational synthetic data for math reasoning. Leveraging MIND, we produce 64B tokens of synthetic data using 14B tokens from OpenWebMath corpus.
We conduct comprehensive experiments with various conversational styles, altering participant roles to assess their impact on conversation quality and reasoning tasks. Our findings empha- size the importance of the knowledge imbalance between participants in producing high-quality mathematical data.
We scale our approach to higher number of tokens and two math specific datasets, demonstrating its efficacy in large and nigh-quality raw corpus.
We demonstrate an effective way for integrating synthetic and raw data during pretraining to en- hance mathematical reasoning ability of LLMS, emphasizing the importance of carefully reformat- ting raw data to optimize reasoning processes instead of using it in its original form.
In this paper, we evaluate MIND across three dimensions: (1) the effectiveness of each conver- sational style in mathematical reasoning, (2) whether the impact of conversation persist as data scales, and (3) whether MIND remains beneficial when the raw text originates from high-quality sources. Continuously pretraining a 7B LLM on synthetic conversations (MIND-OWM-Bl) gener- ated from a subset of Open Weh (OWM-4B), results in 6.29% average improvement across three mathematical reasoning benchmarks, 4.30% on specialized knowledge tasks (MMLU), and 2.20% boost across zero-shot tasks, compared to the model trained with raw OWM-4B. Additionally, our experiment with entire OptnWebMath (OWM-14B) and its corresponding synthetic conversa- tions shows a consistent trend, indicating that the benefits conversational data continue to hold as the data scales. In fact, with all conversations generated from OWM-4B, can outperform model trained with OWM- OWM-14B, a 3.6x larger tasks, 1.56% across all benchmarks (Figure 1). This underlines the value of synthetic conversations,
average improvement across GSM8K and MATH
2

---

Published as a conference paper at ICLR 2025
particularly when qugh-qualtty in-domain data is limited. Moreover, analysis with other datasets reveals that conversational data further amplifies reasoning capabilities in models even when the raw data originates from nigh-quality sources. We hope that MIND will pave way to improve com- plex reasoning ability of smaller models with limited training data and accelerate further innovation towards building strong reasoning ability with structured high-quality data.
MIND: MATH INFORMED SYNTHETIC DIALOGUE GENERATION
Prompts, P E [1,7]
Raw Data, R
Conversations, Sig E
Language Model,
Browse Guestions e
Hux many numbers ane thare belween 20000 and in which digit wee 6.7.667 each tan
Stedns Stnkect
Student idontt understend problom. Can you ssking? Teacher: The problem usking to find the number
Continuous Pretraining
Language
Deloser:
R
Model, C
Intencine Frebles Sdiie
Heuristic Filteing, 2
Syn D
Syn D
Syn
Ryp
Conversation Generator, MM
Date,S
Evaluate MIND generated dets E + pretrain(C, D); D= (5 R
Generate Synthetic Dialogue Corpora using MIND
MIND
Figure 2: Math Informed Nyhteiii Dialogue. We (a) manually design prompts of various con- versational styles, (b) provide the prompt along with raw context as input to LLM to obtain diverse synthetic conversations. (c) apply heuristic filtering to refine the generated data and (d) observe the downstream task performance after continuously pretraining 7B I.I.M.
To generate high-quality data at scale, current synthetic data generation approach explores rephras- ing texts using LLMS in varied syntax while preserving the core content (Maini al., 2024). How- ever, their proposed approach limits up-sampling high-quality data in way that does not go beyond grammatical styles surface form transformations-leatiee to performance across complex and logical reasoning tasks. We hypothesize that simple rephrasing does not leverage the full potential of the synthetic data to improve the mathematical and complex multi-hop reasoning ability of LLM. Therefore, propose, MIND, conversational synthetic data generation approach that adds semantic variations and structured complexity to the raw text which required to improve complex reasoning ability of the LLMS. In addition, multi-turn conversations can break down the original context step-by-step while each step addresses sub-context at a time by often injecting complimentary reasoning explanations. This resonates with how human solves a complex problem using consecutive chain-of-thous reasoning.
little no improvement when comes
As depicted in Figure given raw dataset R {ri,...rN), we define set of conversational prompts P = (P))).PP)) and utilize a pretrained LLM, denoted as M, for synthetic data generation. We combine raw data rj with a prompt pt and pass it M produce synthetic conversation st.j.
Si.j M(p (1r))
Here, 8i. represents the synthetic data generated by applying prompt p; the raw example ry. For a specific prompt, the total synthetic data generated can be represented as
S = { j E [1} for fixed i E [1, 7]
We further apply heuristic filtering (H) to remove bad generations:
s' = H(S)
Finally, we have a high-quality synthetic dialogue corpus S' which is specifically designed to im- prove mathematical and logical reasoning ability. To summarize MIND:
R MIND S'
3

---

Published as a conference paper at ICLR 2025
To evaluate the effectiveness of S' in pretraining, we conduct continuous pretraining base LLM, C, to minimize the computational costs associated with full pretraining. Our prior experiments on complete pretraining with raw data, R and synthetic data, S' validates that the ranking between models trained on S' or R remains consistent whether we use continuous pretraining or full-scale pretraining (detailed in Appendix B. Moreover, continuous pretraining has emerged tive way to improve performance of LLMS in target domains (Guo et al., 2024; Huang et al., 2023; Chen al., 2023) and even boost their general capabilities (lbrahim al., 2024; Parmar et al., 2024c) with reduced training cost. Given the similar outcomes and significant savings in computa- tional resources, we adopt continued pretraining for evaluating our approach throughout the paper.
effec-
Using S' and a subset of pretraining data (Rpt), the model C is continuously pretrained, yielding an enhanced model E, which possess improved mathematical reasoning capabilities.
f pretrain(C. (C, D); where D {S' U RRp)
2.1 COMPOSE CONVERSATIONAL PROMPTS
To generate conversation using document T; we prompt M in a way that preserves all informa- tion from the original context in the conversation and remains faithful to the context. We manually compose several prompts on diverse conversation setting and topics. We finalize seven prompts (P) featuring conversations between (1) TWO STUDENTS, (2) TEACHER STUDENT, (3) TWO PROFES- SORS, (4) DERATE, (5) PROBLEM SOLVING, (6) LAYMAN KNOWALI, and (7) INTERVIEW which can be found in Appendix A.1. These prompts specifically designed to guide LLM in breaking down the imput context step-by-step, with each step being discussed in depth through explanations
and reasoning.
2.2 GENERATE CONVERSATION
Given unstructured raw text (ry), we instruct the LLM convert the raw text into multi-turn conversation ($t.)) using a prompt (p;) where p; E twwo_studenss, paacher_studentt debate).
Seed Data Selection. The benefit of MIND will get amplified for texts that require step-by-step analysis and chain thought reasoning- the key features of a math corpus. Therefore, we choose Open MethWehMath (Paster et al., 2023) quality mathematical web text.
seed corpus, R, which contains 14.7B tokens of high
Large Language Model.
We use M = LLAMA3-708E -ONNAS--OB--SSTRUCT (AI@ Meta, 2024) generate conversations from raw text, due to its superior performance across a variety of tasks compared to other open-source models. The instruction-tuned version is specifically fine-tuned and optimized for dialogue and chat-based applications.
Generation Configuration. We observe that with increasing context length, conversations tend to lose details from the original texts, as discussed in Appendix C.I. Therefore, for each generation, we iteratively take contexts 500 tokens to obtain accurate and informative conversations. To evaluate the quality of the generated conversations, we test various filtering methods, from simple heuristics to 4L.--based scoring. However, as noted in Appendix C.3, LLM scoring consistently rates all generations highly, making it unsuitable for our approach. Hence, we rely on heuristic filtering to discard bad generations before using them for training.
3 EXPERIMENTAL SETUP
Conversation Generator Configuration. To generate conversation, we consider zero-shot prompting M, where we only pass basic prompt (Appendix A.l) and the raw text. We sample conversations with temperature-1. 1.0 0 and top_ p=0 9 where the total number of input-output tokes is limited to 4096. We use the TenSorRT-LLMM toolkit to deploy large scale generation'.
Pretrained Model Architecture. We train a standard decoder-only Transformer (Vaswani al., 2017) architecture of 7B parameters (C). The framework uses causal attention masks and Rotary
4

---

Published as a conference paper at ICLR 2025
Position Embeddings (RoPE) (Su et al., 2021). Tiktoken tokenizer, SwiGLU (Shazeer, 2020) acti- vations in the MLP layers, and grouped query attention (GQA) (Ainslie et al., 2023). The model consists of 32 layers, 32 attention heads, sequence length of 4096, and a hidden dimension size of 4096. It has no bias terms, has dropout rate of zero, and untied input-output embeddings. The models trained using NVIDIA's Megatron-L.M (Shoeybi al., 2019) repository.
3.1 TRAINING DETAILS
Pretraining Data. Our pretraining data blend comprises of publicly available datasets from 13 snapshots of CommonCrawl (73.37%) (Gao et al., 2020), books/patents (9%), papers (9%), code (5. (5.12%), stack-exchange (2.66%), and Wikipedia (0.8%). Our code data consists of 42 program- ming languages while the other datasets come from various sources including web documents, news articles, scientific papers, and books.
General Pretraining. To prepare a base model, we pretrain 7B LLM on our pretraining data blend till 700B tokens using 512 H100 80GB SXM5 GPUs. During training, we use the Adam optimizer (Loshchilov & Hutter, 2019) with 31 = 0.9, B2 = 0.95 and weight decay of 0.1. We use 2-way tensor and pipeline parallelism to train the model. We set the maximum value of learning rate to 3e-1, minimum to 3c-6, and use batch size of 6M tokens with a 4096 context length.
Continued Pretraining. After pretraining the base model (C) on 700B tokens, we proceed with continuous pretraining using additional 50B tokens to obtain &. To reduce the shift between pre- training and continuous pretraining token distributions (Guo et al., 2024) we create a new data blend (D) for this phase. To ensure the model is exposed to more math tokens, blend D consists of 2:1 ratio Open WebMahh (33B tokens)-eiin either raw (R) or synthetic (S') and 13 snapshots of Com- monCrawl (17B tokens) (Rpt) to maintain consistency with the pretraining blend. To ensure fair comparison, we always keep this token distribution constant in every experiment i.e., every model will a the same amount of tokens from data source regardless of its size. Unlike the pretrain- ing blend, we use a high quality version of CommonCraw/ data (Rnz) filtered by the Eebeuu (Penedo al., 2024) classifier achieve reasonable performance in generative tasks. This Ryp remains constant across all our continued pretraining experiments, while we vary the OpenWeb- Math with or S' or combining both to assess their relative significance. We maintain the same training configuration before and continue pretraining until reaching 50B tokens, using the same pretraining loss objective. In this paper, we use two versions of OpenWeeMath:
OWM-4B: To quickly evaluate the effectiveness of all seven prompts, we take a smaller subset of Open WebMath containing 4B tokens. Synthetic data generated from this subset is labeled as MIND-OWM-4F throughout the paper.
OWM-14B: This version contains the full 14.7B tokens of Open and the synthetic data of this is called MIND-OWN-14B.
3.2 EVALUATION METRICS
To evaluate the zero-shot and few-shot learning capabilities of models, conduct thorough benchmark assessment using a series of datasets using LM Eval Harness (Gao et al., 2024).
General Purpose Reasoning Tasks. This category comprises datasets testing broader cognitive skills and language comprehension. We consider nine standard commonsense and logical reasoning tasks in 0-shot: ARC easy (ARC-E) & challenge (ARC-C) (Clark al., 2018), PIQA (Bisk al., 2020), SIQA (Sap et al., 2019), HellaSwag (Zellers et al., 2019), WinoGrande (Sakaguchi et al., 2021), OpenBookQA (Mihaylov et al., 2018), TruthfulQA (Lin al., 2022). CommonsenseQA (Talmor et al., 2019) and reading comprehension task: RACE (Lai et al., 2017). We report the average results across ten general reasoning tasks under the metric 'GENERAL REASONING'.
Math and Specialized Knowledge Tasks. We consider three diverse math benchmarks to compre- hensively evaluate the mathematical reasoning ability. These benchmarks encompass mathematical challenges from elementary to college level complexity demanding qualitative reasoning (8-shot GSM8K (Cobbe et al., 2021b), 4-shot MATH (Hendrycks et al., 2021c)) and conceptual science and math reasoning (h-shot MMLU-STEM (Hendrycks et al., 2021a)). In the Specialized Knowledge cat- egory, we evaluate on MMLU that spans multiple domains, from professional to academic, testing the model specialized subjects.
5

---

Published as a conference paper at ICLR 2025
4 EXPERIMENTS AND RESULTS
By leveraging MIND with seven conversational prompts and the raw OWM-4B, we generate new corpus 43 billion tokens (All Conversations), Additionally, employing the entire OWM-14B dataset and TWO STUDENTS conversation style, MIND produces an additional 21 billion tokens- resulting in total of 64 billion tokens. This underscores MIND's potential to generate vast amount of high-quality data from relatively limited source material..
Performance across Individual Prompt Style. We observe the effect of each conversation style by generating synthetic data with seven prompts for a smaller subset of Open WehMath, denoted as OWM-4B. To establish baseline, we continue pretraining using D = {RR Rep. where R. E OWM-4B. To further assess the significance of MIND over other synthetic data generation approach, we add another baseline 'Rephrase' introduced by Maini et al. (2024). We generate rephrases with M using the highest performing prompt from their paper maintain consistency among generation quality and training setup. We continuously train C with D where R E Rephrase-Ows Owaassee------B-- In subsequent experiments,
replace R with S' where S' MIND-OWM-4B, corresponding particular conversation style, and repeat the training. To assess the utility of combining multiple conversations, create new dataset by selecting the longest conversation for each context from the seven generated conversations, labeling it as the LONGEST CONVERSATION dataset.
As shown in Table 1, models trained on MIND-generated data of individual styles consistently outperform those trained on rephrased or raw data across all reasoning tasks. Specifically, models trained on synthetic data exhibit significant improvements in mathematical reasoning compared to the baseline, achieving absolute gains ranging from 4.78% to 12.82% on GSM8K, 0,54% to 1 1.28% on MATH, and 0.79% to 4.28% on MMLJ-STEM. thetic data leads to improvements ranging from 1.08% to 4.55%. Furthermore, synthetic data yields an overall enhancement in general reasoning ability, with up to 2% absolute average improvement across the ten reasoning tasks. The LONGEST CONVERSATION delivers the highest gains across all tasks, demonstrating the potential of incorporating multiple perspectives into the training corpus.
specialized knowledge tasks such as MML.U syn-
MMLU- STEM
GENERAL REASONING
Dataset
Style
GSM8K
MATH
MMLU
(Avg)
Avg-AII*
Raw Rephrase
12.96
4.92
39.39
45.91
52.90
29.17
OWM-4B
11.68
5,46
39.71
46.17
53.58
29.22
TEACHER STUDENT
22.74
5.96
40.72
47.93
54.84
32.87
STUDENTS
21.30
6.20
41.90
48.77
54.32
32.65
LAYMAN KNOWALL
17.74
5,46
41,96
48.87
54.89
31.74
MIND-OWM-4B
DERATE
23.96
6.12
40.18
47.61
54.76
33.11
INTERVIEW
20.92
40.53
46.99
54.73
32.12
PROBLEM SOLVING
24.72
6.16
41.36
47.74
54.90
33.38
LONGEST CONVERSATION
25.78
6.30
42.72
49.37
54.86
34.08
Table 1: Results of 7B LLM pretrained on Diverse Conversational Styles. Continuous training with different conversation styles improves all reasoning tasks. Selecting the longest conversation for each raw text further enhances performance in math and specialized knowledge tasks". *Average of GSM8K, MATH, MMLU and General Reasoning.
The disparity between Rephrase and MIND is closely related to the limitations of the rephrasing process. Rephrase adds linguistic variations to the older data, preserving the syntactic meaning of the document, but can not generate semantic/pragmatiic variations. Moreover, rephrases to the information in the raw text and unable to inject new knowledge into the data. As evidenced in our experiments, while rephrasing offers some benefits, it falls short in addressing the deeper, more complex reasoning challenges that conversational data can resolve. The structured and interactive nature of conversations facilitates a more nuanced understanding of the problem space, making an effective approach for improving mathematical reasoning of LLMS.
limited
Analysis with Complete Open Math. Building on the findings from OWM-4B experiments, we establish that all seven conversational styles contribute to significant improvements compared to the raw data, This insight prompted us to explore the effect of increased data in reasoning by scaling
TTo maintain consistency, we use subset of the data (33B tokens) in all experiments. Further breakdown of individual tasks are in Appendix B.2.

---

Published as a conference paper at ICLR 2025
synthetic conversation generation for the complete OWM-14B corpus. To generate data, follow the similar recipe as before and apply only one conversation style to minimize the generation cost. Among the top three highest-performing prompts across all tasks, randomly choose TWo STUDENTS prompt style to generate conversations (MIND-OWM: 14B). We then continuously train C on OwM-14B and MIND-OWM-141 alternatively assess the impact phase, we include another experiment by continuously training C on 50B additional tokens using D { to observe how much gain we can attain across all tasks from math-centric pretraining.
larger data scale. In this
MMLU- STEM
GENERAL REASONING
Dataset
Style
GSM8K
MATH
MMLU
(Avg)
Avg-All
Pretraining Data
9.33
4.74
37.84
45.41
$3.22
28.17
Raw
OWM-14B
20.47
7.24
42.82
49.49
53.95
32.79
MIND-OWM-14R
TWO STUDENTS
27.29
8.24
43.55
49.91
55.54
35.25
Table 2: Results 7B LLM trained on Complete OWM-14B and MIND-OWM-14B: Continuous training of LLM with synthetic conversation outperforms models trained with original pretraining blend and raw Open WebMath across all tasks.
As consistent with previous findings, Table 2 shows that model trained on synthetic conversations is undoubtedly the best for math benchmarks while it also improves overall average for all other reasoning tasks. This underscores that, with data scaling, MIND maintains significant gains in mathematical reasoning while preserving and enhancing performance across other reasoning tasks, including commonsense, factual, and specialized knowledge.
ABLATIONS
Does the Prompt Style matter? From Table observe improvement across all tasks using six conversational styles. However, our experiment with TWO PROFESSORS conversations yield relatively equivalent or worse performance compared to the raw data (Table 3).
MMLU- STEM
GENERAL REASONING
Dataset
Style
GSM8K
MATH
MMLJ
(Avg)
Avg-All
OWM-4B
Raw
12.96
4.92
39.39
45.91
52.90
29.17
MIND-OWM-4B
PROFESSORS
13.50
4.52
37,93
45.25
$3,21
29.12
Table 3: TWO PROFESSORS prompt style Raw data. Continuous pretraining with TWO PRO- FESSORS conversations does not provide gain over raw data compared to other conversational styles.
Rougol.sum
BLEU
This outcome can be attributed to the nature of the TWO PROFESSORS conversation style. Upon reviewing the generated conversations, we hypothesize that the relatively lower per- formance is due the zero-knowledge gap be- tween participants. In this setup, both partic- ipants assume that the other already has suf- ficient knowledge as they are the domain ex- perts, leading surface-level engagement and less detailed discussions.
0.00
Layman
Figure 3: Similarity between Raw Text & Syn- thetic Dialogues. The TWO PROFESSORS style exhibits greater similarity MAN KNOWALL shows the lowest similarity due its richer context with details and explanations.
To further investigate, we measure the BLEU and ROUGE scores between the raw text and the corresponding conversation, as shown in Figure 3, and find that the TWO PROFESSORS style exhibits the highest similarity to raw text. not fully exploit the potential of the generation model to introduce new reasoning breakdowns of complex problems, aligning with our qualitative observation that the professors are not engag- ing in deeper analysis of concepts. This contrasts with other conversational styles where there is either clear knowledge gap between participants (LAYMAN KNOWALL, TEACHER STUDENT, IN- terview), forcing to explain concepts in more depth, both participants, being non-experts
text, while LAY-
This implies that TWO PROFESSORS dialogues do

---

Published as a conference paper at ICLR 2025
actively analyzing and solving the problem (PROBLEM SOLVING, DEBATE, TWO STUDENTS) which results in expanded dialogues with complementary explanations and reasoning. In the latter case, the lack of expertise creates an implicit knowledge gap- sup-instead of one participant being more knowledgeable, both non-experts collaborate to bridge their shared knowledge gap. As depicted in Figure 3, the LAYMAN KNOWALL style, which features the greatest knowledge imbalance between participants, has the lowest BLEU and ROUGE scores. This supports our hypothesis that a larger information gap encourages the knowledgeable participant explain concepts thoroughly, leading to more explicit and detailed conversations.
Relating these insights our findings in Table 1, we see that incorporating explicit knowledge gaps in dialogues is beneficial for MMLU and general reasoning tasks. Conversely, collaborative problem solving, to close the implicit knowledge gap, is crucial for improving performance on math tasks. This highlights a key characteristic of high-quality math data-mere breaking down the problem is insufficient for effective math reasoning. Instead, dynamic knowledge exchange and analysis within the dialogues are essential to achieve maximum improvement in math reasoning.
Does Conversation benefit other datasets? MatMahh used in our current experiments is predominantly collected from mathematical web pages that can contain noisy web contexts. Gen- erating synthetic conversations for such noisy contexts upsamples high-quality data and hence we observe a huge gain in performance with high-quality conversations. Here, we investigate if MIND works on high-quality datasets such as books or papers. We consider new seed corpus, MATHPILE (Wang et al., 2023), that consists of 9.3B tokens extracted from high-quality data sources such as ArXiv papers, textbooks, StackExchange, Wikipedia, ProofWiki, and CommonCrawl pages.
MMLU- STEM
GENERAL REASONING
Dataset
Style
GSM8K
MATH
MMLU
(Avg)
Avg-All
Pretraining Data
9.33
4.74
37.84
45.41
53,22
28.
Raw
MATUPILE
8.79
4.96
42.82
49.49
54.16
29.35
MIND-MATHPILE TWO STUDENTS
12.74
5.74
43.55
49.91
53.98
30.59
Table MATHPILE vs Synthetic Conversation from MATHPILE (MIND-MATHPILE)) Conver- sation generated from high-quality raw data further improves the performance of math tasks.
By employing M, we generate conversations from raw text with the TWO STUDENTS prompt. Later, we replicate the experiments by replacing OWM with MATIPILE and MIND-MATIIPIS cordingly. Table 4 shows that MIND-MATUPILE outperforms the raw counterpart in all three math benchmarks along with specialized knowledge tasks, achieving comparable scores in general rea- soning task. In addition, majority of MATHPILE data is from ArXiV papers and recent work has found this source ineffective improving mathematical reasoning (Shao et al., 2024). We observe a similar trend, where non-math focused pretraining corpora yields better GSM8K score than raw MATHPILE corpus. However, our synthetic conversation on MATHPILE rather amplifies the quality of the corpus resulting in 3.95% absolute improvement GSM8K in comparison with raw data. This highlights the superior structured complexity conversations, which proves particularly effective for multi-hop and mathematical reasoning tasks, over high high-quality data from Arxiv papers.
Is replacing with Synthetic Data the best option? Our findings in Table 2 indicate that com- pletely replacing OpenWebMath with synthetic data yields the best performance across benchmarks. However, Maini et al. (2024) emphasizes the importance of combining real data and synthetic rephrases to achieve consistent improvements across a broader range of tasks-a similar trend we observe in our experiment with rephrased data, as shown in Table 5. To investigate this further, we conduct experiments with four data combinations using OWM-4B while the Rpp remains constant:
OWM-4B + MIND-OWM-4B [1:1]. We combine R and S' in a 1:1 ratio, ensuring an equal number of tokens to be seen during pretraining from both sources. For the synthetic data, utilize the LONGEST CONVERSATION, as this shows the most improvement across tasks (Table 1).
OWM-4B + MIND-OWM-4B [Concat]. We concatenate each raw context with all seven synthetic conversations sequentially.
MIND-OWM-4B [Longest Conversation]. From the seven conversations generated for each con- text, select the longest conversation in token count.
8

---

Published as a conference paper at ICLR 2025
MIND-OWM-4B [All Conversations]. This data incorporates all conversation across all styles.
GENERAL REASONING
Dataset
GSM8K
MATH
MMLU-STEM
MMLU
(Arg)
Avg-AIl
OWM-4B
4.92
39.39
45.91
52.90
29.17
OWM-14B
20.47
7.24
42.82
49.49
53.95
32.79
Rephrase-OWM-IB
11.68
5.46
39.71
46.17
53.58
29.22
OWX--BBRRpprasc-oww-BB [1:1]
14.25
6,20
42.31
48.74
53.68
30.72
OWM-4B-MIND--WWM-BBB [1:1]
21.68
6.14
42.56
49.57
54.50
32.97
OWX-BB-MIND-OW---- [Concat]
24.49
6.22
43.67
50.46
55.10
34.07
MIND-OWM-4B [Longest Conversation]
42.72
49.37
54.86
34.08
MIND-OWM-4B (All Conversations]
26.38
7.22
42.53
50.21
55.41
34.80
Table 5: Comparison of 7B L.L trained with and combination of synthetic data. Synthetic conversation outperforms raw data all combinations. Specifically, combinations of all conversa- tions generated from OWM-4B surpasses the performance of OwM-14B (3.6x larger corpus) across all tasks, underscoring the superior quality and diversity of the conversations.
Our finding in Table 5 indicates that combinations provide substantial boost performance across all tasks. However, for math-centric benchmarks (GSM8K and MATH), training solely with synthetic conversations clicits the best improvements. This is likely as these tasks require complex and multi-step reasoning and conversations are designed to replicate these type of thinking. In parallel, having both raw data and conversation beneficial for specialized and general purpose reasoning tasks, aligning with the findings in Maini et al. (2024). Since synthetic data tends to remove special tags, styles, and code indentations, the inclusion of raw data helps improve the generalizability of LLMS across diverse domains. Additionally, to measure the maximum gain can achieve from conversations for a limited data, we continuously train C with all synthetic dialogues generated from OWM-4B. As shown in Table using conversations generated from OWM-4B, we can outperform the model trained with 3.6x bigger corpus (OWM-14B) on GSM8K, MMLU and general reasoning tasks while showing comparable performance on other tasks. Inspired by this, we further compare MIND with DEEPSEEKMATH (Shao et al., 2024) that extract 1 120B unique math tokens from CommonCrawi (Appendix C.4). The results from Table 14 demonstrate that diverse conversations from MIND based small seed corpus can yield comparable math accuracy to the DEEPSEEKMATH model. This illustrates the potential to enhance reasoning with limited data by generating synthetic conversations of infinite styles.
Does the improvement persist with smaller M? In the previous experiments, we used a constant M, a powerful instruction-tuned model. However, it remains unclear whether the improvements in downstream reasoning tasks stem from the quality of the generated dialogues or are primarily due model distillation from the powerful LLM. To asses the impact of M on the downstream task performance, we re-run MIND with smaller M=LLAMA3-888 -AA-MAA--B--AA -N-INTRUCT on PROBLEM SOLVING style, the best performing style in Table I and continuously pretrained 7B LLM following the training setup in Section 3.1.
MMLU- STEM
GENERAL REASONING
Dataset
M
GSMBK
MATH
MMLU
(Avg)
Avg-All
OWM-4B
12.96
$2.90
29.17
LLAMA3-8B INSTRUCT
22.37
5.72
41.36
48.48
$5.21
32,95
MIND-OWM-4B
LLAMA3-70B. -N---RISTICCT
24.72
6.16
41.36
47.74
54.90
33.38
Table 6: Results of 7B LLM trained on MIND-OWM-4B using M of different sizes: Regardless of the sizes of M, model trained on MIND-OWM-4P outperforms the one trained with raw data.
As shown in Table 6, even with smaller M. the MIND-generatie data provides significant boost in math and general reasoning abilities compared to the raw/rephrased data. This demonstrates that the gains not solely dependent on the capabilities of the larger M but largely driven by the quality and structure of the MIND-generated dialogues. Additionally, regardless of model size and method of synthetic data generation, all LLM-generated synthetic data involves some form of knowledge distillation. However, we demonstrate an effective distillation approach that significantly enhances the reasoning ability of LLMS compared existing approaches (Maini al., 2024).

---

Published as a conference paper at ICLR 2025
6 RELATED WORKS
Mathematical Data Curation. Selecting high quality data for pretraining LLMS is essential for producing state-of-the-art large language models (Brown et al., 2020; Chowdhery et al., 2023; Par- mar et al., 2024a;b; Rae et al., 2021; Feng et al., 2024). Several mathematical datasets have been introduced in recent years (Paster ct al., 2023; Wang et al., 2023; Azerbayev et al., 2023a; Welleck et al., 2021) which have been carefully collected from the web using different heuristics. OpenWeb- Web- Math contains 14.7B tokens of mathematical web pages filtered from CommonCraw/ based on math strings, I and a math document classifier. Building on this corpus, DEEPSEEKMAN MTH (Shao et al., 2024) trains fastText (Joulin, 2016) classifier to further extract mathematical docu- ments from CommonCrawl. They cluster the extracted documents based on the URL domain and label domain math-related where over 10% of the web pages have been collected are classified as math content. Finally, web pages linked to these URLs. yet uncollected. will be added to the seed corpus which will be used to retrain the fastText classifier fetch diverse math contexts. MATH- PIL.E (Wang et al., 2023), multi-source corpus (8.9B tokens), has been aggregated from textbooks, Wikipedia, ProofWiki, CommonCrawl, StackExchange, and arXiv, with the majority (over 85%) sourced from high quality data source arXiv. Although these datasets can effectively capture the diverse mathematical information from web, it is difficult to detect and filter out noisy dataset. Re- cently, many powerful models (OpenAI, 2024; Jiang et al., 2023; Gemini, 2024; Anthropic, 2024; Team, 2024b), in addition to not open sourcing their data, are also refraining from disclosing de- tailed information about their corpus. For the open- source community, constructing high-quality and diverse pretraining corpora is a crucial factor in bridging the performance gap with closed-source models which is the main objective of our work.
Synthetic Math Data. Generating synthetic math data using LLMS has been widely explored in recent days (Trinh et al., 2024; Li et al., 2024; Gunasekar al., 2023; Madaan al., 2024; Patel et al,, 2024; Toshniwal et al., 2024) specifically during alignment using supervised fine-tuning (SFT) (Taori et al., 2023). Some of the latest approaches focus on generating data from seed problems. For instance, Yu et al. (2023) rewrites existing benchmark questions from multiple perspectives using LLMS to create new mathematical problems, while Huang et al. (2024); Shah et al. (2024) leverage GPT-4 to extract topics and key points from seed samples and recombine them into new questions. To further improve diversity, Chan et al. (2024) uses GPT-4 to generate questions and answers at scale, incorporating over one million personas. Previous approaches to generate synthetic data is primarily designed for fine-tuning rather than pretraining, distinguishing it from our effort. Similar to ours, Dai et al. (2022) converts documents into dialogues by predicting unobserved questions without altering the original document. However, MIND expands knowledge by adding comple- mentary reasoning and explanations, leveraging diverse conversational styles to enhance reasoning and enrich diversity, which is infeasible with Dai et al. (2022). In the context of pretraining, re- cent works have generated synthetic datasets (Gunasekar et al., 2023; Li et al., 2023) train smaller language models that demonstrate equivalent performance as the larger models on certain mathemat- ical benchmarks. However, these methods remain largely opaque, costly, and reliant on proprictary models to produce billions of tokens. Additionally, such data generation can be biased towards specifically generating data related to tasks that we want to perform well on. In contrast, MIND provides a feasible alternative to upsample high quality structured data from diverse web contexts, that embeds multi-step and chain-of-though reasoning, using an off-the-shelf open source LLM.
7 CONCLUSION
In this paper, we focus on improving the mathematical reasoning abilities of open-source LLMS. We propose a simple approach to generate complex and structured data at scale, called MIND, that produces a new conversational synthetic math corpus, MIND-OWM, using an off-the-shelf open- source I.M.. Models trained on MIND-OWM, corpus generated through our approach, consistently outperform those trained on raw data, achieving up to a 6.29% improvement across mathematical reasoning benchmarks and outperforming models trained on 3.6x larger datasets. Importantly, these gains persist across ceneral-purpor reasoning tasks and when scaling up the data, highlighting the versatility of synthetic conversations. This work demonstrates the potential of structured conver- sational data to enhance reasoning, especially in cases where domain-specific high-quality data is limited, paving the way for more effective and resource-efficient pretraining of L.I.Ms.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
AI@ Meta. Llama 3 model card. 2024. URL https//gttthh.ooo/mreeell/////at MODEL_CARD.mn
Joshua Ainslie, James Lee-Thorp. Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. GQA: Training generalized multi-query transformer models from multi-head check- points. Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 4895-4901, Singapore, December 2023. Association for Computational Linguisties. doi: 10. 10018655/112223.emmml main 298. URL http://aalaanbooooggoo/222...mm wet.:mmmmimmm.mm.mmm main.298.
Authropic, The claude model family: Opus, sonnet, haiku, 2024. URL URL https::/wwwwwcd...tt. antthroopi......cc.............. wumdde/sshbbll911bc0aaaaccce0000 (0t.//bboo661887777////oo6 Catidddeel_ara___eeee Claude_3.pde
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, al. Program synthesis with large language models. arXiv preprint urXiv:2108.07751 2021.
Zhangir Azerbayev, Bartosz Piotrowski, Hailey Schoelkopf, Edward W Ayers, Dragomir Radev, and Jeremy Avigad. Proofnet: Autoformalizing and formally proving undergraduate-Ievel mathemat- ics, arXiv preprint arXiv:2302.122433 2023a.
Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Al- bert Q Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics. arXiv preprint arXiv:2310. 10631, 2023b.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical com- monsense natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranay Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information P Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL https//proceedings.nrripp.. 1atpat/8ac4142f44a.. 22ff64a-Paeer.ppp
Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, and Dong Yu. Scaling synthetic data creation with 1,000,000,000 personas, 2024. URL https://axxixorr/////2266..00000
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan. Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda. Nicholas Joseph, Greg Brockman, al. Evaluating large language models trained on code. arXiv preprint vrXiv:2107.03333 2021.
Zeming Chen, Alejandro Hernández Cano. Angelika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi, Matteo Pagliardini, Simin Fan, Andreas Köpf. Amirkeivan Mohtashami, et al. Meditron-70b: Scaling medical pretraining for large language models. arXiv preprint arXiv:2311.16079. 2023.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240): 1-113, 2023.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018. URL https://annii.orr/a////8....3.. 803.05457.
11

---

Published as a conference paper at ICLR 2025
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakamo, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint 0rXv:2110.14168, 2021a.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint 0rXiv:211014466. 2021b.
Zhuyun Dai, Arun Tejasvi Chaganty, Vincent Zhao, Aida Amini, Mike Green, Qazi Rashid, and Kelvin Guu. Dialog inpainting: Turning documents to dialogs. In International Conference on Machine Learning (ICML). PMLR, 2022.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Ilama 3 herd of models. arXiv preprint arXiv:240..21788 2024.
Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english?, 2023. URL https//aarvv.or//bx22200..75..
Steven Feng. Shrimai Prabhumoye, Kezhi Kong, Dan Su, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Maximize your data's potential: Enhancing Ilm accuracy with two-phase pretraining. arXiv preprint ar2iv:2412.15285. 2024.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.0000 2020.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang. Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024. URL https://eenndooorr/rrooosssss//s
Gemini. Gemini: A family of highly capable multimodal models, 2024. URL https://arxxv.org//b// 1312.11805.
Gael Gendron, Qiming Bao, Michael Withrock, and Gillian Dobbie. Large language models are not strong abstract reasoners yet. In ICLR 2024 Workshop: How Far Are We From AGI, 2024. URI.
Suriya Gunasekar, Yi Zhang. Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorn, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li. Textbooks are all you need, 2023. URL https://ariiv.rr/ab//2300 11644.
Yiduo Guo, Jie Fu, Huishuai Zhang, Dongyan Zhao, and Yikang Shen. Efficient continual pre- training by mitigating the stability gap. arXiv preprint arXiv:2406. 14833, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the Interna- tional Conference on Learning Representations (ICIR), 2021a
Dan Hendrycks. Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. J. Van- schoren and S. Yeung (eds.), Proceedings of the Neural Information Processing Systems Track htt.://aassttttteeeecncnsaneeeee
17 b0a178361 Paper- on Datasets and Benchmarks, volume 1, 2021b. URL - roff...........ff. neurips.cc/paper_ httppeperwlleeeppppe2221211111
Dan Hendrycks. Collin Burns. Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021c.
12

---

Published as a conference paper at ICLR 2025
Quzhe Huang. Mingxu Tao, Chen Zhang, Zhenwei An, Cong Jiang. Zhibin Chen, Zirui Wu, and Yansong Feng. Lawyer llama technical report. arXiv preprint arXiv:2305. 2023.
Yiming Huang, Xiao Liu, Yeyun Gong, Zhibin Gou, Yelong Shen, Nan Duan, and Weizhu Chen. Key-point-driven data synthesis with its enhancement on mathematical reasoning, 2024, URL
http:://ariivorr//a/224......22
Adam Ibrahim, Benjamin Thérien, Kshitij Gupta, Mats Leon Richter, Quentin Gregory Anthony, Eugene Belilovsky, Timothée Lesort, and Irina Rish. Simple and scalable strategies to continually pre-train large language models. Transactions on Machine Learning Research, 2024. ISSN 2835-
8856. URL http:///ppoeeeiiioeeo//ooooo//nn
Mojan Javaheripi. Sébastien Bubeck, Marah Abdin, Jyoti Aneja, Sebastien Bubeck, Caio César Teodoro Mendes, Weizhu Chen, Allie Del Giorno, Ronen Eldan, Sivakanth Gopi, et al. Phi-2: The surprising power of small language models. Microsoft Research Blog, 1:3, 2023.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample. Lucile Saulnier, Lélio Renard Lavaud, Marie-Ane Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang. Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https://arxiv.org// org//b/ 2310.06825.
Armand Joulin. Fasttext. zip: Compressing text classification models. arXiv:1612.03661, 2016..
arXiv preprint
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. RACE: Large-scale ReAd- ing comprehension dataset from examinations. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel (eds.), Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 785-794, Copenhagen, Denmark, September 2017, Association for Computa- tional Linguistics. doi: 10. 10.18653/
URL https//accanttooog..oo///11111
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo. Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with lan- guage models, 2022a. URL httpss//arxiv.orr/bs//2000 14858.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857, 2022b.
Chen Li, Weiqi Wang, Jingeheng Hu, Yixuan Wei, Nanning Zheng, Han Hu, Zheng Zhang, and Houwen Peng. Common 7b language models already possess strong math capabilities, 2024. URL https://ariiv.or//as//40... 047006.
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lec. Textbooks are all you need ii: phi-1.5 technical report, 2023, URL https://aniivorr/ab///000 05463.
Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2214-3252, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v 002653//1/2022..222 wwl.02..clllonn..... URL http:://acaatthloggoog////2//// acl- long.
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chat- gpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36, 2024.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- on Learning Representations, 2019. URL
13

---

Published as a conference paper at ICLR 2025
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self- feedback. Advances in Neural Information Processing Systems, 36, 2024.
Pratyush Maini, Skyler Seto, He Bai, David Grangier, Yizhe Zhang, and Navdeep Jaitly, Rephrasing the web: A recipe for compute and data-efficient language modeling. In Data Problems for Foundation Models Workshop at ICLR, 2024. URL http://armiv.org/abs/240.....
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? new dataset for open book question answering. In EMNLP, 2018.
Nvidia, :, Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H. Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan Cohen, Sirshak Das, Ayush Dattagupta, Olivier Delalleau, Leon Derezynski, Yi Dong, Daniel Egert, Ellie Evans, Alek- sander Ficek, Denys Fridman, Shaona Ghosh, Boris Ginsburg, Igor Gitman, Tomasz Grze- gorzek, Robert Hero, Jining Huang, Vibhu Jawa, Joseph Jennings, Aastha Jhunjhunwala, John Kamalu, Sadaf Khan, Oleksii Kuchaiev, Patrick LeGresley, Hui Li, Jiwei Liu, Zihan Liu, Fileen Long. Ameya Sunil Mahabaleshwarkar, Somshubra Majumdar, James Maki, Miguel Martinez, Maer Rodrigues de Melo, Ivan Moshkov, Deepak Narayanan, Sean Narenthiran, Jesus Navarro, Phong Nguyen, Osvald Nitski, Vahid Noroozi, Guruprasad Nutheti, Christopher Parisien, Jupin- der Parmar, Mostofa Patwary, Krzysztof Pawelec, Wei Ping, Shrimai Prabhumoye, Rajarshi Roy, Trisha Saar, Vasanth Rao Naik Sabavat, Sanjeev Satheesh, Jane Polak Scowcroft, Jason Se- wall, Pavel Shamis, Gerald Shen, Mohammad Shoeybi, Dave Sizer, Misha Smelyanskiy, Felipe Soares, Makesh Narsimhan Sreedhar, Dan Su, Sandeep Subramanian, Shengyang Sun, Shub- ham Toshniwal, Hao Wang. Zhilin Wang, Jiaxuan You. Jiaqi Zeng. Jimmy Zhang, Jing Zhang, Vivienne Zhang, Yian Zhang, and Chen Zhu. Nemotron-4 340b technical report, 2024. URL https//arxiv.org/ab//2400 1704.
OpenAl. Gpt-4 technical report, 2024. URL http:///axiv.orgaaa22333...74
Jupinder Parmar, Shrimai Prabhumoye, Joseph Jennings. Bo Liu, Aastha Jhunjhunwala, Zhilin Wang, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Data, data everywhere: A guide for pretraining dataset construction. In Proceedings the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 10671-10695, 2024a.
Jupinder Parmar, Shrimai Prabhumoye, Joseph Jennings, Mostofa Patwary, Sandeep Subrama- nian, Dan Su, Chen Zhu, Deepak Narayanan, Aastha Jhunjhunwala. Ayush Dattagupta, et al. Nemotron-4 15b technical report. arXiv preprint arXiv:2402. 16819, 2024b.
Jupinder Parmar, Sanjev Satheesh, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Reuse, don't retrain: A recipe for continued pretraining of language models, 2024c. URL. https: //arxiv.rrgaab//2007.7722 0776666........
Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. Openwebmath: An open dataset of high-quality mathematical web text, 2023.
Ajay Patel, Colin Raffel, and Chris Callison-Buree eration and reproducible LLM workflows. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguis- fies (Volume 1: Long Papers), pp. 3781-3799, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL. http:://allmmboooog..gg/........
DataDreamer: A tool for synthetic data gen-
Guilherme Penedo, Hynek Kydliček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale, 2024. URL https://arii..rr//b//444..... 17557.
Jack Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:21 1446, 2021.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. arXiv preprint arXiv: 1907. 10641, 2019.
14

---

Published as a conference paper at ICLR 2025
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications of the ACM, (4(9):99-106 2021.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Common- sense reasoning about social interactions. In Kentaro Inui, Jing Jiang. Vincent Ng, and Xiaojun Wan (eds.). Proceedings of the 2019 Conference on Empirical Methods Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pp. 4463-4473, Hong Kong, China, November 2019. Association for Computational URL http://acaamhoolgyorg/////94
Linguistics. doi: 10. N0.18633v1//D19-111
Vedant Shah, Dingli Yu, Kaifeng Lyu, Simon Park, Nan Rosemary Ke, Michael Mozer, Yoshua Bengio, Sanjeev Arora, and Anirudh Goyal. Ai-assisted generation of difficult math questions, 2024. URL https//artix.or///bh/440.......
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathe- matical reasoning open language models, 2024. URL https://arrivorrgab///2200.....0
Noam Shazeer. Glu variants improve transformer, 2020. URL https://axiivorgaas22000.0222220
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training multi-billion parameter language models using model par- allelism. arXiv preprint arXiv: 0900//99900000,,.. 2019.
Robyn Speer, Joshua Chin, and Catherine Havasi. Conceptnet 5.5: An open multilingual graph of general knowledge, 2017. URL
wwwp/esiiiiccpppphAAAAAAAAAAAAA 14977
Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. 2021.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQQ:: A ques- tion answering challenge targeting commonsense knowledge. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers). pp. 3149-4155 Minneapolis, Minnesota, June 2019. Association for Compu- tational Linguisties. doi: 10. 1e.11855//1/1111110 1421. URL https//accaathoolgg.ogg//// 1421.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An enstrucioon-oolloimn llama model. alpaca, 2023.
Gemma Team. Gemma: Open models based on gemini research and technology. 2024a. URL http:://axivoorrabb/2223.8.999
Qwen Team. Qwen2.5: A party of foundation models, September 2024b. URL https://qwwnlm. githubioo/hlog//wen2.5//
Shubham Toshniwal, Ivan Moshkov, Sean Narenthiran, Daria Gitman, Fei Jia, and Igor Gitman. Openmathinstructt-I: 1.8 million math instruction tuning dataset, 2024, URL https://arxiv.or/ abs/2402.10076..
Trieu Trinh, Yuhuai Tony Wu, Quoc Le, He He, and Thang Luong. Solving olympiad geome- try without human demonstrations. Nature, 625:476-483 2024. URL https:/wwww.nature.et 14Milles/4118860003-0077771
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gamett (eds.), Advances in Neural Information Processing Systems. volume 30. Cur- ran Associates, Inc., 2017. URL https//ppoceednnss.uuri...cc/ / 9eee244477ddee1fdddss 1c405 14a85511a--
15

---

Published as a conference paper at ICLR 2025
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou, consistest improves chain thought reasoning in language models. arXiv preprint arXh::2203.11170 2022.
Zengzhi Wang, Rui Xia, and Pengfei Liu. Generative ai for math: Part i mathpile: A billion-token- scale pretraining corpus for math. arXiv preprint arXiv:2312.171222 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le. Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 2-248724-24877, 2022.
Sean Welleck, Jiacheng Liu, Ronan Le Bras, Hannaneh Hajishirzi, Yejin Choi, and Kyunghyun Cho. Naturalproofs: Mathematical theorem proving in natural language. arXiv preprint arXiv:2104.01112, 2021.
Longhui Yu, Weisen Jiang. Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James Kwok, Zhen- guo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your mathematical questions for large language models. arXiv preprint arXiv:2309. 12284, 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
A PROMPTS AND DATASETS
A.1 PROMPTS FOR CONVERSATION
Two PROFESSORS
Convert the context above as a multi-turn discussions between two professors. Make sure that their discussions strictly adhere to the context above and remains faithful to information the context. Please DONOT add any new informmation/referrnce other than the context.
TEACHER STUDENT
Convert the context above as multi-turn discussions between teacher and student. The student has questions about the context and the teacher solves each of them step-by-step. Make sure that their discussions strictly adhere to the context above and remains faithful to information in the context. Please DONOT add any new information/reference other than the context.
Two STUDENTS
Convert the context above as multi-turn discussions between two students who are working on their assignment related to the given context. Make sure that their discussions strictly ad- here to the context above and remains faithful to information in the context. Please DONOT add any new informattion/rfferencc other than the context.
INTERVIEW
Conduct an interview-style conversation where one participant acts as the interviewer, asking questions exclusively related the content provided, while the other participant serves as the subject matter expert, providing detailed responses based on the content. Make sure that their discussions strictly adhere the context above and remains faithful information in the context. Please DONOT add any new informaation/rfecrence other than the context.
PROBLEM SOLVING
16

---

Published as a conference paper at ICLR 2025
Convert the context above multi-turn problem-solving conversation where participants analyze challenges or scenarios presented in the content and brainstorm solutions within the context of the provided material, avoiding speculation sure that their conversation strictly adhere to the context above and remains faithful to in- formation in the context. Please DONOT add any new informatiio//reeernnce other than the context.
unrelated discussions. Make
LAYMAN KNOW-ALL
Imagine you are presenting the content above step-by-step to layman. While you are presenting, the layman has a lot of followup questions regarding your presentation. You answer the questions step-by-step with chain-of-thoughts. Design this interaction between you and the layman strictly adhere to the context above and remains faithful to information the context. Please DONOT add any new information//rffrrnee other than the context.
multi-turn conversational manner. Make sure that the interaction
DEBATE
Convert the context above multi-turn debate-style conversation where the participants present arguments and counterarguments based solely on the content provided, without in- troducing external information personal opinions. Each participant defends others argu- ments step-by-step with chain-of-thoughts. Make sure that the conversation strictly adhere the context above and remains faithful to information the context. Please DONOT add any new information/rrfeerccee other than the context.
A.2 EVALUATION METRIC DETAILS
We evaluate the LLM trained raw and synthetic data using ten diverse general reasoning tasks, three mathematical tasks and one specialized knowledge tasks.
General Purpose Reasoning Tasks. All the benchmarks under this category are evaluated in zero- shot manner.
ARC Easy (ARC-E) and ARC Challenge (ARC-C) (Clark et al., 2018): This dataset is proposed by the A12 Reasoning Challenge (ARC). There are two sets of this data: (1) ARC-E and (2) ARC- C, containing science exam questions from grades 3 to 9. The ARC Challenge set includes more difficult questions compared to ARC-E that necessitate higher-order reasoning.
RACE (Lai et al., 2017): This dataset has been collected from English reading comprehension
exams designed for middle and high school Chinese students.
PIQA (Bisk et al., 2020): Physical Interaction Question Answering evaluates physical common- sense reasoning ability of the language model.
Winogrande (SakaulSa al., 2019): This benchmark structured task with binary options, requiring the LLM to select the correct option for a given sentence, primarily focusing on commonsense reasoning and pronoun disambiguation tasks.
in-the-blant
HellaSwag (Zellers et al., 2019): This dataset evaluates model's ability to resolve scenarios in a way that is both contextually appropriate and logically consistent, testing its grasp of language comprehension and commonsense reasoning.
OpenBookQA JOBQAJ(Mihaylo et al., 2018): This dataset is designed to evaluate deeper un- derstanding of elementary science facts by requiring models apply these facts novel situations using both open book knowledge and external commonsense reasoning.
TruthfulQA [TFQA] (Lin et al., 2022): Evaluates models' ability to generate factually correct answers by presenting 817 questions across 38 categories, designed to challenge common mis- conceptions.
17

---

Published as a conference paper at ICLR 2025
CommonSenseQA [CSQA] (Talmor et al., 2019): This dataset has been designed to test com- monsense reasoning through multiple-choice questions created from CONCEPTNET (Speer et al., 2017) relations, which requires prior knowledge beyond contextual associations for accurate swering.
Social-IQA [SIQA] (Sap et al., 2019): Evaluates LLM's ability to reason about people's actions and their social implications.
Math and Specialized Knowledge Tasks. For these tasks, we evaluate the L.I.M in few-shot man- ner.
GSM8K (Cobbe et al., 2021a): This benchmark comprises of high quality linguistically diverse grade school math word problems that evaluates the multi-step and logical reasoning ability of LLM. In this setup, we prompt the LLM with eight chain-of-thought examples from Wei et al. (2022) and take the majority vote of the answers from greedy decoding following the approach in Wang et al. (2022).
MATH (Hendrycks al.. 2021c): This dataset contains challenging competition mathematics problems that requires step-by-step processing of the problem to derive the solution. We choose 4-shot prompt from Lewkowycz.ed al. (2022b) for our evaluation process.
MMLU (Hendrycks et al., 2021a): This task is designed to evaluate a LLM'S multitask accuracy across 57 diverse subjects, including elementary mathematics, US history, and law in multiple- choice question format, requiring extensive world knowledge and problem-solving skills for high performance. We explicitly consider MMLU-STEM it contains comprehensive math and science problems that requires multi-hop and complex reasoning ability. Using the evaluation pipeline of LM Eval Harness, we evaluate the LLM with 5-shot prompts for this task.
B ADDITIONAL EXPERIMENTS AND RESULTS
B. RESULTS OF PRETRAINING LLM FROM SCRATCII
We pretrain a 8B LLM from scratch with 300B tokens using (i) 4 snapshots of CommonCraw) (ii) OWM-4B and (iii) wikipedia, books and epubs corpus corresponding 486B, 4B and 84B original tokens respectively. To emphasize math over other datasets, we provide epochs of OWM-4B in the pretraining blend resulting 35B OWM tokens that will be seen by the LLM during pretraining. For all other datasets, we maintain 0.46 epochs. For our experimentation with synthetic corpus, we analyze four variations in the OWM corpus while keeping the other data constant:
MIND-OWM-4B [TWO STUDENTS J. This data includes conversations between two students.
OWM-4B MIND-OWM-4B [1:1]. We sample raw and synthetic conversations in 1:1 ratio, ensuring an equal number of tokens to seen during pretraining from both sources. For the synthetic data, utilize the TWO STUDENTS conversations.
owM-4B + MIND-OWM-4B [Coneat]. We concatenate each raw context with all seven synthetic conversations sequentially.
MIND-OWM-4B [Longest Conversation]. From the seven conversations generated for each con- text, we select the longest conversation in token count.
PIQA
HellaSwag
OBQA
TFQA
CSQA
SIQA
Avg=All
MIND-OWM-4H 4-48 (rwo STUDENTS
68.14
36.75
77.86
63.06
69.11
40.19
39.40
37. 81)
19.66
45.55
49.75
OWM-IB-XUND-ON -OW-- [I:1]
69.71
37.33
77.61
63.69
69.51
40.87
38.20
34.97
20.39
41.77
19.68
-OW-- [Concal]
MIND-OWM-ID 4B [Longest Conversation)
68.39
36.75
77.64
62.04
68.91
40.02
39.40
38.23
20.23
44.52
19.61
Table 7: Evaluation of 8B LLM General Reasoning tasks: Conversations provide improve- ment over raw data in general purpose reasoning tasks including commonsense, factual and social reasoning tasks.
18

---

Published as a conference paper at ICLR 2025
As shown in Table 7, conversational synthetic data improves general purpose reasoning ability of LLM. Specifically, the concatenation of raw text and conversations yields the best average score for all combinations- honbiattios-highhlggtnnn the efficacy of both data towards generalizability of LLM across wide range of reasoning tasks.
MMLU- STEM
MMLU- Humanities
MMLU- Social-Sciences
MMLU- Others
Dataset
GSM8K
MATH
MMLUU
Avg-AIl
OWM-4B
4.78
4.92
26.29
25.93
26.75
27.16
26.46
12.05
MIND-OWM-4B [TWO STUDENTS
10.77
5.30
26.93
26.78
26.81
27.87
27.06
14.38
OWM-4B-MING oWM-4B [1:1]
8.49
5.02
28.01
28.44
28.
28.39
28.32
13.94
OWM-IB-MIND-OWW OWW---- [Concat]
8.04
4.98
29.18
29.22
29.51
31.54
29.79
14.27
MIND-OWM-iB [Longest Conversation)
8.57
4.60
26.77
27.16
29.12
29.29
27.97
13.71
Table 8: Evaluation of 8B LLM on Math and Specialized Knowledge tasks: Conversations are always beneficial for mathematical reasoning where specialized knowledge tasks further benefit from having both raw and synthetic data in the corpus.
In addition, for mathematical benchmarks, only synthetic data produce the best imrpovement over the raw data (Table 8). The nature of conversational data being composite and structured helps the LLM to perform well in tasks that requires step-by-step processing of complex problem. Con- versely, specialized knowledge tasks require both raw and synthetic data to attain the maximum gain.
B.2 BREAKDOWN OF INDIVIDUAL TASKS RESULTS OF CONTINUED PRETRAINED LLM
In this section, we further breakdown the performance of models trained on individual and com- binations of conversation styles across general purpose reasoning tasks and specialized knowledge tasks.
Performance across Individual Prompt Style. As shown in Table 9, synthetic data overall achieves highest gain for general purpose reasoning task compared using raw or rephrased data. Table 10 further validates the efficacy of synthetic conversations where model trained with all styles of conversational data generated from OWM-4B gets the highest the potential of upsampling high-quality data by gener-
mathematical reasoning tasks
gain across all other models models-highlighhnng ating synthetic data of diverse styles using a small seed data.
ARC-E
HellaWegg
ARC-C
AsguAli
OWN 43 Rephrase www.0033000000
Raw
21.89
37.89
75,34
65.95
71,42
46.33
41.40
36.95
32.35
46.57
$2,90
Replinuee
72.05
38.29
75.07
63.14
71.16
45.31
42.20
47.09
33.33
45.19
$3.58
TWO PROFESSORS
22.18
33.85
77.00
68.58
71.54
40.40
42.51
46 41
53.31
TRACHER STUIENY
26.30
STUDENTS
43.00
LAYMAN SNOWALL
65.75
$4.76
42.00
45.45
54.73
LONGEST CONVERSATION
34.71
37.99
76.18
64.80
72.10
47.61
41.40
45.40
19.80
46.52
$4.66
ALL CONVERSATIONS
75.17
39.04
77.86
65.43
72.31
49.40
41.00
46.65
40.79
46.42
55.41
OWN--BANIND-OWN-BB [1:1] OWX--BANIND-DWN-BBB [Concal
Combinstion
74.92
74.13
37.90
58.78
77.58
76.18
66.54
67.32
72.63
72.08
48.55
12
41. 50
41.40
42
39.27
40.70
40.05
46 97
46 72
$4.50
55.10
Table 9: Results of 7B LLM on General Reasoning Tasks: We evaluate both the baseline and model trained with synthetic data across diverse tasks that focus on general reasoning, language understanding and commonsense.
Analysis with Complete Open MatMaa... Our experiment with complete OWM-14B shows the similar trend as before. The comprehensive nature of this larger dataset continues to reinforce the advantages of synthetic data, models trained on it also exhibit enhanced performance across both general purpose reasoning (Table 11) and mathematical reasoning tasks (Table 11). This consistency across different dataset sizes highlights the robustness of the benefits gained from incorporating diverse conversational styles, further supporting the notion that expanding training data through synthetic means can lead significant advancements in the capabilities of language models.
19

---

Published as a conference paper at ICLR 2025
MIMEE-
MINLI--
STEM
Others
AggAIl
OW3--B
Raw
12.96
39.39
41.15
52.84
21.25
Reyhrase-owo--4B
Roplunse
39.71
54.76
21.10
37.93
22.74
40.73
42.21
16.78
55.65
KNOWALL
DURATTI
INTERVIEW
24.59
PROBLES SOLVING
24.72
41.
55.23
76.21
LONGEST CONVERSATION
23.78
42.72
57.92
27.15
ALL CONVERSATIONS
26.35
7.22
42.53
44.38
58.03
38.51
50.21
27.94
OWX--88 WWX-4B-NIND--O OW3L-4B [1:11 OWN 4B-KIND owN 4B (Concat]
Combication
21.68
24.49
0.22
0.14
42.56
43.67
43.85
44.87
59.21
$7.59
37.16
57.42
50.46
89.57
27.05
25.80
Table 10: Results of 7B LLM Specialized Knowledge Tasks: In this setup, assess the domain specific knowledge of LLM specifically on mathematics, science and general knowledge. We emphasize on the GSM8K, MATH and MMLU-STEM task, as these tasks predominantly checks the mathematical reasoning ability of the LLM.
Dataset
ARC-E
PIQA
Wino.
HellaSwag
ARC-C
OBQA
TFQA
CSQA
SIQA
Avg-All
Pretraining Data
70,88
38.76
78.78
67.80
73,90
43.86
42,60
41.35
29.65
44.63
53.22
OWM-14B
73.40
37.32
77.91
65.90
72.15
47.10
41.40
38.39
39.64
46.26
53.95
MIND-OWM-14BB
75.84
78.56
65.67
48.55
42,80
45.06
39.89
47.08
55.54
Table 11: Evaluations on General Reasoning Tasks with complete OwM-14B: Conversational data is beneficial for general purpose reasoning tasks.
MMLU- STEM
MMLU- Humanities
MMLU- Social-Sciences
MMLU- Others
Dataset
GSM8K
MATH
MMLU
Avg-All
Pretraining Data
9.33
4.74
37.93
41.23
51.80
53.07
45.43
34.79
OWM-14B
20.47
7.24
42.82
44.48
56.61
56.78
49.49
39.70
MIND-OWM-14B
27.29
8.24
43.55
43.95
$7.95
57.45
49.91
41.19
Table 12: Evaluations Math and Specialized Knowledge Tasks with complete OWM-14B:
Conversations improve mathematical reasoning over raw data.
C ADDITIONAL ABLATIONS
C.1 CONTEXT LENGTH VS CONVERSATION QUALITY
To generate conversations, we utilize M, which supports input sequences of up to 8K tokens. How- ever, the Open WebMatt corpus, composed of mathematical web pages from Common Crawl, often contains documents exceeding this 8K token limit, leading to errors when processing them with the LLM. A straightforward approach is to split these inputs into 8K-token windows, but initial exper- iments with this method reveal significant drawbacks. Conversations generated from the 8K-token inputs tend to summarize the lengthy context, resulting in a loss of substantial information from the original text.
20

---

Published as a conference paper at ICLR 2025
Two Professors
Teacher Student
1500
Token Length Conversation
1000
500
500
1000
1500
2000
Taken Length of Original Context
Figure 4: With increasing context length the generated conversation length decreases!
Therefore, conduct experiment 140k samples from the Open WebMath corpus of varying input length to determine the optimal input token length that generates conversations of following characteristics: (1) retains all relevant information from the original context, (2) remains grounded to the source material and (3) enhances the conversation with complementary explanations and rea- soning. For each sample, we generate conversations using two prompt (TWO PROFESSORS and TEACHER STUDENT) and observe the token length of the generations. As depicted in Figure 4, with increasing input token length (X-axis), the token length of the generated conversation (Y-axis) does not scale up linearly. For example, an input context of 2K tokens results in conversation that has tokens resulting in a lot of information loss during conversion. Analyzing the Figure 4, we see that the input token length of 500 can generate conversation that goes beyond 500 tokens meaning that the conversation not only retains information but also adds necessary reasoning resulting in more tokens.
C.2 CONVERSATION LENGTH VS MODEL PERFORMANCE
As shown in Table 1, LONGEST CONVERSATION achieves the best results among all styles. Since LONGEST CONVERSATION is curated by selecting the longest dialogue (in terms of token count) from seven conversations for single context, it raises the question of how dialogue length impacts downstream task accuracy.
To explore the relationship between dialogue length and accuracy, we measured the aver-
Style
Avg Token Length
Accuracy (Avp-All)
PROFESSORS
451.95
29.12
STUDENTS
age token length of dialogues across all con-
PROBLEM SOLVING
465.29
33, 33,38
versational styles, including LONGEST CON-
TEACHER STUDENT
494.03
32,87
INTERVIEW
497.21
32.12
VERSATION. As seen Table 13, reasoning accuracy does not cxhibit linear correlation with dialogue length. For example, with PROB- LEM SOLVING style we can achieve comparable accuracy LONGEST CONVERSATION even when the average token length for PROBLEM SOLVING is 188 lower than LONGEST CON- VERSATION. This highlights that the conversation length is not the only important factor to attain the maximum gain in reasoming ability. As mentioned Section the structure and dynamics of the conversations also play crucial role in maximizing reasoning gains.
DEBATE
511.90
33.11
LAYMAN
630.23
31.74
LONGEST CONVERSATION
653.48
34.08
Table Conversation Length vs Downstream Task Accuracy: Conversation length is not cor- related with downstream task accuracy.
C.3 CONVERSATION QUALITY ASSESSMENT
While the conversations generated by the LLM typically appear coherent, there are instances where the conversation fails to preserve the context or lacks grounding to the source material. In some cases, conversations may even be incomplete. Detecting poor-quality generation becomes challeng- ing scale. To address this, explore two quality-filtering approaches:
21

---

Published as a conference paper at ICLR 2025
Heuristic Filtering. We employ a simple heuristic based token length. Given that the input context is limited to a maximum 500 tokens and split into subcontexts of 500 tokens each to maximize information retention, we discard any generated conversations that fall below 50 tokens.
This ensures that minimal information loss detected early.
LLM-based Scoring. For more comprehensive assessment, the generated conversations. We introduce four key metrics for evaluation:
LLM score the quality of
Correctness: Verifies that all information, such as numbers and parameters, accurately reflected in the conversation.
Faithfulness: Ensures the conversation remains grounded in the context provided.
Information Preservation: Checks whether all relevant facts and knowledge from the original context are retained the conversation.
New Knowledge: Evaluates whether the conversation introduces additional explanations, reason- ing, or definitions not present in the raw input.
Given a raw context and its corresponding conversation, we ask M to rate conversation on scale of to 5 in each of four metries, with I representing poor quality and 5 representing the best possible conversation. To determine the overall quality, we compute the average score across the metrics and choose conversations with average scores more than equal to 3. Additionally, the prompt from the WebhWu (Penedo et al., 2024) annotation framework to further check the correlation between two scoring approaches. In Figure using Fine WebEdu metrics and our metrics. It is clearly visible from the figure is that LLM tends to rate own generation higher almost all the time resulting in a skewed distribution of rating. Around 96% of conversations are labelled as high quality, However, compared to WebEdu, our metric results in less skewed distribution peaattioonmmiinnggg our approach more suitable for evaluating synthetic data derived from seed corpus.
utilize
plot the scores for 140K conversations
To further investigate, we choose 20 con- texts and their corresponding conversations and manually label them on the above four metrics. We later pass these samples to LLM to obtain the quality scores. The correctness and faithful- ness metries showing a generation correct 96% of times and human annotators labeling a conversation rect 98% of times (with spearman correlation between two being 0.82) which validates the quality and reliability of the generated synthetic dialogues. When comparing the overall human scores with those from the LLM across the four metries, we observe a weak correlation between two sets (Spearman's p 0.03) and the reason- ing behind them. Human annotators prioritized the information preservation metric, while the LLM often overlooked minor information loss. Additionally, the interpretation of "New Knowledge" dif- fered between humans and the LLM. Humans valued extra reasoning and explanation as forms of new knowledge, whereas the LLM assigned higher "New Knowledge" scores to conversations taining out-of-conter information that is difficult to verify. Given these differences in the results from human and LLM b-bedd quality filtering, we simple heuristic filtering in this study and plan to explore other approaches in the future.
FineWeb Metric Our Metric
1.00
0.75
consistently high, with LLM
0.50
0.25
0.00
Figure 5: LLM tends to rate its generation higher most of the times.
C.4 COMPARE WITH DEEPSEEKMATH
To asses the quality of our data, we run pre-training experiments to compare MIND-OWM with the recently released DEEPSEEKMATII (Shao et al., 2024). The DEEPSEEKMATII approach is iterative. They construct a dataset for binary classification consisting of 500K positive data points randomly sampled from OpenWebMath (the seed corpus) and 500K negative data points randomly sampled from CommonCravi They train a fast (Joulin, 2016) classifier on these data which they then use extract samples from CommonCrawl math content. All CommonCrawi domains for which over
22

---

Published as a conference paper at ICLR 2025
10% of the existing web pages have been extracted are at this point understood to be math-related. URLs which are associated with these domains but which have yet to be collected are manually labeled as math content. The web pages hosted these addresses are added to the seed corpus and the classifier retrained. DEAPSRFK M ATH performs 4 rounds in total resulting in the SEEP MATH Corpus, consisting of some 120B math tokens. They continuously train partially converged 7B DELPSEEKCODER-VV
model on a 500B token blend to attain the DEEPSEEKMATH model and achieve substantial improvement on several math tasks. In contrast, MIND proposes a simple alternative for generating high-quality math data that boosts the mathematical reasoning ability of LLM given access to a small seed corpus.
As the DEEPSEEKMATH dataset is not public, we replicate our previous blend, D = {x U RRt), where X' = (MIND-OWM-4B (conversations of all styles except the TWO STUDENTS one) U HAND-OWM-14B (TWO STUDENTS c.nversations)). We maintain 2:1 ratio of X and Rye in the training blend. Similar the approach of DEEPSEEKMATH, we take a converged DEEPSECKCODER-VV..
model C - the unconverged model weights are unpublished as far as we are aware - and convert the model weights to format compatible with Megatron-LM, which serves training framework, before continuously training for 500B tokens. We use cosine learning rate schedule with a 19B token linear ramp-up, a maximum learning rate of 3e-4, and a minimum learning rate of 3e-6, and we anneal the learning rate over 500B tokens. We use Adam with parameters 31 = 0.9 and (2 = 0.95, a weight decay of 0.1, a gradient clipping threshold of 1.0, a sequence length of 4096, and global batch size of 2304 sequences.
GENERAL REASONING
Dataset
Tokens
GSM8K
MATH
STEM
MMLU
(Avg)
Avg-Math
Avg-All
DERESERKMATH (Shan et al., 2024)
$9.29
4.37
55.41
$4.98
$5.94
39,69
43.64
MIND-OWM-IB/14B (Combinations) 1
S00B
57.32
2.36
51,95
56.54
59.16
37.21
43.84
Table 14: DEEPSEEKMATH vs All Synthetic Conversations. A model trained on conversa- tions generated by MIND from small seed corpus can achieve math accuracy comparable to the DEEPSEEKMATI model trained on 120B unique tokens.
that model trained conversations which MIND generated given small seed corpus can attain math accuracies comparable to the DEEPSEEKMATH model with access to 120B unique math tokens in its continuous training blend. In fact, we outperform DEEPSEEK- MATH in MMLU and general reasoning tasks, reaching higher average accuracy across all tasks. This underscores the quality of MIND generated conversations and signifies the efficacy of MIND in improving mathematical reasoning ability of I.LM when the underlying raw data is limited.
From Table 14,
In contrast to our prior C. DEEPSEEK. cally designed for addressing mathematical reasoning ability and surpasses Azerbayev et al. (2023b), Team (2024a), Jiang et al. (2023), Lewkowycz et al. (2022a), Javaheripi et al. (2023), Dubey et al. (2024) [8B] base models on diverse math tasks. To evaluate the effectiveness of MIND with stronger pretrained model, we perform an additional experiment, similar to our training setup in Section 3.1 using C DEEPSEEKMATTH-T
LLM is a strong math baseline that has been specifi-
Specifically, we have continuously trained the C on 500B tokens maintaining a 2:1 ratio of math (R) and 13 CC (Rpx) dataset where the total blend is = (R We conduct two experiments by alternating R with raw (0wM-14B)) and X.
MMLC-
GENERAL REASONING
Dataset
Tokens
GSM8K
MATH
STEM
MMLU
(Avg)
Avg-Math
Avg-AIl
OWM-14B
39:42
1.59
49.92
52.87
55.47
30.31
37.34
MIND-OWM (ALL CONVERSATIONS]
500B
57.32
2.36
51.95
56.54
59.16
37.21
43,84
Table 15: Training DEEPSEEKMATHH-7 with Raw Data vs All Synthetic Dialogues. A strong pretrained LLM continously trained on conversations generated by MIND provides significant boost in math accuracy than the same model trained on raw data showing the effectiveness of MIND regardless of pretraining model quality.
As shown in Table 15, model trained on MIND-OWM data shows consistent improvement over model trained on data- resulting in 17.90% gain on GSM8K, 6.90% average improvement across math
23

---

Published as a conference paper at ICLR 2025
tasks and 3.43% average improvement across ten general reasoning tasks. These results further solidifies the effectiveness MIND regardless of the quality of the pretrained model.
C.5 CONVERSATIONS ON CODE TASKS
Unlike raw data, conversations tend break down the context into sub-context and participants exchange their reasoning about the sub-context in a single turn. This feature is particularly useful for mathematical logical reasoning which require step-by-step reasoming. However, this structure might hurt performance of I.L.M in domains where sequence of context needs to be preserved such as in codes. To further investigate the impact of conversational data on the coding capabilities of I.I.M, conduct evaluation of models trained on both raw and synthetic data across four established coding benchmarks: HvmanEval (Chen et al., 2021), MBPP (Austin al., 2021), HvmanEval+, and MBPP+ (Liu et al., 2024). These benchmarks are specifically designed to assess the model's ability to generate functional code response to given prompts.
Our results, presented in Table 16, demonstrate that conversational synthetic data does not hance coding performance. This is largely due to the way conversations tend to fragment code, wrapping natural language and thereby obscuring the intended sequence and logic inherent in programming tasks. Consequently, while conversations may be effective in contexts that benefit from collaborative reasoning, they are not suited for preserving the integrity of code, leading to diminished performance in coding benchmarks.
Dataset
Style
HumanEval
HumanEvale
(Sanitized)
Avg-All
OWM-4B
Raw
12.20
10.98
23.74
0.00
11.73
Rephrase-owM-4B
Rephrase
5.49
4.27
20.23
0.53
7.63
TWO PROFESSORS
8.54
4.88
20.62
0,00
8.51
9,76
12.47
TWO STUDENTS
10.37
7,93
26.07
0.26
11.16
LAYMAN KNOWALL
10.37
8.54
26.46
0.79
11.54
MIND-OW
DEBATE
9.15
INTERVIEW
7.32
4,88
23.35
0.26
8.98
LONGEST CONVERSATION
9.15
7.32
28.40
0.53
11.35
12.20
OWM--B--NIND-Ow---B8 [1:1] Oww-4B-NIND-owN--BB [Concar]
Combination
13:41
10.37
10.98
7.93
23.35
31.52
0.00
11.94
12.46
Table 16: Results of 7B LLM on Code Tasks: Conversations and rephrases are ineffective for code benchmarks.
Interestingly, also observe that rephrasing. which resembles raw data more closely in structure, further degrades coding accuracy. Our qualitative analysis of the rephrased documents reveals that the conversation generator important tags and code indentation, which are essential for comprehending and executing code correctly. This finding underscores the necessity of designing code-specific prompts that retain the structural components vital for coding tasks.
often removes critical elements such as
24

---

Published as a conference paper at ICLR 2025
D DIALOGUE EXAMPLES
Raw TEXT
# CAT Quantitative Aptitude Questions CAT Ratios, Mixtures, Alligations and Averages Questions
#*#fI CAT Questions Mixtures and Alligations Mixtures- Range
The question is from CAT Ratio, Mixtures and Averages. Given the details about two alloys, need to determine the possible values the weight of ally take. CAT exam is known to test on basics rather than high funda ideas. A range of CAT questions can be asked from Ratios and Proportions, Mixtures, Alligations and Averages. Make sure you master the topics. 21MMs CAT questions bank provides you with CAT questions that can help you gear for CAT Exam CAT 2020.
Question 16: 100 kgs of an alloy of tin and lead in the ratio 1:3 is mixed with kgs of an alloy of tin and lead in the ratio 3:2. If the overall alloy should contain between 40% and 50% tin, what is the range of values x can take?
100 kgs I x < 200 kgs
2. 80 kgs S S 240 kgs
110 kgs S x < 220 kgs
4. 75 kgs
S 250 kgs
## Best CAT Coaching in Chennai
#### CAT Coaching in Chennai CAT 2020Online Batches Available Now!
SYNTHETIC DIALOGUE [TWO PROFESSORS
Good morning, colleague. I'd like to discuss question from CAT Ratio, Mixtures, and Averages. It's about two alloys of tin and lead, and we need to determine the possible values of the weight of alloy.
Professor 1:
Professor 2: Ah, yes. I'm familiar with this type of question. It's a classic example of Mixtures and Alligations. Please go ahead and share the details of the question.
Professor 1: We have 100 kgs of an alloy of tin and lead in the ratio 1:3, which means it has 25 kgs of tin and 75 kgs of lead. This alloy is mixed with kgs of another alloy of tin and lead in the ratio 3:2. The overall alloy should contain between 40% and 50% tin. We need find the range of values can take.
Professor 2: That's great problem. We can start by analyzing the second alloy. If it has ratio of 3:2, that means has 3x/5 kgs of tin and 2x/5 kgs of lead, with a total weight of kgs.
Professor 1: Exactly. Now, when we mix these two alloys, how does the mixture change? We need to find the overall percentage of tin in the resulting alloy.
Professor 2: Right. Let's think about it. the overall alloy should contain between 40% and 50% tin, what does that imply about the range of values can take? Professor 1: That's the key question. we can determine the range of x, we can answer the question. Do you think it's possible to narrow down the options to of the given choices? Professor 2: Yes, I think so. Let's 's analyze the options. We have 1. 100 kgs < x < 200 kgs, 2. 80 kgs
240 kgs, 110 kgs < x S 220 kgs, and 75 kgs one do you think is the correct answer? Professor 1: Well, after analyzing the problem. think the correct answer is option 80 kgs < x < 240 kgs. What do you think?
250 kgs. Which
Professor 2: agree with you. The correct answer is indeed option 2. Well done! This question is a great example of how CAT exam tests our understanding of basics rather than high-funda ideas.
25

---

Published as a conference paper at ICLR 2025
Raw TEXT
Free Version
Moderate
# Interpretation of Equilibrium from Graph of Concentration vs Time
CHEM-NWU9LE
Consider the graph below of chemical reaction observing the concentration of two chemi- cal species with time.
Which species is the reactant? $$
Which species is the product?
$$
At what region does the solution appear to be in equilibrium? S$
What is the K2 for the reaction?
SYNTHETIC DIALOGUE [TEACHER STUDENT
Student: Hi, teacher. I'm having trouble understanding this graph of a chemical reaction.
Can you help me?
Teacher: Of course! What's your first question about the graph?
Student: Okay, so which species is the reactant?
Teacher: Let's take look the graph. The species whose concentration is decreasing with time is the reactant. Can you identify which one that is?
Student: Ah, I think it's... (looks the graph) ...the whose concentration starts high and decreases.
Teacher: That's correct! The reactant is the species whose concentration starts high and decreases with time. Now, what's 's your next question?
Student: Which species is the product?
Teacher: Good question! The product is the species whose concentration starts low and increases with time. Can you identify which one that is?
Student: Yeah, think it's... (looks at the graph) ...the one whose concentration starts low and increases.
Teacher: That's right! The product is the species whose concentration starts low and in- creases with time. Now, what's 's your next question?
Student: At what region does the solution appear to be in equilibrium?
Teacher: Look at the graph again. At what point does the concentration of both species seem to be constant, changing with time?
Student: Hmm... think it's... (looks at the graph) ...after a certain time, when the lines are flat.
Teacher: Exactly! The solution appears to be in equilibrium when the concentration of both species is constant, which is after a certain time when the lines are flat. Now, what's your next question?
Student: What is the Ke for the reaction?
Teacher: Unfortunately, the graph doesn't provide enough information to calculate the Ke value. We would need more data or information about the reaction to determine the equilib- rium constant, Ke.
Student: Okay, think understand now. Thank you for your help!
26

---

Published as a conference paper at ICLR 2025
RAW TEXT
What's the probability that: The person is not age 18 34 and does not own an iPhone?
The presidential election is coming.
Five survey companies (A, B, C. D, and E) are doing survey to forecast whether or not the Republican candidate will win the election.
Each company randomly selects sample size between 1000 and 1500 people.
All of these five companies interview people over the phone during Tuesday and Wednesday. The interviewee will be asked if he or she 18 years old or above and U.S. citizen who registered vote. If yes, the interviewee will be further asked: will you vote for the Republican candidate?
On Thursday moriing these five companies announce their survey sample and results the same time on the newspapers.
The results show that a% (from A). b% (from B), c% (from C), d% (from D), and e% (from E) will support the Republican candidate.
The margin of error is plus/minus 3% for all results. Suppose that c>a>d>>>b. When you see these results from the newspapers, can you ex- actly identify which result(s) (are) not reliable and not accurate?
That is, can you identify which estimation interval(s) does (do) not include the true popula- tion proportion?
If you can, explain why you can, if no, explain why you cannot and what information you need to identify.
Discuss and explain your reasons. You must provide your statistical analysis and reasons.
SYNTHETIC DIALOGUE [TwO STUDENTS
**Turn 1**
Alex: Hey Ben, have you started working on our assignment about the presidential election survey?
Ben: Yeah, I've been going over the context. seems like we need to identify which survey
result is not reliable and accurate.
Alex: Exactly! Let's break down. We have five survey companies, A, B, C, D, and E, each with sample size between 1000 and 1500 people.
Ben: Right, and they all have a margin of error of plus/minus 3%.
**Turn 2
Alex: Okay, so we're given that e a>> d e Ben: tells us the order of the proportions of people who will support the Republican candidate according to each survey company.
What does this tell us?
Alex: That's right. And since the margin of error is the same for all companies, we can as- sume that the true population proportion should lie within a similar range for all companies. **Turn 3
Ben: But wait, if the true population proportion lies within similar range, then shouldn't all the estimation intervals overlap?
Alex: Exactly! If the estimation intervals don't overlap, it means that least one of them is not reliable and accurate.
Ben: So, need check if the intervals overlap. How do we do that?
**Turn 40*
Alex: Let's caleulate the estimation intervals for each company. For company it's a% + 3%, and similarly for the others.
Ben: Okay, got it. So, we have a% + 3%, b% 3%, c% 3%, d% + 3%, and e% + 3%. Alex: Now, let's check if these intervals overlap. Since > d > e > b, the interval for company C should be the highest, and the interval for company B should be the lowest. **Turn $**
Ben: see what you're getting at. If the intervals overlap, then the highest value of the lowest interval (company B) should be greater than equal to the lowest value of the highest interval (company C).
27

---

Published as a conference paper at ICLR 2025
Alex: That's right! If that's not the case, then we know that at least one of the intervals is not reliable and accurate.
Ben: Let's plug in some numbers and see if we can identify which one is not reliable.
**Turn 6**
45%, b 38%, 50%, d 42%, and 40%. Can we identify which one is not reliable?
Alex: Alright, let's say the results
Ben: Let's calculate the estimation intervals... (calculates)...and check if they overlap.
Alex: (checks the calculations) Ah, look at that! The interval for company B (38% + 3%) doesn't overlap with the interval for company C (50% + 3%). Ben: That means company B result is not reliable and accurate!
**Turn 7**
Ben: But why can't 't we identify the other unreliable results, if any?
Alex: That's because we only have the relative order the proportions, not the exact values.
We need more information to identify other unreliable results, if any.
Ben: I see. So, we can only identify company B' result as not reliable and accurate, but we can't say for sure about the others.
Alex: That's ' correct. We ve done our best with the given information.
28

---

Published as a conference paper at ICLR 2025
RAw TEXT
# Decimal to Percent Caleulator.
The calculator will convert decimal into percent, with steps shown. If the calculator did compute something or you have identified an error, or you have suggestion/feedbakk, please write it in the comments below.
Convert 1.4 into decimal. ## Solution From the converting decimal to percent note, we know that to obtain a percent, we need to multiply decimal by 100.
But multiplying by 100is equivalent to moving a decimal point 2 positions to the right.
SYNTHETIC DIALOGUE [LAYMAN KNOWALL
]
*TTurn 1
Layman: Hi, I'm not sure what this calculator does. Can you explain it to me?
Me: This calculator converts decimals into percents. shows the steps to the conversion.
**Turn 2
Layman: Okay, got it. So what's the problem we'r re trying to solve? Me: We're trying to convert 1.4 into percent. **Turn 3**
Layman: How do do that? Is there formula something? Me: Yes, there is. According to the note on converting decimal to percent, we need to multiply the decimal by 100 obtain percent. **Turn 40*
Layman: Why do we need multiply by 100? What's the logic behind it?
Me: Well, I'm not sure about the logic, but the note says that's 's what we need to do. But I can tell you that multiplying by 100 equivalent moving the decimal point 2 positions to the right.
**Turn 5**
Layman: What do you mean by moving the decimal point 2 positions to the right? Can you explain that?
Me: Sure. When you multiply number by 100, the decimal point shifts 2 places to the right. For example, if you have 1.4 and you multiply it by 100, the decimal point will move 2 places to the right, making it 140.
**Turn 6**
Layman: Ah, I think understand now. So. if we multiply 1.4 by 100, what do we get? Me: That's ' right! If we multiply 1.4 by 100, we get 140. And that's the equivalent percent value of the decimal 1.4.
29

---

Published as a conference paper at ICLR 2025
RAW TEXT
Try to finish the derivation and how applies to the sum you trying compute. Sorry if this is hard to understand.
Jun 14, 2009 ### jgens
Cyosis, why would he/she use the sum for the first natural numbers when his sum asks for odd natural numbers only?
Jun 14, 2009
### Cyosis If you know the sum for the first n natural numbers you can quickly solve the sum for the first odd numbers. Had he known it, would have been the easiest way to calculate the sum he's interested in.
Jun 14, 2009
### Manchot
The story that six-year-old Gauss was told by his teacher add the numbers to 100 as a form of busywork. In seconds, he came with the answer, much to his teacher's 's surprise.
What he did was notice that since 1+100=101, 2+99=101, 3+98=101, etc., the sum becomes simple multiplication: there are 50 of the above pairs,ão the sum is just 50*101 = 5050. Anyway, that's the intuition behind the identity that Cyosis posted. You can pretty much use the exact same intuition.
10. Jun 14, 2009
### gabrielh Okay thanks, I get it now.
11. Jun 14, 2009
SYNTHETIC DIALOGUE [DEBATE
jgens: I don't understand why Cyosis suggested using the sum of the first n natural numbers to solve the problem. The original problem asks for the sum of the first odd natural numbers, not all natural numbers.
Cyosis: I see what jgens is saying, but think my approach is still valid. you know sum of the first natural numbers, you can easily derive the sum of the first odd natural numbers. It's actually the easiest way to calculate the sum.
Manchot: think I can help clarify things. The story about Gauss adding numbers from to 100 relevant here. He noticed that 1+100=101, 2+99=101, and so on, which makes the sum simple multiplication problem. There are 50 such pairs, the sum is 50 101 5050. This is the intuition behind Cyosis' suggestion.
jgensL I what Manchot saying, but I'm still convinced. How does this story about Gauss relate to the sum of odd natural numbers?
Cyosis: Well, jgens, the same intuition applies here. We can use similar pairing method to calculate the sum of odd natural numbers. It's not direct application, but the principle is the same.
gabrielh: I think I'm starting to get it now. So, we can use the sum of the first natural numbers as a stepping stone calculate the sum of the first odd natural numbers, and the Gauss story provides the intuition behind this approach.
Manchot: Exactly, gabriclh! That's the idea. By understanding the sum the first n natural numbers, can derive the sum of the first odd natural numbers using similar method. Cyosis: I'm glad we could clarify things. The key is to recognize that the sum of the first more general problem that be used solve the more specific
natural numbers problem of the sum of the first odd natural numbers.
30

---

Published as a conference paper at ICLR 2025
RAW TEXT
The Cryptographic Lens Shafi Goldwasser Colloquium Lecture POSTPONED!
Tuesday, 21.10.2014, 14:30 Room 337-8 Taub Bld.
ceClub: Centralized Zuee-QuueeNetworrk Architecture
Jonathan Perry (MIT, CSAIL)
Monday, 06. 10.2014, 11:00
EE Meyer Building 861
Current datacenter networks inherit the principles that went into the design of the Internet, where packet transmission and path selection decisions are distributed among the endpoints and routers. Instead, we propose that each sender should delegate control-to centralized arbiter-of when each packet should be transmitted and what path should follow. Fastpass datacenter network architecture built using this principle. Fastpass incorporates two fast algorit...
Parameterized Automata Constructions and Their Applications
Ran Ben-Ban Basat Monday, 22.09.2014, 15:30
Taub 701 Parameterization is useful tool for handling NP-hard problems in the real world. It aims to reduce the running times of algorithms for such problems, by confining the combinatorial explosion some parameter k. As this parameter is often significantly smaller than the input size, it allows to develop practical algorithms for non-trivial classes of instances for these problems. In this talk we present novel framework for developing parameterized algorithms, using construction...
Observing the Observers: Social Context Analysis Using Computer Vision
Meir Cohen
Wednesday, 10.09.2014, 12:00
Taub 601
is quite common that multiple human observers attend to a single point of interest.
Mutual awareness activity (MAWA) refers the dynamic of this social phenomena.
A peak of a MAWA is known as a mutual awareness event (MAWE) and can be interpreted buzzëvent, which draws the attention of many observers.
A preferred way to monitor those social phenomenon is with a camera that captures the human observers while they observe the activity the scene. Our work studies the
SYNTHETIC DIALOGUE [INTERVIEW ]
Subject Matter Expert: Unfortunately, the lecture was postponed, so we don't have any information about the specific topic she going to cover. However, the title of the lecture was The Cryptographic wens,which suggests that it might have been related to cryptography and its applications.
Interviewer: That's 's shame it was postponed. Moving on, let's talk about the ceClub lecture by Jonathan Perry from MIT, CSAIL. Can you summarize the main idea behind his proposed datacenter network architecture, Fastpass?
Subject Matter Expert: Yes, certainly. Jonathan Perry proposed centralized žero- queuerietwork architecture called Fastpass, which is departure from the traditional dis- tributed approach used Internet design. In Fastpass, each sender delegates control centralized arbiter, which decides when each packet should be transmitted and what path it should follow. This approach aims to improve the efficiency of datacenter networks.
Interviewer: That's 's fascinating, What you tell about the talk by Ran Ben- Ben-Basat parameterized automata constructions and their applications?
31

---

Published as a conference paper at ICLR 2025
Subject Matter Expert: Ran Ben-Basat discussed the concept of parameterization, which useful tool for handling NP-hard problems the real world. The idea to reduce rumning times of algorithms for these problems by confining the combinatorial explosion to some parameter He presented a novel framework for developing parameterized algorithms using construction-bassed methods.
Interviewer: see. Now, let's switch gears and talk about the lecture by Meir Cohen on social context analysis using computer vision. Can you explain what mutual awareness activity (MAWA) and mutual awareness event (MAWE) refer to in this context?
Subject Matter Expert: Yes. Mutual awareness activity (MAWA) refers to the dynamic of multiple human observers attending to single point of interest. A peak of this activity is known as mutuaa awareness event (MAWE). which can be interpreted as a buzzevent that draws the attention of many observers. Meir Cohen's work focuses monitoring and ana- lyzing these social phenomena using computer vision, specifically by capturing the human observers with camera while they observe the activity in the scene.
32

---

Published as a conference paper at ICLR 2025
RAw TEXT
Note: At section X 12 m, the speed is close 1.5 Thus for a return trip of 2x20m 40 m to return to the profile X = 14m, the wave takes 27 s. Thus, the graph of Figure. 11 is located before the (eventual) reflected wave coming from the absorbing upstream boundary, which can disturb the sensors.
## Comparative analysis of records
We have exploited some of the tests in transversal manner, to identify particular behaviors of the surface waves:
impact of the variation of the amplitude for several tests characterized by the same param-
eters but with roughnesses, or geometries either uniform, or linear
impact of the change in the period
impact of the variation of the initial amplitude
comparison for the same test carried out in uniform converging section
impact of roughness variation
### Impact of the variation of the amplitude for several tests with the same parameters
We have compared several tests characterized by D70, by Ab absorbing boundary conditions
and for the same period T5. These are tests D70A5T5, D70A2T2 and D70A5T7 in RoAbUn, SmAbCo and SmAbUn conditions
Figure Figure 17 Figure 18
Figure 16: amplitudes variations for the 3 runs are increasing according to the slope, which in accordance with the theory.
Figure 17: the initial amplitudes of the waves generated by the wave maker not always well respected: all the curves of the same diagram should start from the same point to X = 0
Figure 18: there is sharp change in amplitude at the arrival at the top of the slope.
SYNTHETIC DIALOGUE [PROBLEM SOLVING ]
User: Hi, I'm analyzing some data on surface waves and I'd like to discuss some obser- vations with you. Can you help understand the impact of varying amplitude surface waves?
AI Assistant: I'd be happy to help. From the context, I see that you have several tests with the same parameters but different amplitudes. Specifically, you have tests D70ASTS, D70A2T2, and D70AST7 in RoAbUn, SmAbCo, and SmAbUn conditions. What would you like to know about these tests?
User: Yes, that's correct. I'm looking at Figures 16, 17, and 18. Can you explain what's happening in Figure 16?
AT Assistant: In Figure 16, the amplitudes of the waves are increasing according to the slope, which is consistent with theoretical expectations. This suggests that the amplitude of the wave is directly related to the slope of the surface.
User: That makes sense. But what about Figure 17? The initial amplitudes of the waves generated by the wave maker don't 't seem to be consistent.
AI Assistant: That's good observation. Yes, the initial amplitudes of the waves don't start from the same point at = O, which is unexpected. This could be an issue with the wave maker the experimental setup.
User: Okay, got it. And what about Figure 18? There seems to be sudden change in amplitude the top of the slope.
Al Assistant: Yes, that's 's correct. Figure 18 shows sharp change in amplitude at the arrival the top of the slope. This could be due to the interaction between the wave and the slope, causing a sudden increase or decrease in amplitude.
User: That's helpful. So, what can we conclude about the impact varying amplitude on surface waves?
AI Assistant: Based on these figures, it seems that the amplitude of the wave is directly related to the slope of the surface, and changes in amplitude can affect the behavior of the wave. However, we also need to consider the potential issues with the wave maker or experimental setup that may be affecting the results.
User: Okay, that's a good summary. Thanks for your help in analyzing these figures!
33


Reviews:
[{"review_id": "CrY01hhntG", "reviewer": "Reviewer_vrdW", "summary": "The paper introduces MIND (Math Informed syNthetic Dialogue), a novel method for generating large-scale synthetic dialogues to improve the mathematical reasoning abilities of large language models (LLMs). Traditional synthetic data often fails to enhance complex, multihop reasoning tasks because it doesn't add new knowledge to existing corpora. MIND addresses this by generating structured conversations based on the OpenWebMath (OWM) corpus, resulting in a new dataset called MIND-OWM.\n\nThese synthetic dialogues decompose complex math problems into multi-turn conversations, promoting step-by-step reasoning. Experiments reveal that incorporating knowledge gaps between dialogue participants is crucial for generating high-quality math data. Models pretrained on MIND-OWM show significant improvements in mathematical reasoning benchmarks (e.g., GSM8K: +13.42%, MATH: +2.30%), specialized knowledge tasks (MMLU: +4.55%, MMLU-STEM: +4.28%), and general reasoning tasks (+2.51%) compared to models pretrained on raw data alone", "questions": "N/A", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. This paper articulates the contributions, methodologies, and results in a clear way. It is pleasant to read.\n2. Despite simple and straightforward, strong experimental results showcase the effectiveness of the proposed MIND framework and the data synthesized by it. The results are evaluated on three different math corpus and they all demonstrate substantial improvements by training on the generated data.", "weaknesses": "One thing I would like to point out is there is a very relevant work to discuss and compare to. \nDialog Inpainting: Turning Documents into Dialogs (https://proceedings.mlr.press/v162/dai22a.html) \nThis paper also proposes ways to synthesize conversations from knowledge sources. Although it is not specifically for math, the authors should at least discuss the differences in methods in rebuttals.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mU7sOaVdf9", "reviewer": "Reviewer_ikXv", "summary": "This paper proposes MIND to generate large-scale, high-quality synthetic math dialogues to pretrain LMs and enhance their mathematical reasoning abilities. MIND prompts a pretrained LLM (llama3-70B-instruct) to convert raw mathematical text into structured multi-turn conversations using diverse conversational styles. These synthetic dialogues break down complex problems step-by-step while injecting complementary explanations and reasoning. The authors generated 64B tokens of synthetic data using the 14B token OpenWebMath corpus. They conducted experiments with varying conversational styles and participants to assess impact on reasoning. Models pretrained on MIND-generated dialogues outperformed those trained on raw data in mathematical reasoning and general reasoning. In summary, MIND provides a way to upsample limited high-quality math data into structured dialogues that embed multi-hop reasoning.", "questions": "- What is the impact of conversation length on model performance? Is there an optimal length?\n\n- How well does this approach generalize to other technical domains beyond mathematics?\n\n- The continued pretraining setup upsamples OpenWebMath but keeps CommonCrawl data constant. How much does the relative ratio of math vs general data impact results?\n\n- Will the data be released?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Originality: The paper presents an approach (MIND) to generate high-quality synthetic math dialogues that improve math reasoning in LMs. Compared to prior work on synthetic pretraining data that mostly rephrases raw text, MIND adds semantic variations and step-by-step reasoning that are crucial for complex math problem solving.\n\nQuality: The paper is thorough in evaluating MIND across multiple dimensions - testing different conversational styles, scaling behavior, and applicability to varying seed corpora. Ablations provide good insights, like the importance of knowledge gaps between dialogue participants. \n\nClarity: The paper is easy to read. Key aspects of the approach, experiments and results are clearly explained.\n\nSignificance: This work demonstrates the potential of structured conversational data to enhance reasoning in language models, especially when domain-specific high-quality data is limited.", "weaknesses": "- Since the method uses the LLAMA3-70B-INSTRUCT model to generate conversations, it is unclear whether the improvements in downstream reasoning tasks come from the quality of the generated dialogues or are simply a result of model distillation from the powerful LLAMA3-70B model. The authors should investigate this and isolate the impact of the MIND-generated dialogues from the influence of the underlying LLM.\n\n- The experiments are based on a single in-house pretrained model checkpoint. It is possible that this model is not very well pretrained, making the improvements from synthetic dialogues appear more significant than they would be for a highly-optimized model. To demonstrate the generality of the MIND approach, the authors can experiment with multiple popular and high-quality pretrained models such as LLAMA-3, Mistral, GEMMA, etc. Consistent improvements across a range of strong baseline models would provide more convincing evidence for the effectiveness of the proposed method.\n\n- The paper focuses on math reasoning. It would be nice to see if MIND generalizes to other technical domains needing step-by-step reasoning, like physics, engineering, coding etc. Some preliminary experiments could help to see the broader applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a3GUAyc1Xp", "reviewer": "Reviewer_pFmJ", "summary": "This paper introduces a new method of generating synthetic math data for continued pretraining. The authors design a conversation setting to let two speakers talk about a given text in the math domain. Each speaker can be a teacher, student, professor, etc. Seven types of person pairs are used as prompts (e.g., teacher-student), though it seems that professor-professor is less helpful from the perspective of downstream performance.", "questions": "please refer to the previous section.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Experiments on two pretraining corpora demonstrate the helpfulness of the proposed method on reasoning tasks.\n\nThis method can also be used to clean pre-training corpora.", "weaknesses": "I have several concerns/questions regarding the motivation/details.\n\n- The quality of the synthetic math data: The authors introduced that heuristics are applied to filter the generated conversations and provided the similarity between raw text and the synthetic dialogues. I'm wondering about the length difference between the original text and the corresponding conversation (on average), and it is unclear the degree of hallucination issues in the generated text. \n\n- The impact of knowledge distillation: LLaMa3-70B-Instruct is prompted with different persona-pair for conversation generation. However, its contribution is not well justified. The authors may consider weaker models such as LLaMa3-8B or even an SFT one based on the pre-trained model in this paper.\n\n- A stronger baseline may be considered: The authors only compare their method with a rephrasing text baseline. The conversation format explored in this paper seems to be useful for improving the multi-turn instruction following ability. A more advanced baseline can be \"given a text, (follow-up) questions and their answers are generated automatically\" as other continued pretraining studies (e.g., [1])\n\n- Please elaborate on the details in the evaluation: \n    * e.g., specify zero-shot or few-shot for different tasks. I'm wondering whether the gains will be decreased in the few-shot setting.\n    * The longest-conversation achieves the best performance (Table 1). Why is \"student-student\" considered for experiments on 14B and ablation studies?\n\n- The authors may consider more diverse persona combinations for diversity and data scale-up.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)