System-2 Mathematical Reasoning via Enriched Instruction Tuning
Authors: Huanqia Cai
Yijun Yang
Zhifeng Li
Venue: Accepted by TMLR ()

Abstract: Solving complex mathematical problems via system-2 reasoning is a natural human skill, yet it remains a significant challenge for current large language models (LLMs). We identify the scarcity of deliberate multi-step reasoning data as a primary limiting factor.  To this end, we introduce Enriched Instruction Tuning (EIT), a method that enriches existing human-annotated mathematical datasets by augmenting human-annotated data with AI-generated feedback to create fine-grained reasoning trajectories. These datasets are then used to fine-tune open-source LLMs, enhancing their mathematical reasoning abilities without reliance on any symbolic verification program. Concretely, EIT is composed of two critical steps: Enriching with Reasoning Plan (ERP) and Enriching with Reasoning Step (ERS). The former generates a high-level plan that breaks down complex instructions into a sequence of simpler objectives, while ERS fills in reasoning contexts often overlooked by human annotators, creating a smoother reasoning trajectory for LLM fine-tuning.  Unlike existing CoT prompting methods that generate reasoning chains only depending on LLM's internal knowledge, our method leverages human-annotated initial answers as ``meta-knowledge'' to help LLMs generate more detailed and precise reasoning processes, leading to a more trustworthy LLM expert for complex mathematical problems. In experiments, EIT achieves an accuracy of 84.1% on GSM8K and 32.5% on MATH, surpassing state-of-the-art fine-tuning and prompting methods, and even matching the performance of tool-augmented methods.

Published in Transactions on Machine Learning Research (09/2025)
System-2 Mathematical Reasoning via Enriched Instruc- tion Tuning
Huanqia Cai Tencent
caihuanqia gmail.com
Yijun Yang
yijun.steven.yang gmail.com
Tencent
University of Technology Sydney
Zhifeng Li +1
zhifengO. li@ gmail.com
XIntelligence Technology Co., Ltd.
Reviewed on OpenReview:
Abstract
Solving complex mathematical problems via system-2 reasoning is a natural human skill, yet remains significant challenge for current large language models (LLMs). We identify the scarcity of deliberate multi-step reasoning data primary limiting factor. To this end, we introduce Enriched Instruction Tuning (EIT), method that enriches existing human-annotated mathematical datasets by augmenting human-annotated data with Al-generated feedback to create fine-grained reasoning trajectories. These datasets are then used to fine-tune open-source LLMs, enhancing their mathematical reasoning abilities without reliance on any symbolic verification program. Concretely, EIT' composed of two critical steps: Enriching with Reasoning Plan (ERP) and Enriching with Reasoning Step (ERS). The former generates high-level plan that breaks down complex instructions into sequence of simpler objectives, while ERS fills in reasoning contexts often overlooked by human annotators, creating a smoother reasoning trajectory for LLM fine-tuning. Unlike existing CoT prompting methods that generate reasoning chains only depending L.LM's 's internal knowledge, our method leverages human-annotated initial answers as "meta-knowledge" to help LLMs generate more detailed and precise reasoning processes, leading to a more trustworthy LLM expert for complex mathematical problems. experiments, EIT achieves an accuracy of 84. % on GSM8K and 32.5% on MATH, surpassing tfate-of-the-ar fine-tumin Ane-tuning and prompting methods, and even matching the performance of tool-augmented methods.
Introduction
Large language models (LLMMs) are increasingly recognized as a promising stride towards realizing Artificial General Intelligence (AGI) due to (1) their potential for a wide range of intelligent activities (2) their scaling up of performance increased dataset size, model size, and computing budget (Bubeck al., [2033 Kaplan al. (022)). Surprisingly, all of the current advances in LLMMs established i.e., Transformer decoder + next-token prediction (Radford al.) 2018). Though promising, the most powerful LLMs, e.g., GPT-4 (Achiam et al. 2023). still frequently make ridiculous mistakes when solving fundamental mathematical problems, e.g., multiplication of four digits (Hagendorff et al. 2022), hindering their usage in scenarios requiring mathematical reasoning, such as autonomous driving (Fu et al. 2024). embodied intelligence (Brohan et al., 2023
embarrassingly "simple" training paradigm,
Yang et al. 2024 Zhou et al. 2024), and teaching assistance (Hicke et al. 2023). The root of this issue is that
"The work done during working Tencent.
1 "Corresponding author.

---

Published in Transactions on Machine Learning Research (09/2025)
uestion The product
1200, a is an integer, and b is an odd integer. What is the largest possible value of b?
Ground Truth
(b) CoT
(c) EIT
Answan (ERP)) find the largest possible value of
Factoring out the highest power of from
Answer To Lind the Jargest possible value of b, can follow stages
1200, we fize that
consider the prime factorization of 1200 and the fact that b must be edd, Since cannot include the factor of 2 present largest
1200 2
1.A cknowledge that is is an odd integee; which can be expressed 24 + where is an integer:
factor of 1200 can be the product of the odd prime factors raised to the appropriate power (ERS) To solve this, we List expeess 1200 as product of it juime factors:
Thorefom, the largest possible value of is 75.
2. Roalize that find the lorpest passible ndd intager iy we need find the larpest value of since l Increases incresses.
(a) MetaMath
1200 2' x 3
Answer; Since a is un integer, must be fastor of 1200. The prime facborization of 1200 is 2 x 3 5?.
Since is an odd Intoger, cannot include the factor of 2 that is present in the prime factorization of 1200. Therefore, b must be composed only the odd prime factors of 1200, which are and 5. To maximize b we should use the highest powers of these primus that am factors of 1200.
Since b odd, it can be written as 26 + where an intager. Thus,
ab aç2k + 1) 2nk + 1200.
To maximize 6, we want to choose the largest of 5, which is 5 25. Thereforu, the lugest possible value is 25.
integee, largest possible value when a - 1, in which case 1200 (and it - 599).
The highest power 3 that divides 1200 3 and the highest power i 5 Multtplying ling these together giives the largest possible value for b:
If b is any larger, then a beoomes fraction, so m
1200 the largest possible value.
b 5 - 3 25 - 75
The iss
Therefor, the loggest possible value of b 1200 22000033 (XC)
Therefory, the Largest possible value of is 75. (v/)
Figure I: Comparison of LLM's response generated by four different methods on randomly selected problem from the MATH dataset (Hendrycks) ct al. 2021]. (a) Meta considers factorization, but the problem-solving plan is wrong and ignores other odd factors. (b) Chain-of-thought
prompting (CoT) generates a valid plan, but hallucinations occur during the solution process, leading to trivial calculation error: if a equals and A is 599.5, then b is 1200, which is not odd integer. (e) The L.I fine-tuned by our Enriched Instruction Tuning (EIT) first generates high-level plan (blue parts) decomposing the original question sequence of lower-level objectives and then produces fine-grained reasoning trajectory (purple parts) with the guidance from human-provided initial answers.
LLM-generated responses entirely rely on next-token prediction, a process akin to our intuitive, quick- thick-thinking system-1 reasoning (Daniel 2017). This is fine for easy tasks but falls short for complex mathematical problems, in which humans usually switch to thoughtful, logical system-2 reasoning, a slower and more deliberate way of thinking while making our answers more accurate and less biased (Daniel 2017
The deficiency of system-2 mathematical reasoning in LLMs has motivated many recent works that can be divided into two categories: fine-tuning- based and prompting braeet based methods. sane-ttuinn--bseee methods, such as Phi-GSM (Liu) et al. 2023), MetaMath (Yu et al. 2023), MathGenie (Lu et al. 2024), and Orca-Math (Mitra et al., 2024), update LLMs' parameters by distilling privileged LLMs or learning from human's 's reasoning trajectories. Many of them adopt very complicated augmentation strategies in order to boost performance (see Table D. For example, Phi-GSM and Orea-Math generate more reliable training data via the ensemble of multiple L.LMs. MetaMath and MathGinie adopt several elaborate augmentation strategies, including question rephrasing, backward reasoning, and self-verification, to augment mathematical reasoning data for fine-tuning. Additionally, some methods use enormous amounts of data (e.g.. 12M for Phi-GSM) external tools, c.g., code verification (Wang et al.) 2023a) and tree search (Xie et al. 2024 Chen et al., 2024 Brandfonbrener et al., 2024), to correct calculation and reasoning errors. However, such a mixture of many augmentation strategies slows down the run time of these methods and makes causal attributions of performance gains difficult. It even impairs the most attractive property of L.LMs (i.e., "scaling law"), as demonstrated in the MetaMath (Yu et al. 2023) and our experiments: more augmented data may hurt performance.
Another line of research studies prompting- bbased based methods, such as CoT (Wei et al., 2022), ToT YYo et al. 2024), ReAct (Yao al., 2022), and Reflexion (Shinn et al. 2024). They try to induce the potential reasoning capacities of LLMs with only in-context learning. While these methods are conceptually simple and training-free,, they often suffer from factual hallucination and error propagation over a mogghorizon reasoning trajectory, as demonstrated by Fig. (b). Therefore, in this paper. we ask: Can LLMs learn to execute system-2 mathematical reasoning on top of the established training paradigm (i.e., next-token prediction)?
A plausible solution to this problem is learning from human feedback (LHF)), which collects a very large-scale dataset of high-quality and multi-step reasoning trajectories from human feedback and then fine-tunes a pre-trained LLM on it via supervised reinforcement learning (Ouyang al. 2022 Joshi et al., 2023). Unfortunately, collecting such
dataset is prohibitively expensive and impractical due to (1) the lack of well-trained professional human annotators for
The general concept of System-2 reasoning is also employed by other work [LeCun 2022
Weston & Sukhbaatar 2023 Yu et al. 2024 which
involves generating deliherate and structured thought processes that enahle model (or human) to reason and plan effectively, therehy completing given task successfully.
2

---

Published in Transactions on Machine Learning Research (09/2025)
complex mathematical problems (Wang et al. 2020) and (2) the biased assessment of annotation's 's correctness (Huang al. 2023). Instead of learning from expensive human feedback, an alternative solution is to learn from Al feedback (LAIF) (Lee et al. 2023), which leverages powerful off-the-shelf LLMs, c.g., ChatGPT (Achiam et al., 2023). to automatically produce complete reasoning trajectories and evaluate their quality. Despite promising and scalable, LAIF reintroduces a chicken-and-egg dilemma similar to what this method trying to address from the beginning: L.MSs are not good at system-2 2 mathematical reasoning, but they are expected to learn such capabilities from their own (potentially low-quality) reasoning trajectories.
To better answer the above question, combine the strengths of LHF and LAIF in novel way that avoids their limitations. Concretely, we leverage the synergy of human and AI feedback to produce enriched instruction datasets consisting of fine-grained reasoning trajectories. These datasets are then used to fine-tune open-source LLMs. Such human-Al collaboration achieves a win-win result: high- human reasoning trajectories mitigate LLMs' hallucination issue while the almost infinite generative power of LLMSs reduces the workload of human annotators. As illustrated in 2 given an instruction-responee pair from any existing human-annotater mathematical dataset, our method EIT (standing for Enriched Instruction Tuning) first prompts an LLM generate plan decomposing the complex instruction into sequence of lower-level objectives. With the guidance of the high-level plan, the same LLM is then used to complement reasoning contexts between or within steps omitted in the original response through ERS prompting, leading to a smoother reasoning trajectory for LLM fine-tuning.
We evaluate EIT and compare it with finetuning-bast based, prompting-basee and tool-augmented methods on two widely used mathematical benchmarks, MATH and GSMSK. Specifically, EIT achieves an accuracy of 32.5% on the MATH dataset and 84.1% on GSM8K, marking an improvement of 2.7% on MATH over MetaMath (Yu et al., 2023). Notably, HIT matches the performance of methods using external tools on the GSM8K benchmark, even outperforming MathCoder (Wang et al. 2023a). We also demonstrated that more fine- fine-grained reasoning trajectories lead to better testing performance, shedding a novel insight that the completeness and quantity of data are equally important.
2 Related Work
Mathematical reasoning is a key aspect of human intelligence that enables us to comprehend and make decisions based on numerical data and language (Lu et al., 2022). This cognitive faculty is also an important factor in evaluating the capabilities of LLMs. Mathematical reasoning is still a great challenge for LLMs, which struggle with complex computations and symbolic manipulations. Prompt-based methods are proposed to improve reasoning capabilities.
Chain-of-thoughh prompting (CoT) (Wei et al. 2022. Yao al. 2024 Shinn et al. 2024 Yao al. 2022) proposes that LLMs can improve reasoning capabilities by generating reasoning chains by leveraging intermediate natural language reasoning trajectories as prompts. Some recent studies also proposed to select woontext-learnnng examples, since the chosen examples in prompts have a large impact on the accuracy and stability of reasoning (Rubin et al., [021] Zhang et al. 2022). (Jin et al. 2024) is prompting-basee method, which claims that more reasoning steps of CoT can achieve better performance. However, only using few-shot examples to prompt L.MSs to generate a longer reasoning trajectory may cause severe hallucination issues (Hagendorff et al. 2022).
Fine-tuning is another way improve reasoning capabilities, which collects very large-scale dataset of high-quality and multi-step reasoning trajectories from human feedback and then fine-tunes a pre-trained I.M on it via supervised or reinforcement learning (Ouyang et al. 2022 Joshi et al., 2023). Unfortunately, collecting such a dataset is prohibitively expensive and impractical due to (1) the lack of well-trained professional human annotators for complex mathematical problems [Wang et al. (020) and (2) the biased assessment of annotation's 's correctness (Huang et al., 2023). Instead of learning from expensive human feedback, an alternative solution is to learn from AI feedback (LAIF) (Lee al.
2023), which leverages powerful off-the-shelf LLMs e.g., ChatGPT (Achiam et al.. 2023 to automatically produce complete reasoning trajectories and evaluate their quality (Xu et al.) [2033] Yu et al. 2023 Liu et al. 2023]] Lu et al.
2024 Mitra et al. 2024). Despite being promising and scalable, LAIF cannot directly learn such capabilities from their own reasoning trajectories since they are not good at system-2 mathematical reasoning.
Different from prior works that focus on either LHF or LAIF, we leverage the synergy of human and AI feedback to produce enriched instruction datasets consisting of fine-grained reasoning trajectories. These datasets are then used to fine-tune open-source LLMs, thereby enhancing their own ability to execute system-2 -2 mathematical reasoning without any usage of external tools.
3

---

Published in Transactions on Machine Learning Research (09/2025)
3 Enriched Instruction Tuning
Instruction: Krista put 1 cent into her now bank on a Sunday moming. On Manday she put cants into her lank. On Tuesday she put cents into bank, and continusd to double the amount money she put into her bank each day for two weeks. On what day of the week did the total amount of money in her bank first erceed $2?
Enriched Response
O Original Response
To the day Krista's is bank exceeds $2, calculate sum of her dally doubled deposits as grometric series, Identify the smallest power of 2 $2 and count the days from Sunday to reach this total.
ERP:
The formula for a geometric scrites short
ERS: The lotai amount of money in her bank forms geometric series. The lormula for the sum of a goometric secies is where a is the first turm, r is the common ratio, and is the number of terms.
Taking to be the initial 1 l-nntt deposit be the number ul days Xaista badd money her bank so far, have inequality 1-2" =>x 2d 5 => S
Prompting GPT-4
In this case, the initial 1 cunt. deposit, r is (since she doubles the amount wuch day', and the number of days Kittt had money her ban's far. We want to find the smallest such that total amount money ber bank exceeds giv the inequality
1-2
The smallest power of that greater than 201 Thus, Sunday days away day 1.
The smallest of that grester than 201 is 2 Thus, n 8.
Since she started Sunday, and there total amount of money in hee bank first exceedac $2 is Sunday, which is days away from day 1.
week, the day the week when
Original Math Dataset, e.g.
Enriching
Our Enriched Instruction Dataset
Instruction Tuning
LLM capable of System-2 Math Reasoning
GSMBK
Figure 2: Pipeline of Enriched Instruction Tuning (EIT), which first leverages privileged LLM. e.g., GPT-4, to produce enriched reasoning for the existing mathematical instruction dutaset through our proposed ERP and ERS prompting methods, and then trains LLM on this enriched dataset via instruction tuning. Note that the Original Response provided by human annotators overlooks the bold context in Enriched Response, which however is critical problem--ooviing
system-2 reasoning is crucial for hu- mans or machines to solve complex mathematical problems (refer to our detailed discussion in Sec. [ Based on the exist- ing proven training paradigm. straightforward method to incorporate such reasoning capabilities into LLMs is to collect very large-scale dataset consisting of high-quality multi-step reasoning data from human feedback and to fine-tune a pre- trained LLM on this dataset via supervised or reinforcement learning (Ouyang et al. 2022 Joshi et al. 2023p. However, collecting high-quality multi-step reasoning data highly challenging: requires well-trained professional human anno- tators [Wang et al.||200. (2024). and on the other hand, the reliable and scalable assessment of feedback correctness remains open problem (Huang et al. (202)) An alternative method is learning from AI feedback (LAIF) (Lee et al.) [023] Yu et al. 2023: Lu et al. 2024), which leverages powerful off-the-shelf LLMs to generate complete reasoning trajectories and eval- uate their correctness in lieu of human annotators. Despite promising, LAIF cannot master system-2 reasoning capabili- ties before generating such data. Hence, how to generate high-quality, detailed reasoning data is still an open challenge.
According to Daniel Kahneman's 's theory of fast and slow thinking (Daniel 2017),
To this end, we combine the strengths of the two above methods in a novel way that avoids their limitations. Specifically, for existing human-annotated mathematical datasets, we prompt powerful L.MM (e.g., GPT-4) to fill in missing contextual information high-quality but usually sparse aanaan-anottted reasoning trajectories, forming series of enriched and fine-graine -he-grained "thought chains", which is then used to fine-tune an open-source LLM. Our approach, named Enriched Instruction Tuning (EIT), comprises two main steps: (1) generation of enriched instruction dataset and (2) LLM fine-tuning, which we will introduce in the following sections respectively. Fig. 2 jillustrates the main idea of EIT.
3.1 Generation of Enriched Instruction Dataset
Given human-annotated mathematical instruction datasets such as GSM8K (Cobbe et al. 2021) and MATH (Hendrycks) et al. 2021), the responses provided by human annotators are typically accurate yet sparse, omitting many of the implicit reasoning steps that humans adopt when solving complex tasks. These omitted steps may include the usage of common sense, identification of variable meanings, determination of causal relationships, and even the pre- and re-planning that humans and many animals implicitly execute in their brains (Daniel, 2017] LeCun] 2022 Hagendorff al. 2022). As demonstrated in Fig. (b), LLMs fine-tuned such datasets only learn to imitate intuitive and
4

---

Published in Transactions on Machine Learning Research (09/2025)
Table Comparison of augmentation strategies: EIT vs. other SOTA methods.
Methods
Accuracy on GSM8K (3)
Tool
Verifier/Cumector
PPO/DPO
Mixture Data Aug. Strategies
Qwen-Chat et
2024
76.4
MAmmoTTH Yue
76.9
DeepSeek Chat (B1
1124
86.7
/
LLEMMA (Azerbiyev
2123a
62.6
/
K
X
ToRA (Gou ef 2025
84,3
/
/
X
MathCoder Wang 0i
20233
83.9
WizardMath
[uo
81.6
MetaMath
et ml.
0123
82.3
X
CoSC-code
Gao et al.
21124
82.3
X
Ell (Ours)
84.1
quick-thinking system-1 reasoning. Moreover, relying human annotators write out the comprehensive reasoning trajectories for tackling each complex problem is prohibitively expensive and impractical (Wang et al. 2024).
To address this, we leverage powerful off-the-shelf LLMs to enrich the existing mathematical datasets with those missing reasoning contexts through few-shot prompting, The prompts we used for enriching MATH and GSM8K can be found in Example 3.2 and Example 3.3 Example 1.1imppee...pree provides a detailed comparison between the original responses provided by humans and those enriched using our method. worth noting that our proposed method strictly follows the original response without involving additional mathematical or symbolic calculations, maintaining the accuracy of human annotations while significantly mitigating LLMs' hallucinations on long long-horizon reasoning trajectories.
Concretely, we design two general prompting methods: (1) Enriching with Reasoning Plan (ERP) and (2) Enriching with Reasoning Step (ERS), which respectively consider the essential roles of planning and reasoning in human's 's System-2 thinking. Given a human-annotaaed instruction-responss pair from any existing mathematical dataset, ERP prompting leverages an LLM to generate plan decomposing the complex instruction into a sequence of lower-level objectives (see the red parts in Example 3.1.. With the guidance of the semantic plan generated by ERP, the same LLM is then used to fill in missing contextual information and complement reasoning steps omitted in the original response (see the blue parts in Example 3.1 through ERS prompting, which reduces implicitly omitted steps in the response caused by human annotators and results in more coherent reasoning trajectory for LLM finetuning. Table illuustrats the composition of our enriched instruction dataset (i.e., EITMath), which is constructed by prompting GPT-4 with our proposed ERP and ERS prompting methods based on MATH (Hendrycks et al.) 2021) and GSM8K (Cobbe et al. 2021) datascts.
We also conduct quality analysis of our EITMath by employing perplexity as a metric evaluate and calculating the accuracy of final (nseers(Aeerbayev al. 2023b Brown al. 2020). As discussed in Sec. 6.3] our analysis indicates that perplexity is inversely proportional accuracy. The perplexity of our generated data is significantly lower than that of the original answers, demonstrating the high quality of our generated data. More importantly, we verify output fidelity by comparing final answers (e.g., 60 in Example 3.1 enriched responses achieve 99.7% alignment with original answers, whereas GPT-4's direct outputs without ETT prompts show only 71.5% accuracy. These findings indicate that our EIT approach substantially enhances the quality of the generated responses and mitigates issues related to hallucinations which lead to wrong final answers.
3.2 Finetuning LLMs on Enriched Instruction Dataset
We train an open-source L.M on our collected ElT'Math dataset Dr = { v with token-level supervised finetuning (Ouyang et al. 2022). During training, M expected to produce the corresponding response yr given the specific instruction i, as input. The training objective involves next-token prediction, consistent with all decoder-only language models (Radford et al. 2018 Touvron al. 2023a b). The likelihood of target response is modeled below.
Par (w/l=t) = II Pa (1)))
in which v denotes the target response tokens before the current predicted token t,. Eq. can be optimized using the maximum likelihood estimation with mini-batch stochastic gradient descent (Kingma & Ba 2014)
5

---

Published in Transactions on Machine Learning Research (09/2025)
Example 3.1: Comparison of Original and Enriched Responses
Instruction: A yogurt shop sells four flavors of yogurt and has six different toppings. How many combinations of one flavor and two different toppings are available?
Original Response: There are 4 choices of flavor, and ( 15 ways to choose two of the six toppings. The total number of
combinations is 15 60
Enriched Response: To solve this, consider the number of ways choose one flavor from the four available and then calculate the number ways to choose two different toppings from the six available. Multiply these two numbers to find the total number of unique combinations.
There are 4 choices flavor. Since we are choosing toppings out 6, we can use the combination formula (*) e (-nn--) 1!! where n is the total numher of items, and is the number of items choose. In this case, 6 and 2, so we have ways to choose two of the six toppings. Finally, since the choice of flavor and the choice of toppings are independent, we multiply the number of choices for each get the total number of combinations. Therefore, the total number of combinations is 4 15
15
60
Example 3.2:
Prompt for Enriching MATH (ERP and ERS)
You are an answer enricher. hope you will refine your answers based on the questions and answers provided and avoid jumping your thinking. Pay attention to filling the missing contexts and logical gaps in existing answers. Besides, be careful not to change the original pathways or modify formulas and results, well as mathematical symbol formats. The first step is to give the idea of the answer, which helps the respondent to solve the question better. Finally, solve the problem.
question: Krista put cent into her new hank on a Sunday morning. On Monday she put cents into her hank. On Tuesday she put 4 cents into her bank, and she continued to double the amount of money she put into her bank each day for two weeks. On what day of the week did the total amount of money in her bank first exceed $2?
answer: The formula for geometric series is a-ur" Taking to be the initial 1-cent deposit and to be the number of days 1-r Krista had money in her bank so far, we have the inequality
2
200 = 2n 200
201 2"
The smallest power of 2 that is greater than 201 is 25. Thus, n = and Sunday is 7 days away from day 1.
enriched answer: <planning> To find the day Krista's 's bank exceeds $2, calculate the sum of her daily doubled deposits as geometric series. Identify the smallest power of over $2 and count the days from Sunday reach this total.
<solution> The total amount of money in her bank forms a geometric series. The formula for the sum of a geometric series is u=ar"
-I-- where a is the first lerm, r is common ratio, and is the number terms. this case, a is initial 1-cent deposit, r is 2 (since she doubles the amount each day), and n is the number of days Krista had money in her bank so far. We want to find smallest n such that total amount money in her bank exceeds $2, or 200 cents. This gives us the inequality:
2"
200 = 1 2" < 200 201 2".
1-2
The smallest power of 2 that is greater than 201 is 2.h Thus, = 8, Since she started on Sunday, and there are 7 days in week, the day of the week when the total amount of money in her bank first exceeded $2 is Sunday which is 7 days away from day 1.
question: (Q)
answer: [A]
enriched answer:<planning>
4 Experiments
In order to verify the effectiveness of EIT, we collect ElTMath (see Table and use it as the training dataset to fine-tune open-source LLaMA-2 (Touvron et al. 2023b 13B and 70B models and evaluate their testing performance on two popular benchmarks, MATH (Hendrycks al. 202T) and GSM8K (Cobbe et al. 2021).

---

Published in Transactions on Machine Learning Research (09/2025)
Example 3.3: Prompt for Enriching GSM8K (ERP and ERS)
You an answer enricher. hope you will refine answers based on the questions and answers provided. avoid jumping your thinking. Pay attention to filling the missing contexts logical gaps in existing answers. Besides, careful not to change original pathways modify formulas and results, as well as special symbol expressions, such as 4 x 15 155==000.00. The first step is to give the idea the answer, which helps question:Alber is wondering how much pizza he cat 16 slices and a small pizza has slices. If he cats it all, how many pieces does he cat that day? answer:He eats 32 from the largest pizzas because 2 x 16 x2* 12==32333
respondent solve the question better. Finally, solve the problem. day. Hc buys large pizzas and small pizzas. large pizza has
He eats from the small pizza because 2 x 8 42"8=16=16 He eats 48 pieces because 32 16 +32+16 182466488.-8 #### 48
enriched answer: <planning> To solve this, calculate the total number of slices in the large pizzas and the small pizzas separately, then add those numbers together to find the total number of slices Albert will cat. Aeluuton>AAbert purchases two large pizzas and two small pizzas for his day of pizza eating. Each large pizza is divided into 16 slices, thus he has 2 x 16 = <2* 12=32=332 slices from the large pizzas.
Similarly, each small pizza divided into slices. Therefore, from the small pizzas, By adding up the slices from both the large and small pizzas, find that Albert eats total of 32 16 = <32+16=48+48 slices of pizza one day. So, Albert consumes the pizzas bought, will have eaten 48 slices pizza a single day.
2 xx8= <228==1==16 slices.
#### 48
question: (Q)
answer: {A)
enriched answer:<planning
Table Ablation study ERP, ERS, and MetaMath MATH and GSM8K datasets.
Table 2: Our ElTMath Dataset.
ERP
ERS
MetaMath
MATH
GSM8K
Datasets
w/ ERP & ERS
14.9
67.3
17.2
72.0
Enriched MATH (endryycks 2021 Enriched GSM8K (Cobbe al. 2021
7.5k 7.5k
19.5
76.3
19.0
74.2
21.3
78.8
V
22.6
81.1
4.1 Datasets
MATH dataset collects a total of 12,500 competition-les mathematics problems, which are partitioned into 7,500 for training and 5,000 for testing. Each of them is accompanied by step-by-step solution and concludes with a distinct final answer, which is formatted for straightforward comparison with model-generated solutions. Moreover, the MATH dataset spans broad spectrum of subjects and difficulty levels, including seven categories: Prealgebra, Algebra, Number Theory, Counting and Probability, Geometry, Intermediate Algebra, and Precalculus.
GSM8K dataset is a diverse collection of grade school mathematical word problems, as recognized for its high quality. While is generally considered less challenging than the MATH dataset, provides more fine-graing mme-grained step-level solutions with basic arithmetic operations (addition, subtraction, multiplication, division). Following the same setting as prior works, we 7,473 and 1,319 problems for training and testing, respectively.
EITMath Table 2 illustrates the composition of our collected EITMath dataset, which contains 15k problems (7.5k from MATH and 7.5k from GSM8K) and the corresponding answers enriched with our proposed ERP and ERS prompting methods. The detailed construction pipeline can be found in Sec. 3.I and illustrated in Fig. 2
4.2 Experimental Setup
We use open-soures LLMs LLaMA-2 as the base model for fine-tuning. GPT-4-1106 enriched responses for constructing ErTMath. Following the prior work YYu al. 2023 Xu et al. 2023), adopt the
-weevoee---wwwwe-- is used generate
7

---

Published in Transactions on Machine Learning Research (09/2025)
AdamW optimizer to train the model with 3 epochs, and the learning rate is set to 2e-5. The batch size is 32 for the 70B model and 128 for the 13B model. 32 A100 GPUs are used to fine-tune the above models.
We conduct the ablation study on MATH and GSM8K datasets based on LLaMA-2-70B in Table 3 To make a fair comparison, we use the same dataset size of 7.5k as MATH and GSM8K's 's training dataset. Furthermore, we combine HIT with MetaMath (Yu et al. 2023), a method focusing on augmenting questions themselves, to explore whether combining them results in better performance. Considering other SOTA methods in Table f using larger datasets (e.g., MetaMath (Yu et al.) 2023) for 395k, MAmmoTH (Yue et al. 2023 for 260k), we also expand EITMath to 70k in order to make fair comparison. To this end, construct such dataset with more diverse responses by sampling GPT-4's output distribution with different temperature coefficients, which slightly improves EIT's performance.
5 Ablation Study
To evaluate the effectiveness of our proposed Enriching with Reasoning Planning (ERP) and Enriching with Reasoning Step (ERS) prompting methods. we conducted the ablation study on MATH and GSM8K datasets based on LLaMA-2- 70B. We construct ERP and ERS datasets separately by prompting GPT4. The prompts that used for ablation study can be found in Appendix 1.4.1 and Appendix A.4.2
Effects of ERP Table 3 shows the performance of baseline methods and the enhancements achieved by finetuning LLMs on enriched datasets with ERP prompting. The baseline exhibits an accuracy of 14.9% on the MATH dataset and 67.3% on the GSM8K dataset. Incorporating ERP results in a notable accuracy increase to 17.2% on MATH and 72.0% on GSM8K, corresponding to improvements of 2.3% and 4.7%, respectively. These results underscore the significant role of ERP in improving LLM's 's reasoning capabilities. By generating a high-level plan, ERP effectively decomposes complex instructions into lower-level objectives, thereby enhancing the coherence and direction of the ensuing reasoning process.
Effects of ERS The performance of the ERS is also quantified in Table ERS leads rise in accuracy from 14.9% to 19.5% on the MATH dataset, marking 4.6% improvement. Similarly, on the GSM8K dataset, the LLM's accuracy is propelled from 67.3% to 76.3%. obtaining a 9% improvement. This significant improvement can be attributed to the ERS's ability to bridge the gaps in implicit contextual information or steps omitted by human annotators, thereby improving LLM's reasoning capabilities. Notably, the improvement of GSM8K is greater than that of MATH, with an increase of 9%, which shows that ERS can more significantly improve performance on easier mathematical problems, such as those from the GSM8K, compared to the more challenging MATH.
Finally, when incorporated with ERP and ERS, our EIT yields the most pronounced performance gains, culminating in a peak accuracy of 21.3% on the MATH dataset and 78.8% on the GSM8K dataset.
Effects of Combining EIT and MetaMath There are various methods that emphasize the augmentation of questions with multiple strategies (e.g., MetaMath, WizardMath), while our EIT focuses on enriching the responses. Given the complementary nature of these approaches, we explored their combined effect by conducting ablation experiments with MetaMath and incorporating our proposed EIT. To ensure a fair comparison, we curated a subset of the augmented MATH data from MetaMathQA, matching the size of the original MATH dataset. similar approach was taken with the GSM8K dataset. The results, detailed in Table B indicate a notable performance enhancement when both question augmentation and response enrichment strategies are employed. Specifically, the model exhibits an incremental gain of 1.3% on the MATH dataset and 2.3% on GSM8K when compared to the response enrichment-only configuration. Moreover, the improvements are even more pronounced when EIT is applied to question augmentation-only configuration, with increases of 3.6% on MATH and 6.9% on GSM8K. These findings confirm that augmenting both questions and responses can substantially elevate the model's reasoning capabilities, with the combined approach yielding superior accuracy. Furthermore, the data suggests that our EIT contributes more effectively performance enhancement than question augmentation when the volume of data is held constant.
8

---

Published in Transactions on Machine Learning Research (09/2025)
Table 4: Comparison of testing accuracy existing LLMs GSM&K and MATH testing means that external tools used. f that
MetaMath is fine-tuned by LLoAA with batch of 128 from Yu 2023 while the full fine-tuned version from (Wag 2023b
Methods
Model Size
MATH
GSM8K
1
Touvron et al.
2023a
13B
3.9
17.8
LLaMA-2
Touvron et
2023a
13B
3.9
28.7
MPT
Team al.
2023)
30B
3.1
15.2
Falcon
Penedo et al.
2023
40B
2.5
19.6
Vicuna
(Chang et al.
2023
13B
27.6
WizardMath
(Luo et al.
2023
13B
14.0
63.9
MetaMath
Yu et al.
2023)
13B
22.4
72.3
EIT
13B
23.0
73.1
LLaMA-1 Touvron et al.
2023a)
65B
10.6
50.9
LLaMA-2
Touvron et
2023a
70B
13.5
56.8
Platypus-2
70B
15
45,9
WizardMath (Luo al.
2023
70B
22.7
81.6
MetaMath
Yu et al.
2023
70B
26.6
82.3
MetaMath Wang et al
2023b
70B
29.8
80.4
Qwen-Chat (Xu et al.
2024)
72B
31.8
76.4
EIT
70B
32.5
84.1
PAL(LLaMA-2) (Gou et al.
2023)
70B
18.3
55.2
MAmmoTH' (Yue et al. 2023)
70B
41.8
76.9
MathCoder Waag al. 2023a
70B
45.1
83.9
ToRA Gou et 2023)
70B
49.7
84.3
DeepSeek- Chat (Bi et al. 2024)
67B
51.1
86.7
CoSC-code
(Gao al. 2024)
34B
53.5
82.3
6 Comparison with State of the Art
6.1 Results on MATH and GSMSK
As shown in Table compared to the same open-source LLMs that do not use any external tools, such as code verification, EIT achieves the best performance. Our EIT-70B model achieves a leading accuracy of 32.5%, with a 2.7% improvement over MetaMath-70B and more than double the accuracy of the baseline LLaMA-2-70BB model. For the 13B model, our EIT model also achieves the best performance among models the same type, achieving 23% accuracy. These results underscore the superiority of our ERP and ERS methodologies in comparison existing approaches. To further analyze the improvement of method, show the accuracy of subtopies MATH in Table Our EIT exceeds WizardMath and MetaMath across all subtopics. Notably, EIT achieves 15.4% on the most challenging topic, Precalculus, surpassing MetaMath 4.6% and WizardMath 2.8%.
On the GSM8K benchmark, our EIT-70B records an 84.1% 1% accuracy. with 2.5% improvement over WizardMath and 1.8% over MetaMath with same parameters. Furthermore, 13B model demonstrates an even more remarkable performance, surpassing WizardMath by 9.2% and MetaMath by 0.8%. noteworthy that the performance of EIT-70B on GSM8K comparable to that models utilizing external computational tools, and even outperforms MathCoder. This underscores the significant potential of harnessing the intrinsic reasoning faculties of LLMs without reliance additional aids.
Table 5: Comparison of resting accuracy (%) of 70B models on MATH Suhtopics.
Methods
Intermediate Algebra
Precalculus
Geometry
Number Theory
Counting
Prealgebra
Algebra
Overall
WizardMath
7.1
12.6
15.7
16.3
17.3
41.7
33.3
22.7
MetaMath
14.28
10.8
20.9
23.7
24.3
46.9
44.56
29.8
EIT
15.0
15.4
25.9
26.7
28.7
50.0
47.1
32.5

---

Published in Transactions on Machine Learning Research (09/2025)
Accuracy on MATH (16)
MeraMathOA
Accuracy on MATH (96)
0.02 11.38
Perplexity
Accuracy
GSMBK (%)
35-
MeiaMaithQA + EITMach (70k)
21
88
Perplexity
70
25
19
63
Accurar Sacuracyy
60
55
15
G5Mesilwwcooo Metatlan
Dataset
Average Token Numbers Responses
Methods
Figure A.1: left: Scaling of performance on MATH as adding our EITMath dataset for LLM fine-tuning. Different colored bars represent MctaMathQA combined with Rejection sampling Fine- Fine-Tuning (RFT) and EITMath as the training set. middle: Scaling of performance on MATH us more fine-grained reasoning steps created for fine-tuingg We use average tokens (inore is betler) of the response to measure its granularity. right: Perplexity and accuracy of different methods on GSM8K. Following (Yu al. [2023). the perplexity is calculated using inder-finetunce LLaMA-2-7B, and the accuracy is reported based on our fine-tuned LLaMA-2-70BB 70B model on these datasets, It is clear that our ETTMath has less perplexity compared to other mathematical datasets, which leads to better performance.
6.2 Analysis from Scaling Law Perspective
Better Performance with More Data The scaling law of dataset size is the most attractive property of LLMs. However, recent findings from MetaMath demonstrate that more data is not always better. By combining the existing augmented dataset Rejection sampling Fine-Tuning (RFT) (Yuan et al. 2023) with MetaMathQA of different scales for finetuning, we found that more augmented data hurts performance Yu et al. 2023). In contrast, our method presents a different perspective. As shown in Fig. A.I left, we conducted a series of finetuning experiments on the LLaMA-2-700 model using different subsets of MetaMathQA data 20k, 40k, 80k, and 120k samples camples-ccmbineed combined with our EITMath dataset. When compared to the integration with RFT, our approach yields a markedly better performance. The inclusion of RFT consistently led to decline performance across all data scales. On the other hand, the incorporation of EITMath not only enhanced the model's performance but also demonstrated the benefits of additional data scale positively with performance, in line with the scaling law. This indicates that the quality of data augmentation and the strategic approach to dataset construction are pivotal factors. Our findings affirm that, based on EIT, more data can indeed lead to better performance in training LLMs.
Better Performance with More Fine-grained Reasoning Steps Observing the performance improvement brought by scaling dataset size, we raise a question: whether more fine-graing -re-grained reasoning steps can also enhance performance? To explore this, we curated multiple datasets derived from the MATH benchmark, each featuring different average token number of responses. These datasets were constructed using the same strategy as ElTMath. We control the granularity step-by-step reasonings by adjusting the prompts used when requesting responses from GPT-4. This includes specifying a desired number of tokens for the reply and carefully selecting few-shot examples that demonstrate the desired level of detail. By inereasing the granularity of enriched responses, depicted in Fig. A. middle, we observed a positive correlation between model performance and tokens for finetuning.
6.3 Analysis from Perplexity Perspective
Prior work (Yu et al. 2023) demonstrates that the simplicity of the data can better elicit the reasoning abilities of LLMs, because such data are easier for the models to learn from, leading to improved performance. The simplicity can be measured by perplexity (Marion et al., 2023). We calculate the average perplexity across all tokens in generated responses from our ElTMath using the LLaMA-2-7B model without any additional fine-tuning. The results show that the perplexity of ErTMath is lower than all other methods in Fig. A.Iright, and the LLM fine-tuned on ElTMath also achieves the highest accuracy on GSM8K.
6.4 Comparison between EIT and CoT
Our method significantly differs from the Chain-of-Thought (CoT) prompting method in several key aspects. While COT relies on generating intermediate natural language trajectories to improve reasoning capabilities using the model's "internal knowledge", EIT leverages human-annotated initial answers "meta-knowledge" guide the model in
10

---

Published in Transactions on Machine Learning Research (09/2025)
generating more detailed and smoother reasoning, making the task more similar to fill-in-the-bland question. This reduces the complexity of the task, as the model only needs to elaborate on the given answer rather than generate the entire reasoning chain from scratch. Additionally, compared to the standard CoT prompting method that requires complex and carefully selected few-shot examples, EIT's data construction process simplifies the prompting process by providing clear and structured initial answers, reducing the reasoning load on the model.
To further validate the effectiveness of our EIT, we conducted comparative analysis on fine-tuning and prompting methods, especially the CoT method. As illustrated in Table 6 the performance of prompting with few-shot examples including ERP and ERS is inferior to that of standard CoT prompting. This is because ERP and ERS place higher demands on the model's capabilities, like requiring more nuanced understanding of complex instructions and stronger reasoning abilities to generate more detailed and smoother reasoning trajectories. Conversely, fine-tuning with standard CoT performs significantly better than standard CoT prompting, particularly on the GSM8K dataset, where it outperforms CoT prompting by 11.5%. This improvement can be attributed to fine-tuning's ability to stimulate the model's CoT capabilities while avoiding the instability introduced by complex prompts.
Notably, our EIT achieves the best performance, with the accuracy of 32.5% the MATH dataset and 84.1% 1% on the GSM8K dataset, representing nearly 25.3% and 51.2% improvement compared to prompting with ERP and ERS and over 15% improvement than fine-tuning with standard COT. Unlike data constructed using the CoT method, EIT initial answers as "meta-knowledge" which helps LLMS generate more detailed and smoother reasoning reasoning trajectories. This approach effectively mitigates the accumulation of model hallucinations as the length of inference increases, thereby ensuring the accuracy of our constructed data.
further leverages
Table 6: Comparison of accuracy (%) of prompting and fine-tuning methods.
Methods
Model Size
MATH
GSM8K
LLaMA-2 (prompting with standard Co'I)
70B
14.4
57.8
L.aMM-22 (prompting with few-shot examples including ERP and FRS)
70B
7.2
32.9
LLaMA-2 (fine-tuning with standard CoT)
70B
14.9
69.3
EIT (ours)
70B
32.5
84.1
6.5 Analysis Case Study
Example 6.1 shows a randomly selected case including responses generated by SFT (supervised fine-tuning using publicly available training sets), MetaMath, and our EIT on the test set of MATH. It is obvious that the first half of the solution from SFT is correct, but there was an error when calculating 13 36 due to merging calculations. As for MetaMath, unlike SFT, it divides 3 yards into 3 and ] to calculate inches respectively and find their sum finally. MetaMath aims convert 1/4 yards into inches by utilizing the yard-foot and foot-inch relationships. However, it neglects the relationship between yards and foot, which leads to hallucinations finally. In contrast to the above methods, firstly, our EIT illustrates ERP accurately, guiding the subsequent steps to convert the mixed number to an improper fraction and then multiply by the conversion factor that relates yards to inches. Secondly, EIT demonstrates ERS following the guidance of ERP, which illustrates the conversion factor as the base knowledge and multiplies the converted improper fraction with the conversion factor. Furthermore, EIT divides x 36 into 13 36 and then divides by 4, which avoids the calculation error that occurs in SFT. As result, EIT gets the final correct answer. More cases can be found in Example A.I and Example A.2
7 Conclusion
In this paper, we aim to build an LLM capable of system-2 mathematical reasoning. To achieve this, we introduce Enriched Instruction Tuning (EIT), which leverages the synergy between human and AI feedback to produce enriched instruction datasets consisting of fine-grained reasoning trajectories. These datasets are then utilized to fine-tune open- source LLMs, thereby boosting their inherent ability to execute system-2 mathematical reasoning without any usage of external tools. Compared with finetuning based, prompting-basee based, and tool-augmented methods, LLMs fine-tuned by EIT achieve competitive performance widely used mathematical benchmarks, i.e., MATH and GSM8K.
11

---

Published in Transactions on Machine Learning Research (09/2025)
Example 6.1: Case Study in Test Set of MATH
Question: How many inches are 3] yards? (Ground truth 117)
SFT: To find out how many inches are in 3 yards, we can use the conversion factor that 1 yard equals 36 inches. First, we convert mixed number 3 yards to an improper fraction. This gives us + 7 = 13 yards. Next, we multiply this fraction by the conversion factor, which is 36 inches per yard. This gives us:
13
216 inches.
So, there 216 inches 3 yards. (X)
MetaMath: There are 36 inches in yard. So, there are 3 x 36 = 108 inches in yards. There are 12 inches in 1 foot. So, there 12
12 inches in foot. Therefore, there are 12 x 12 inches in 1 yards.Adding the inches in 3 yards and
1 yards, we get 108 + 12 = 120 inches. The answer is: 120. (X)
ETT: We need to convert 3 yards into inches. To solve the problem, convert the mixed number improper fraction to represent the yards, then multiply by the conversion factor that relates yards to inches. Simplify resulting fraction to find number of inches.
To convert yards to inches, we use the conversion factor that yard equals 36 inches. We start by expressing 3 yards as a mixed number and then convert it improper fraction: 3
13 Now, multiply the fraction by the conversion factor:
13
13 x 36
x 36
4
4
We can simplify this by multiplying by 36, which gives 468, and then dividing by 4: 117. Therefore, 3 yards are equal to 117 inches. The answer is: 117. (V)
Moreover, find that more fine-gr anee-graied reasoning trajectories consistently lead higher testing accuracy, suggesting that the granularity and size of reasoning datasets are likely to be equally important for training a math L.M expert.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics. arXiv preprint arXiv:2310. 10631, 2023a.
Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Mareo Dos Santos, Stephen Mareus McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics. ArXiv, abs/2310.1103// 2023b. URL /ttht::///ppp. www.ntssscoollaa.nc.... http://oo.....oororrrorro////222
Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, al. Deepseek I1m: Scaling open-source language models with longtermism. arXiv preprint
MmXiv:240.00294.. 2024.
David Brandfonbrener,, Sibi Raja, Tarun Prasad, Chloe Loughridge, Jianang Yang, Simon Henniger, William E Byrd, Robert Zinkov, and Nada Amin. Verified multi-step synthesis using large language models and monte carlo tree search. arXiv preprint arXiv:2402.081477 2024.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: laalon-langage-acttoon models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818 2023.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voon Voss, Gretchen Krueger, Tom
12

---

Published in Transactions on Machine Learning Research (09/2025)
Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Ma teusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. ArXiv, abs/2005.14165. 2020. URL http:s///p....eaaaatttccoooororr
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tar Lec. Yuanzhi Li. Scott Lundberg, al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.122 12712, 2023.
Ziru Chen, Michael White, Raymond Mooney, Ali Payani, Yu Su, and Huan Sun. When is tree search useful for Ilm planning? it depends on the discriminator. arXiv preprint arXiv:2402. 10890, 2024.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuma: An open-source chathot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. Imsys. org (accessed 14 April 2023), 2023.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers solve math word problems. arXiv preprint
1rXiv:2110.4416: 2021.
Kahneman Daniel. Thinking, fast and slow. 2017.
Daocheng Fu, Xin Li, Licheng Wen, Min Dou, Pinlong Cai, Botian Shi, and Yu Qiao. Drive like human: Re- thinking autonomous driving with large language models. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 910-919, 2024.
Kuofeng Gao, Huanqia Cai, Qingyao Shuai, Dihong Gong, and Zhifeng Li. Embedding self-correction an inherent ability in large language models for enhanced mathematical reasoning. arXiv preprint MrXIi:2410010735. 2024.
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yujiu Yang, Minlie Huang, Nan Duan, Weizhu Chen, et al. Tora: A tool-integrated reasoning agent for mathematical problem solving. arXiv preprint a7X442230..17452, 2023.
Thilo Hagendorff, Sarah Fabi, and Michal Kosinski. Thinking fast and slow in large language models. arXiv preprint arXiv:2212.05206. 2022.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021.
Yann Hicke, Anmol Agarwal, Qianou Ma, and Paul Denny. Chata: Towards intelligent question-ans teaching assistant using open-source 1lms. arXiv preprint arXiv:23 11.021751 2023.
Fan Huang, Haewoon Kwak, and Jisun An. Is chatgpt better than human annotators? potential and limitations of chatgpt in explaining implicit hate speech. In Companion proceedings of the ACM web conference 2023, pp. 294-297, 2023.
Mingyu Jin, Qinkai Yu, Haiyan Zhao, Wenyue Hua, Yanda Meng, Yongfeng Zhang, Mengnan Du, et al. The impact of reasoning step length on large language models. arXiv preprint S.XXv:24001.49255, 2024.
Nitish Joshi, Koushik Kalyanaraman, Zhiting Hu, Kumar Chellapilla, He He, and Li Erran Li. Improving multi-hop reasoning in Ilms by learning from rich human feedback. In Neuro-Symbolic Learning and Reasoning in the era of Large Language Models, 2023.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiy preprint arXiv:2001.08361, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1iXiv:1412.6980. 2014.
13

---

Published in Transactions on Machine Learning Research (09/2025)
Yann LeCun. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review, 62(1):1-66 2022.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop. Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267. 2023.
Bingbin Liu, Sebastien Bubeck, Ronen Eldan, Janardhan Kulkarni, Yuanzhi Li, Anh Nguyen, Rachel Ward, and Yi Zhang. Tinygsm: achieving> 80% on gsm8k with small language models. arXiv preprint arXiv:2312.09241, 2023.
Pan Lu, Liang Qiu, Wenhao Yu, Sean Welleck, and Kai-Wei Chang. A survey of deep learning for mathematical reasoning. arXiv preprint arXiv:2212.10535, 2022.
Zimu Lu, Aojun Zhou, Houxing Ren, Ke Wang, Weikang Shi, Junting Pan, Mingjie Zhan, and Hongsheng Li. Mathgenie: Generating synthetic data with question back-translation for enhancing mathematical reasoning of llms. arXiv preprint arXiv:2402.16352. 2024.
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint NoXiv:2338.095880 2023.
Max Marion, Ahmet Üstün, Luiza Pozzobon, Alex Wang, Marzieh Fadace, and Sara Hooker. When less is more: Investigating data pruning for pretraining Iims scale. arXiv preprint 2XXi:23309005644 2023.
Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah. Orca-math: Unlocking the potential of slms in grade school math. arXiv preprint arXiv: 14830, 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon Ilm: outperforming curated corpora with web data, and web data only, arXiv preprint arXiv:2306.01116, 2023.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. Learning retrieve prompts for in-context learning. arXiv preprint arXiv:21 12.0v:21 12.08633,
2021.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
MosaicML NLP Team al. Introducing mpt-7b: new standard for open-source, commercially usable 1lms, 2023. URL www. mosaicml. com/blog/mpt-7b. Accessed, pp. 05-05, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. 2023b.
Dakuo Wang, Elizabeth Churchill, Pattie Maes, Xiangmin Fan, Ben Shneiderman, Yuanchun Shi, and Qianying Wang. From human-human collaboration human-ai collaboration: Designing ai systems that can work together with people. In Extended abstracts of the 2020 CHI conference human factors in computing systems, pp. 1-6, 2020.
14

---

Published in Transactions on Machine Learning Research (09/2025)
Ke Wang, Houxing Ren, Aojun Zhou, Zimu Lu, Sichun Luo, Weikang Shi, Renrui Zhang, Linqi Song, Mingjie Zhan, and Hongsheng Li. Mathcoder: Seamless code integration in llms for enhanced mathematical reasoning. arXiv preprint arXiv:2310.03731, 2023a.
Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Y Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce Ilms step-by-step without human annotations. CoRR, abs/2312.08935,, 2023b.
Xinru Wang. Hannah Kim, Sajjadur Rahman, Kushan Mitra, and Zhengjie Miao. Human- -Im collaborative annotation through effective verification of Ilm lahels. In Proceedings of the CHI Conference Human Factors in Computing Systems, pp. 1-21, 2024.
Xuezhi Wang and Denny Zhou. Chain-of-thoughh reasoning without prompting. arXiv preprint arXiv:2402.10200, 2024.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou, Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203. 2022.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting clicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-22 24837, 2022.
Jason Weston and Sainbayar Sukhbaatar. System 2 attention (is something you might need too). arXiv preprint arXiv:2311.11829 2023.
Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min- Yen Kan, Timothy P Lillicrap. Kenji Kawaguchi, and Michael Shieh. Monte carlo tree search boosts reasoning via iterative preference learning. arXiv preprint arXiv:2405.00455 00045,, 2024.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. WizardIm: Empowering large language models to follow complex instructions. arXiv preprint urXiv:2304.12244 2023.
Yifan Xu, Xiao Liu, Xinghan Liu, Zhenyu Hou, Yueyan Li, Xiaohan Zhang, Zihan Wang, Aohan Zeng, Zhengxiao Du, Wenyi Zhao, et al. Chatglm-math: Improving math problem-solving in large language models with a self-critique pipeline. arXiv preprint arXiv:2404.02899, 2024.
Yijun Yang, Tianyi Zhou, Kanxue Li, Dapeng Tao, Lusong Li, Li Shen, Xiaodong He, Jing Jiang, and Yuhui Shi. Embodied multi-modal agent trained by an Ilm from a parallel textworld. CVPR, 2024.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint
2022.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint mrXiv:2309.12288 2023.
Ping Yu, Jing Xu, Jason Weston, and Ilia Kulikov. Distilling system 2 into system 1. arXiv preprint arXiv:2407.06023, 2024.
Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308.01825, 2023.
Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning. arXiv preprint arXiv:2309.00 05653, 2023.
15

---

Published in Transactions on Machine Learning Research (09/2025)
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493, 2022.
Siyu Zhou, Tiayyi Zhou, Yijun Yang, Guodong Long, Deheng Ye, Jing Jiang, and Chengqi Zhang. Wall-e: World alignment by rule learning improves world model-based Ilm agents. arXiv preprint arXiv:2410.07484, 2024.
A
Appendix
Example A.I: Case Study in Test Set of MATH
Question: A right triangle has leg lengths of 8 and 15 units. What is the length, in units, of the median to the hypotenuse?
17
Express your answer as a common fraction, (Ground truth is
SFT: Let of the legs of the triangle be AB, and let the hypotenuse be AC. Let the length of the median be AM. Since AABM is a right triangle, we can use the Pythagorean Theorem to find AM:
AB + BM = AM²
64 + BM = AM²
BM AM 64
64
BM AM
AM
BM 15
64
2
V
152
15
15
BM 2 2
4
(X)
MctaMath: First, we need to find the length of the hypotenuse. Using the Pythagorean theorem, we have 8 + 152 = c², where is the length of the hypotenuse. hypotenuse.Smmplitingg
we get 64 + 225 = c², or e² 289. Taking the square root of both sides, we find c 17. Now, we need to find the length of the median the hypotenuse. The median to the hypotenuse is the line segment that connects the midpoint of the hypotenuse to one of the legs of the triangle. Since the triangle is right triangle, the median to the hypotenuse is also the altitude of the triangle. Using the Pythagorean theorem again, have 8 + (17/2)² 15a. Simplifying. we get 64 + 289/4 /4- 225. Multiplying both sides by 4, we get 256 + 289 900. Subtracting 256 from both sides, we find 289- 644. Taking the square of both sides, we get v /289- /644. Simplifying, we find 17 25. Therefore, the length of
17
the median to the hypotenuse is
The answer is: 17 (X) 25
25
The solution involves recognizing that the median the hypotenuse of right triangle half the length the hypotenuse. By using the Pythagorean theorem, we can find the length of the hypotenuse and then divide it by 2 to get the length of the median. First, we use the Pythagorean theorem find length of hypotenuse. The theorem states that in a right triangle, the square of the length of the hypotenuse (the side opposite the right angle) is equal to the sum of the squares of the lengths of the other two sides. For our triangle with leg lengths 8 15 units, the hypotemuse length h given by:
+ 15t /
/289-
The median to the hypotenuse of a right triangle is always half the length of the hypotenuse. Therefore, the length of the median m is:
17 8.5.
m
17
Thus, the length of the median to the hypotenuse, expressed as common fraction, is
units. The answer is: 47 (V)
2
16

---

Published in Transactions on Machine Learning Research (09/2025)
Example A.2: Case Study in Test Set of MATH
Question: At a driver education class, the students were told to put their hands on the steering wheel on "3" and "9", just as on the face of clock. If, instead, the students placed their hands on the "2" and the "10", by how many degrees would the angle between the hands (with the center of the wheel at the vertex of the angle) decrease? (Ground truth is 60")
SFT: The angle between the hands 360 90" 270" "3" and "9". If the students place their hands on the "2" the "10". the angle between the hands is 360 20 3/00: Therefore, the angle between the hands decreases by 340 270 degrees. (X)
70
MetaMath: The angle between the hands on "3" and "9" 90 degrees. The angle between the hands on "2" and "10" is 180 degrees. So, the angle between the hands on "2" and "10" is 180 - 90 = 90 degrees less than the angle between the hands on "3" and Therrher,,,,,, the angle between the hands on "2" and "10" would decrease hy 90 degrees. The answer is: 90 (x) ETT: The idea is calculate the initial angle hetween the 3 and 9 hands and then calculate the new angle hetween the 2 and 10 hands. The difference between these two angles will give us the decrease in the angle.
Initially, the hands are the 3 and 9 positions, which are hours apart on the clock. The angle between any two adjacent numbers on a clock is 360 /12 = 30" Therefore, the initial angle between the 3 and 9 hands is 6 x 30 = 180".
When the hands are moved the and 10 positions, they are hours apart. Again, using the fact that the angle between any two adjacent numbers on a clock is 30 the angle between the and 10 hands is 8 x 30 = 240"
The decrease in the angle is the difference between the initial angle and the new angle, which 180 240 -60%. Since we are looking for decrease, we take the absolute value, which is 60% = 60"
Therefore, the angle between the hands decreases by 60"
Note: The negative sign in the calculation indicates that the angle decreasing, and taking the absolute value gives us the positive decrease degrees.
The The answer is: 60%. (V)
A.1 More Results on Self-Consistency
Self-consistency (Wang et al., 2022) helps to boost models' performance, especially arithmetic benchmarks such GSM8K. We investigate the impact of scaling decoding samples and observe that EIT outperforms other methods, demonstrated in Table Z ToRA, with external tools, performs comparably to EIT without self-consistency decoding. Applying self-consistency decoding with 50 samples, ToRA attains 4% improvements from 84. %% to 88.3%. In contrast, our EIT outperforms ToRA and achieves 88.4% accuracy with only samples. Notably, our EIT reaches 90.4% accuracy with 50 decoding samples, representing a 6.3% improvement and surpassing ToRA by 1.1 These results underscore that ETT demonstrates superior reasoning capabilities and robustness compared to existing methods.
Table 7: Comparison of lesting accuracy (%) of more results with ceffscoonistency. (k=50) denotes selfccomniteency/mmjorriyy voting decoding with 50 samples. Notably, EIT SC (k=5) surpasses ToRA + SC (k=50).
Methods
Model Size
GSM8K
ToRA Gou et al.
2023
70B
84.3
ToRA + (k=50)
70B
88.3
EIT
70B
84.1
EIT + SC (k=5)
70B
88.4
ETT + SC (k=10)
70B
89.9
EIT + SC (k=50)
70B
90.4
A.2 Discussion on Datasets Only with Final Answers
Considering that a small portion of datasets contains only the final answers without accompanying reasoning, generating enriched instruction datasets becomes more challenging. We plan to extend our method to such datasets in future work. The potential approach involves a two-stage process. First, LLMs will be prompted to generate sparse responses, those that lead to the correct final answer will kept for the second stage. In the second stage, we enrich the reasoning trajectories progressively in several iterations and use the COT-decoding (Wang & Zhou 2024 to choose reasoning trajectories with higher confidence.
17

---

Published in Transactions on Machine Learning Research (09/2025)
A.3 Pattern Analysis on The Enriched Responses
We performed a pattern analysis on the token length of responses across increasing difficulty levels the MATH dataset. The results are shown in T Our analysis indicates that as the difficulty increases, the length of the responses generated by EIT also increases. Since the responses of simpler questions tend to have more concise explanations and can overlook some steps, the token length for human annotations is relatively short, resulting in larger length ratios for EIT, vice versa. This result demonstrates that EIT can adapt to the difficulty level of the problem and the human annotations, and adaptively enrich necessary details, highlighting the effectiveness of our approach.
Table Pattern analysis the token length responses increasing difficulty levels the MATH dataset
Difficulty Level
2
3
4
5
Human-Amnoaate
54
68
161
152
EIT
240
248
273
345
362
Length Ratio (EIT/ / Human-Annotated
4.2
4.6
4.0
2.1
2.4
A.4 Prompt for Ablation Study of ERP and ERS
We present specific instructions and example few-shot prompts of our EIT's ablation study,
A.4.1
Prompt for Enriching MATH (ERP)
The prompt of ERP for generating a high-level plan for MATH dataset can be found in Example A.3 which is used for ablation study in Sec 5
Example A.3: Prompt for Enriching MATH (ERP)
You are an expert answer assistant who helps respondents understand the given answer. Based on the question and answer provide, your task generate high-level outline or plan of the answer. This plan will guide the respondent formulating their own detailed response and serve as preparatory framework for addressing the question effectively. question: Krista put cent into her new bank Sunday morning. On Monday she put cents into her bank. Tuesday she put 4 cents into her bank, and she continued to double the amount of money she put into her bank cach day for two weeks. On what day of the week did the total amount of money in her bank first exceed $2?
answer: The formula for geometric series is e-ar", Taking to be the initial 1-cent deposit and be the number of days Krista had money in her bank so far, we have the inequality
2"
1 1 2 200 = 1- 2 -00 201 2".
The smallest power of that is greater than 201 25, Thus, n and Sunday iis days away from day
planning:To To find the day Krista's 's hank exceeds $2, calculate the sum of her daily doubled deposits as geometric series. Identify the smallest power of 2 over $2 and count the days from Sunday to reach this total.
question: (Q)
answer: (A)
planning:
A.4.2 Prompt for Enriching MATH (ERS)
The prompt of ERS for enriching MATH dataset can be found in Example A.4
A.4 which is used for ablation study in
See 5
18

---

Published in Transactions on Machine Learning Research (09/2025)
You Prompt for Enriching MATH (ERS)
Example A.4:
enricher. hope you will refine your answers based the questions and provided and avoid jumping your thinking. Pay attention to filling the missing contexts and logical gaps in existing answers, Besides, be careful not to change original pathways modify formulas and results, well mathematical symbol formats.
question: Krista put cent into her new bank on Sunday morning. On Monday she put 2 cents into her bank. On Tuesday she put cents into her bank, and she continued to double the amount of money she put into bank each day for weeks. On what day of the week did the total amount of money in her bank first exceed $2?
answer: formula
geometric series d-ar" Taking to initial 1-cent deposit and I-r
number of days
Krista had money in her bank so far, we have the inequality
2"
200 - - 2n < -200 = 201 2".
1
The smallest power of 2 that is greater than 201 is 2 Thus, 8 and Sunday is 7 days away from day 1.
enriched answer:The total amount of money in her bank forms geometric series. The formula for the sum of geometric series term, is common ratio, and n the number terms.
e-ar". [-r where a the is 2 (since she doubles the amount each day), and is the number of days Krista had money in her bank far. We want to find the smallest n such that the total amount money in her bank exceeds $2, or 200 cents. This gives us the inequality:
case, a the initial 1-cent deposit,
2"
200
2
200 201 2"
The smallest power of 2 that is greater than 201 is 2.h Thus, 8. Since she started Sunday, and there are 7 days in week, the day of the week when the total amount of money in her bank first exceeded $2 is Sunday which is 7 days away from day 1. question: (Q)
answer: (A)
enriched answer:
19


Reviews:
[{"review_id": "shUBxQ063c", "reviewer": "Reviewer_Tp8P", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The proposed method is simple and effective. The prompts in this work represent a nice direction to make human responses more learnable while avoiding too much additional noise from model generations.\n2. Overall, this is a well-written paper with interesting ablations.\n\nWeaknesses:\n1. There is no quantitative analysis on how good these prompts are. I understand that the goal is to avoid adding any \"additional mathematical or symbolic calculations\" (as mentioned in Sec. 3.1), but the prompts will add details to fill in the gaps in human answers, so some of the prompts will still include errors. It will be useful to do (even at a relatively small scale) a human study to check the quality of rewritten responses. It would also be very valuable if there is any pattern in human responses that the proposed approach is good (or bad) at.\n2. There are a few small inclarities about the experiment details:\n    1. How do you control the length or the granularity of the outputs for the analysis in Sec. 6.2?\n    2. Is the perplexity average or max perplexity over the tokens?\n    3. The scaling improvement in Figure A.1 looks quite small. Do you run multiple seeds or compute statistical significance of the improvement?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a prompting method to rewrite human-written math data to improve LLM's math ability. More specifically, the idea is to use one prompt (ERP) to generate planning and the other prompt (ERS) to add more details to human-written answers. The method works well empirically and outperforms several strong augmentation or CoT methods on the GSM8k and the MATH benchmark. This work also includes interesting analyses studying different factors (e.g., instruction length, perplexity, etc).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Add a detailed analysis (beyond the three case studies shown in the current paper) on the answer quality.\n2. Fix issues mentioned in point 2 above.", "broader_impact_concerns": "No ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "oWyoNPTzyO", "reviewer": "Reviewer_kvPW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The paper is well-written. The motivation is generally clear and the proposed approach is easy to understand and reproduce.\n\n2. It is reasonable to enrich the sparse human-annotated dataset with high-level plans as well as fine-grained execution steps. The implementation recipe is thoughtful --- the proposed method strictly follows the original response without involving additional mathematical or symbolic calculations, to maintain the accuracy of human annotations while significantly mitigating LLMs’ hallucinations on long-horizon reasoning trajectories.\n\n3. There are some interesting observations which may be beneficial for related studies: (i) the perplexity is inversely proportional to accuracy; (ii) more fine-grained reasoning trajectories consistently lead to higher testing accuracy.\n\nWeaknesses:\n\n1. My major concern is that only LLaMA-2 is used as the base model for fine-tuning. As the enriched datasets are feasible to be adopted for a wide range of LLMs. It is not clear whether the improvements hold on more powerful LLMs such as Llama-3 as well as the latest o1/R1-style augmented reasoning models. I like the idea of enriching the dataset with ERP and ERS. It would be inspiring to identify the effectiveness of the approach on more latest LLMs.\n\n2. This paper has mentioned system-2 for several times and implied that existing studies lack system-2 deliberate reasoning abilities. On the one hand, the claim might be misleading as recent reasoning models have been widely adopted long CoT to induce fine-grained reasoning steps. On the other hand, though this paper claims to leverage the synergy of human and AI feedback, the approach is more likely to be a data augmentation approach with abstract planning and more fine-grained reasoning steps based on top of the original human annotations, instead of the interactive improvements between human and AI collaboration.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Enriched Instruction Tuning (EIT) for complex mathematical reasoning inspired by system-2. The key idea of EIT is leveraging the synergy between human and AI feedback to produce enriched instruction datasets consisting of fine-grained reasoning trajectories. Concretely, EIT comprises two main steps: (1) generation of enriched instruction dataset and (2) LLM fine-tuning, which we will introduce in the following sections respectively. The enriched instruction datasets are utilized to fine-tune open-source LLMs (i.e., LLaMA-2), thereby boosting their inherent ability to execute system-2 mathematical reasoning without any usage of external tools. Experimental results on MATH and GSM8K show that LLMs fine-tuned by EIT achieve competitive performance compared with fine-tune, prompting-based, and tool-augmented baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "More latest LLMs should be involved to justify if the improvements hold on top of more powerful LLMs as well as o1/R1-style reasoning LLMs.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IWYEKpbZFx", "reviewer": "Reviewer_Uuqu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n- The paper’s methodology is motivated by system 2 reasoning. \n- The experiments show improvement in performance over baselines.\n- The paper is easy to read with good case studies.\n\n**Weaknesses:**\n\nThe paper claims to “combine the strengths of LHF and LAIF in a novel way that avoids their limitations”. However, I am not convinced that this is true.\n- The limitation of LHF is the prohibitively expensive and impracticality in collecting a high quality human annotated dataset. EIT’s ERS step augments human-annotated instruction-response pairs, which suggests that the method still relies on high-quality human annotation for datasets, thus suffers from the same downsides outlined by the authors. \n- The limitation of LAIF is LLM hallucinations. Which the paper claims to mitigate because the “proposed method strictly follows the original response without additional mathematical or symbolic calculations”. However, from my understanding, this constraint is enforced only through prompting, which does not guarantee the elimination of hallucinations. Especially since ERP and ERS generates \"implicit steps omitted by humans\", which has no clear verification that these steps are factual and not hallucinated.\n- Another limitation of LAIF is that they “cannot master system-2 reasoning capabilities”. However, EIT’s ERP step is prompting GPT4 to generate low-level objectives through CoT with few-shot examples, which requires system-2 reasoning.\n\nThe paper states that Auto-CoT is the most closely related work, but no experiments are conducted to make empirical comparisons. Instead, Auto-CoT is dismissed for potentially causing hallucinations, without substantive evaluation.\n\nThe proposed method does not perform as well as tool-augmented LLMs and the paper does not justify enough why fine-tuning is more favorable than tool-augmented LLMs.\n\nThe reported results can be made clearer. \n- In Section 3.1, the paper states, “We compared the final answers of the generated data with the original answers to calculate accuracy. Our results show an accuracy rate of 99.7% for the generated answers, whereas the accuracy of GPT-4’s final answers without our EIT prompts is only 71.5%.” However, the meaning and implications of these accuracy figures are unclear.\n- The numbers reported in Table 3 are lower than those reported elsewhere in the paper, creating ambiguity about which model achieved 84.1% accuracy.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Enriched Instruction Tuning (EIT), which prompts GPT-4 using Enriching with Reasoning Plan (ERP) and Enriching with Reasoning Step (ERS) to generate a dataset that augments human annotations with system 2 reasoning steps. The new augmented dataset is then used to fine-tune a large language model, which shows improvement in performance over other fine-tuning baselines on GSM8K and MATH.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Summary of weaknesses to address:\n- Provide better analysis for the claims that the proposed method mitigates the limitations of LHF and LAIF.\n- Provide empirical analysis to Auto-CoT\n- Clarify reported results.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)