Can LLMs Solve Longer Math Word Problems Better?
Authors: Xin Xu
Tong Xiao
Zitong Chao
Zhenya Huang
Can Yang
Yang Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Models
Math Reasoning
Long Math Word Problems
Primary area: foundation or frontier models, including LLMs
TL;DR: We investigate the impact of long contexts on mathematical reasoning abilities of large language models.

Abstract: Math Word Problems (MWPs) play a vital role in assessing the capabilities of Large Language Models (LLMs), yet current research primarily focuses on questions with concise contexts. The impact of longer contexts on mathematical reasoning remains under-explored. This study pioneers the investigation of Context Length Generalizability (CoLeG), which refers to the ability of LLMs to solve MWPs with extended narratives. We introduce Extended Grade-School Math (E-GSM), a collection of MWPs featuring lengthy narratives, and propose two novel metrics to evaluate the efficacy and resilience of LLMs in tackling these problems. Our analysis of existing zero-shot prompting techniques with proprietary LLMs along with open-source LLMs reveals a general deficiency in CoLeG. To alleviate these issues, we propose tailored approaches for different categories of LLMs. For proprietary LLMs, we introduce a new instructional prompt designed to mitigate the impact of long contexts. For open-source LLMs, we develop a novel auxiliary task for fine-tuning to enhance CoLeG. Our comprehensive results demonstrate the effectiveness of our proposed methods, showing improved performance on E-GSM. Additionally, we conduct an in-depth analysis to differentiate the effects of semantic understanding and reasoning efficacy, showing that our methods improves the latter. We also establish the generalizability of our methods across several other MWP benchmarks. Our findings highlight the limitations of current LLMs and offer practical solutions correspondingly, paving the way for further exploration of model generalizability and training methodologies.

Published as a conference paper at ICLR 2025
CAN LLMs SOLVE LONGER MATH WORD PROBLEMS BETTER?
Xin Xu' Tong Xiao²*, Zitong Chao', Zhenya Huang", Can Yang", Yang Wang'
The Hong Kong University of Science and Technology
University of Science and Technology of China
(xxuca, zchaoaa cownectt.....c.................. ust. hk, {macyang, wangyang) @ust. hk tongriao20C22mail.nomm ustc. edu. huangzhyeustc. edu, cn https///gttbbbbomxxxxxxtttttttto
ABSTRACT
Math Word Problems (MWPs) play a vital role in assessing the capabilities of Large Language Models (LLMs), yet current research primarily focuses on questions with concise contexts. The impact of longer contexts on mathematical reasoning remains under-explored. This study pioneers the investigation of Context Length Generalizability (CoLeG), which refers to the ability of LLMs to solve MWPs with extended narratives. We introduce Extended Grade-Schoo Math (E-GSM), a collection of MWPs featuring lengthy narratives, and propose two novel metrics to evaluate the efficacy and resilience of L.MSs in tackling these problems. Our analysis of existing zero-shot prompting techniques with proprietary LLMs along with speen-souree LLMs reveals a general deficiency in CoLeG. To alleviate these issues, propose tailored approaches for different categories of LLMs. For proprietary LLMs, we introduce new instructional prompt designed to mitigate the impact of long contexts. For open-source LLMs, we develop novel auxiliary task for fine-tuning to enhance CoLeG. Our comprehensive results demonstrate the effectiveness of our proposed methods, showing improved performance on E-GSM. Additionally, we conduct an in-depth analysis to differentiate the effects of semantic understanding and reasoning efficacy. showing that our methods improves the latter. We also establish the generalizability of our methods across several other MWP benchmarks. Our findings highlight the limitations of current L.LMS and offer practical solutions correspondingly, paving the way for further exploration of model generalizability and training methodologies.
INTRODUCTION
Math word problems (MWPs) (Bobrow et al. 1964) are mathematical questions presented in natural language, demanding delicate reasoning for solving. With the flourish of large language models
(LLMs) (Brown et al. 2020: Touvron et al. 2023a OpenAT 2023, Jiang et al.] 2024a). the math reasoning ability measured on MWPs benchmarks (Cobbe et | Hendrycks et al. 202 Xu ct al. 2025c) has emerged models. The representative benchmarks for MWP, including GSM8K (Cobbe et al.; (021) are typically characterized by concise descriptions in a few sentences, whereas the performance of L.LMs in solving longer math problems remains under-explored. (Boaler) [993) suggests that extensive contexts might hinder rather than facilitate the mathematical reasoning process. This observation raises an important question: Do LLMs exhibit performance degradation in solving long MWPs? If so, how can we improve the solving accuracy of LLMs on these long MWPs, and what underlying factors contribute to the improvement?
critical evaluation metric to assess the overall capability of these
* Equal contribution. Corresponding author.

---

Published as a conference paper at ICLR 2025
To answer these questions, the GSM8K benchmark and questions are segregated based on the accuracy of CoT predictions. Rigid statistical hypothesis testing has revealed significant evidence suggesting that LLMs exhibit decreased performance on MWPs with long context (see Section 2. In order to further investigate how different L.MSs are affected by lengthy contexts of MWPs, we construct the Extended Grade-School Math (E-GSM), a benchmark comprising MWPs with extended context derived from GSM8K, and lwo novel evaluation metries. E-GSM maintains minimal alterations the conditions and the order of conditions while extending the contextual information of the original problem. We investigate four proprietary and four open-source LLMs, along with three saute-of-the-re zero-shot prompts on E-GSM. Our results indicate that the Context Length Generalization (CoLeG) of these LLMs, the ability for LLMs to do math reasoning in a long context, is limited, particularly with longer MWPs.
(Wei et al. 2022)
(CoT) is conducted the
To alleviate this issue, we propose two different strategies for proprietary and open-source L.MSs to improve the problem solving accuracy, respectively. For proprietary L.M,s, inspired by cognitive load theory (Sweller et al. 1998), we develop Condition- Rettrivvnn-eeiiiinngg Instruction (CoRe) prompting technique, which encourages LLMs to retrieve problem conditions first and then apply zero-shot- CoT (Kojima et al.] 2022). For open-source LLMs, suggest incorporating extension an auxiliary task during fine tine-tuning. We also examine the underlying factors contributing to accuracy improvements by analyzing informativeness and missing step metries derived from (Golovneva et al.) 2023). The experiment results suggest that the primary reason for this enhancement lies in improved reasoning pathways. Furthermore, We validate our strategies on several other MWP benchmarks such as MAWPS Cioncel-Kedziorg et al. 2016), SVAMP (Patel et al. (2021), and GSM-IC (Shi et al. 2023). demonstrating their effectiveness and generalizability.
We summarize our main contributions as follows:
We construct the E-GSM dataset, comprising MWPs with longer contexts. Comprehensive experiments on both proprictary and open-source LLMs reveal that math reasoning abilities of LLMs are significantly affected by long context.
2. We develop a new instructional prompt named CoRe for proprictary LLMs, which can improve CoLeG and problem solving accuracy of LLMs on E-GSM.
3. We propose to use extension as an auxiliary task to fine-tune open source LLMs and release our fine-tuning dataset comprising 65K CoT data.
4. CoRe and extension have demonstrated their strong generalization on several MWP benchmarks.
Our comprehensive evaluation reveals that long MWPs significantly degrade the mathematical reasoning abilities of LLMs, highlighting the limitations of current models. Furthermore, we offer practical solutions for both proprietary and open-source LLMs, and further experiments demonstrate the effectiveness and generalizability of our proposed methods. Our findings provides valuable insights and directions for future research on model generalizability and training methodologies.
THE E-GSM DATASET
2.1 LLMS STRUGGLE TO ANSWER MATH WORD PROBLEMS WITH LONGER CONTEXT
To explore whether the performance of LLMs in mathematical reasoning is adversely affected by longer textual contexts, similar to human performance, we conducted an experiment on GSM8K using CoT with GP GPT-3 3. 5-turbo as the representative. The experiment employed the 8-shot demonstrations provided by (Wei et al. 2022), and the complete prompt can be found in Appendix D) We can compare the answers generated by LLMs with the ground-truth answers and then divide the examples into two groups based on accuracy: the incorrect answers group G, and the correct answers group G1.
We want to study whether there exists a significant difference in the problem length, characterized by the number of tokens, between these two categories. We hypothesize that the lengths of the problem descriptions for these two groups are from different distributions, denoted X and Y. As illustrated in Figure[] the distributions between X and Y are quite different. To conduct rigorous analysis, apply the one-sided Mann- Whitney test (Mann& & Whitney| 1947):
Ho: P(X < <) P(X Y) + H P(X Y) P(X <Y).
2

---

Published as a conference paper at ICLR 2025
The results reported following (Fritz et al. 2012): There is significant evidence indicating that the number of tokens in G1 is less than in Go with = 141565, = 0.0000, which suggests LLMs perform better short MWPs than longer ones, similar human problem solvers.
To address the potential condounding factors, we fur- ther explore whether longer problem correlates with increased problem difficulty. We utilize the num- ber of steps required by GPT-3. 5-turbo solve the problems as a proxy for problem difficulty (An al. 2023) Wei al., 2022), denoted by S. dis- crete random variable with & levels. We assumed that
200
180
Number of Tokens
140
the number of problem tokens is denoted by T. To analyze this relationship, we consider the following linear model:
120
100
log 7 3o + 81S; B2S 2 +... Sk- + E,
60
40
where 3g is the coefficient corresponding the dif- ference in the dependent variable associated with the i-th level of 5 to the reference category. The logarith- mic transformation of 7' is to ensure homogeneity of variance, confirmed by Levene's test (Levene 1960)
False (Gg)
True(G;)
Figure 1: The visual comparison suggests the number of tokens in Go is larger than G, with Mann-Whitme the significance of these differences. This implies that LLMSs struggle to solve longer MWPs, which is similar to humans.
U test suggesting
for homogeneity of variance (P = 0.1518). Subse- quently, we proceeded with a contrast test (Neter et al. 1996):
Ho C. H1: C 3 O,
where is the contrast coefficients (Appendix B.6). and [S....., 8k The results indicate insufficient statistical evidence confirm the linear contrast (F = 1.9158, , P = 0.1476), suggesting insufficient evidence that a direct relationship exists between the problem length and its difficulty.
These findings eliminate the potential confounding factor that longer questions are more difficult and compellingly indicate that, similar to human solvers, LLMs may lost longer MWPs.
2.2 DATASET CREATION AND QUALITY CONTROL
To conduct comprehensive analysis of Co- LeG across range of LLMs, we have created E- GSM testing ground. Our approach lever- ages GP--4-uurbo to extend the original GSM8K benchmark. Specifically, we primarily construct the data that should meet the following three require- ments: The context of problems should be longer. 2) The ground-truth answers should remain the same. 3) The conditions and their order should not change. This section will detail the construction process.
Round-r Extension Prompt
[Instruction]:
Please exlend the following math question, you need to meet following guidelines:
1. The context
problem should be longer.
2. The ground-trute truth answer should remain the same.
3. The conditions and their order should not change.
[F [Few-shot D Demonstrations]:
Original question 1: di= Extended question 1: di
Initial trails revealed that generated questions were only slightly longer than their original questions, and GPT-4-turie failed to achieve a specified token length out in the instructions. To overcome this issue and facilitate the extension of math problems into more elaborate contexts, adopted sequential, iterative strategy. The process commences with the GSM8K (Cobbe al. 2022) test set During the r-th iteration (I < R, where R is the total num- ber of extension rounds), the i-th question from the preceding iteration (r 1), denoted as d." q.-1 E Qr-1, where Q denotes the set of extended variants after extension round 1, is extended using 2-shot demonstrations with GPT 4-turbo. Following the
Original question 2: dg Extended question 2: dg
[Q (Qusstion to Extend]: Original question: q."
Exlended question [...]
GSMOK Ce
Round-1 Q.
Round
Figure 2: E-GSM creation process and
prompt template for extension.
'The he accessed through this link
3

---

Published as a conference paper at ICLR 2025
expansion in the r-th round, quality control is performed to ensure quality, resulting in the refined set of extended problems, Qr. The prompt structure, along with the entire extension process, is depicted in Figure 2 As the expansion progresses, questions. The average tokens for MWPs each stage are presented in Table | In the later phases, a deceleration in the rate of increase in the number of tokens is observed, leading to end at R 4. As result, our E-GSM incorporates extended problems from all rounds 1 Qr. Examples are provided in Appendix A.I
observe gradual increase in the length of
After obtaining extended questions, we have to ensure they meet our require- ments mentioned above. We perform quality control process in the following two steps: 1) We apply human evaluation on a subset of E-GSM to precisely assess the quality of questions. Evaluation sults shows that 94.5% questions possess accepatable quality. Human evaluation details are provided in Appendix 2) We devise two heuristics to automatically filter undesired extended questions of the entire dataset. These heuristics are elaborated in Appendix A.3 After the quality control, the number of questions retained each round is shown in Table Detailed quality control process is given in Appendix A.4).
Table I: The average tokens of E-GSM dataset where the number of tokens returned by GPT-4-turbo and the number of questions in each round of extension.
Round
Qo
Q1
Q2
Qs
Q+
#Tokens
77.0
192.2
301.5
363.5
385.7
#Questions
1319
1195
1143
1102
1068
2.3 EVALUATION METRICS ON E-GSM
We propose two metrics to evaluate the performance of LLMs on E-GSM from two different angles efficacy and robustness. For efficacy, our goal is to check whether question and all its corresponding variants be consistently solved, thereby evaluating the model'ss 'ss capability to accurately solve the same question regardless of variations context length and circumvanting potential randomness. For robustness, the relative performance drop of the accuracy from Qo Qn is used, where Q, is the set of round-r extended questions. For a given question its ground-trutt is denoted by gt(4), and the answer generated by the method M represented M(q). The following metrics are considered:
Round-r accuracy (c) Acc(M; Q+) is defined as the average accuracy of method M on the set of problems Q,.
Acc(M: Q+) En 1[((g) = gt(4)
1Q,
where I is the indicator function and |Q,| denotes the number of round-r extended questions (from Table D
CoLeG-E quantifies the efficacy of Colee, which is defined as the averaged accuracy of M to solve
seed question and all its corresponding extension variants.
CoLeG-B(M) Eyseon W IM(()) = gtt))))))))))))1
Qul
where A denotes "all" operation and I is the indicator function. CoLeG-E evaluates the proportion of original problems that can be consistently solved across all levels of extended context.
CoLeG-R assesses the robustness facet of CoLeG, which is characterized by the relative accuracy drop rate QR compared to the performance initial questions Qu:
CoLeG-R(M) = -1 Acco(M) - (M)
Ac)
In addition, to further investigate the underlying factors behind L.MMs" abilities in solving long
MWPs, we leverage informativeness and missing step Golovneva et al. (2023) to evaluate the semantic understanding and mathematical reasoning facets of LLMs (will be discussed Section
A detailed calculation and explanation of these metrics can be found in Appendix B.2
"The full prompts can he found in Appendix
4

---

Published as a conference paper at ICLR 2025
Questionn On warm W Weddneday afteroon, Shamnon treats herself to refreshing iced coffee. Shannon makes herself iced coffee by brewing hot coffee and adding ice cubes, Each ice cube cools the coffee by degrees but makes the coffee 12 millilitars weaker. Any added liquid also weakens the coffee. How many millihiters weaker is Shammo'' losd coffee when is cooled by 65 degrees and she adds 15 milliliters of cream?
Step Calculate how many cubes Shanno beeds to cool the coffee by 65 degrees. To cool the coffee by 65 degrees, Sharmon needs to use 65 / 13 =5 ice cubes.
Let's think step by step.
LLMs
Each los cube makes the coffee 12 milliliters weaker, so 5 boo cubes will make the coffee 5* 12 = 60 mallilitees weaker.
0-CoT
Let's first undersland problem, then list all the known conditions which are formed by numbers or quantibative relationships alony with their contexts from problem lext, and identify the final gual the problem. Then let's solve the problem step by step.
1. Known Condltiions
... Shannon cools the coffee by 65 degrees. Shannon adds 15 milliliters of cream.
Goal:
Calculate the total weakening effect on Shamnon's s iced coffee when it is cooled by 65
LLMs
degrees and 15 milliliters cream are added.
3. Solution Strpe
..... Add the weskening effect of the cream: 60 milliliters + 15 milliliters -55 millilitees, Shamnon s iced coffee becomes 75 milliliters weaker when cooled by 65 are added.
CoRe
degreess and 15 miliiittrss
Figure 3: A comparison between solving a long problem (shortened version) with 0-CoT and CoRe.
3 METHODOLOGY
In this section, we detail our novel approaches aimed at enhancing CoLeG, Given that access to the model weights of proprictary LLMs is restricted, we have developed a new instructional prompt, to increase the abilities of LLMs in solving long MWPs (Section 3.T). For open-source LLMs, we introduce a novel auxiliary task, extension, to boost CoLeG by enriching the training data (Section 3.2
3.1
CONDITIO--ETTREVINNG INSTRUCTION FOR PROPRIETARY LLMS
Although proprietary LLMs exhibit strong math reasoning capability, they are still negatively im- pacted by long context in solving MWPs (Section 2.1). Our careful analysis of their generated solutions suggests that this issue stems from the L.MMs' difficulty in simultaneously processing all the information presented in lengthy problems while performing mathematical reasoning. As result, they may overlook critical details necessary for solving the problems, ultimately hindering their reasoning abilities.
Cognitive load theory (Sweller et al. 1998) suggests that humans are only conscious of the contents that exist in a limited-size working memory and all other information is hidden until it is swapped into working memory. Drawing this inspiration, we suppose that similar mechanism exists for LLMs solving MWPs: Long MWPs make the "LLMM'' working memory" saturated by irrelevant contextual details, and missing key conditions that are essential to solve the problem.
To alleviate this issue, we propose a novel
instruction (CoRe), which divides the problem solving process into two parts. Firstly, CoRe begins by guiding LLMs to identify the conditions and ultimate objective of the given problem, making sure "LLMs' working memory" is filled with essential conditions rather than irrelevant contextual details, and then apply O-shot CoT Kojima et al.] (202)) to prompt LLMs to solve the original problem. As shown in Figure 3 LLMs using 0-shot CoT fail to identify the essential information "she adds 15 milliliters of cream after the coffee is cooled" due to the influence of contextual details during reasoning, which leads to the wrong answer. In contrast, with the assistance of CoRe, L.MMs can first focus on parsing the conditions and the ultimate goal of the original problem. This allows them to concentrate on performing mathematical reasoning based solely on the previously identified essential information. By doing so, they can accurately deduce the numerical answer while minimizing the influence of irrelevant contextual details on their reasoning process. For more comparative examples, please refer to Appendix E Min et al.] (2023) proposes a similar approach, suggesting that breaking down information into smaller components can enhance the evaluation of factual precision.
3.2 EXTENSION AS AN AUXILIARY TASK FOR OPEN-SOURCE LLMS
We can generate longer MWPs using extension technique and apply quality control to filter out extended questions with poor quality introduced in Section 2.2 The new questions, their generated
5

---

Published as a conference paper at ICLR 2025
reasoning paths, their corresponding answers collected as augmented data:
D.- {(g.' , eiy, alj): : of = gt(g ; i ...,..,Ni.. 1,,,, KT};
where denotes the round of extension, i is the index of questions, Gi., Ei are j-th augmented reasoning path and j-th corresponding answer for the i-th question in r-th round of extension g. gt(-) is the ground-truth answer, N, is the number of questions left in r-th round after quality control, and KJ is the number of augmented CoT paths for of
We fine-tune LLM (parameterized by 0) D = Do D2 by maximizing the log likelihood of the reasoning path and corresponding answer conditioned on the question. The specific loss we used during supervised fine-tuning is given as follows:
[())= -GgeeaeD log O(e,
0)
RESULTS AND ANALYSIS
4.1 EXPERIMENTAL SETUP
Prompting Baselines.
For prompting methods, include 4 mainstream proprietary LLMs: Claude-3 3-opus, femini-pro (Team et al.] 2023). GPT 5-turbo, GPT-4o-min2 Ope
nAT 2024). To negate the sensitivity of the choice of few-shot demonstrations models' performance
(Fu et al.] 2023 Diao et al.) 2023). we explore zero-shot prompting techniques (Shi et al. 2023 Chen
al.
2024). including zero-shot CoT (0-CoT) (Kojima et al. 2022). Plan-and-Solve (PS) (Wan
et al.
2023a), and a variant of PS (PS+) (see Appendix D).
SFT Dataset. To generate our augmented dataset, We generate five reasoning paths for each question in the training set with GP T-3. 5-turbo and remove any paths that led to incorrect final answers. We get Do that incorporate 38,507 valid CoT data points (we also include the GSM8K training set in Do) and D that includes 26,422 CoT data for extended quesitons. The entire training set, represented as D Do U1, incorporates 64,929 CoT data. Appendix B.3 shows the input-output formats of the SFT data and examples from both Do and D, We also collect D2 of 24.147 CoT data to study the
effect of scaling up the SFT data (see Section C.2
SFT Baselines. For fine-tuning open-source LLMs, our study encompasses LLaMA-2 (Tuvron et al., 2023bb across different model scales and Mistral-7B (Jiang et al. 2023). More detailed descriptions of these LLMs can be found in Appendix B.I The baseline for SFT is fine-tuning without extension
(SFT on Do). Detailed SFT settings can be found in Appendix B.5
4.2 MAIN RESULTT
We compare the performance of various proprietary L.MSs using different zero-shot prompting techniques and open-source LLMS of varying sizes E-GSM. The main results summarized in T
Our observations are as follows:
There noticeable trend of diminishing accuracy from one round to the next in all LLMs, both proprietary and open-source. On average, Acco surpasses Acc, by 15.33% when using various 0-CoT prompts for different proprietary LLMs and by 17.41 % for open-source LLMs trained without extension across various model sizes and backbones. This trend indicates a decline in performance for LLMs when addressing longer questions, even when the difficulty level remains constant. Notably, claude-3-aee 3atee-3-psss achieves the highest accuracy in all rounds among the four proprietary LLMSs evaluated. However, it still exhibits a drop in accuracy of more than 10% from round 0 to round 4, despite using different zero-shot prompts.
CoLeG-E is significantly lower than Acc; and CoLeG-R deviates from for all LLMs. As dis- cussed in Section 2.3 CoLeG-E measures the percentage of problems accurately solved in all rounds, while Accs denotes the average performance of LLMs in the original GSM8K. The discrepancy between these two indicates that even though LLMs achieve high accuracy GSM8K, there inconsistencies when problems become longer. Specifically, a model may sometimes solve problem
'Do refers applying answer augmentation the original GSM8K training (see Section 4.1

---

Published as a conference paper at ICLR 2025
Table 2: Main Results (in %) of CoRe and extension.
Technique
Col.eG-F
Col.eG-R
Acco
Acci
Acda
Acca
Acc1
Claude 3 3-opus
PS 743T
84.07
95.45
92
88.28
85.30
N0.24
PS+
75.00
84,79
95.30
91.38
87.49
84,48
80.81
(-CoT
74.72
86.08
94.09
91.80
87.58
84,57
80,99
CoRe
77.81
86.29
95.38
92.64
88.98
85.39
82.30
Gemini-Pro
PS
46.25
77.35
80.74
75.82
69.99
65.15
62.45
PS+
80.52
76.82
66.88
63.76
0-CoT
49.16
75.69
83.70
77.91
71.83
66.76
63,36
CoRe
53.65
81.44
83.70
81.26
75.48
72.78
68.16
GPT-3.5-turbo
PS
41.76
80.43
78.70
76.65
70.87
67.15
63,30
PS+
48.03
81.14
80.67
78. 83
72.70
68.87
65.45
(-CoT
46.63
79.63
80.80
77.91
72.53
68.24
64.42
CoRe
51.97
83.64
83.40
81.26
75.33
73.23
69.76
GP GPT--o-mini
PS
83.73
93.71
81.49
78.46
PS+
73.60
85.09
93.86
91.72
87.05
82.85
79.87
0-CoT
71.91
81.58
77.90
CoRe
73.78
86.02
93.18
90.79
87.14
83.30
80.15
LLaMA-2-7B
Do
20.22
66.64
58.45
49.62
42.96
40.93
38.95
D
28.09
80.97
59.44
57.57
50.92
49.46
48.13
LLaMA-2-13B
Do
32:40
73.91
67.02
63:10
56.87
51.09
49.53
D
37.27
84.78
66.49
66.03
61.42
58.62
56.37
1..MAA2--00-
15.
81775
76727
7439
6971 1
66741
52.35
D
49.81
84.57
78.17
76.23
71.30
67.15
66.10
Mistral-7B
Do
42.13
75.19
75.39
71380
6435
61.25
5633
D
48.50
83.65
76.12
74.48
69.82
66.70
63.67
successfully but fail when the problem's 's context is extended. CoLeG-R quantifies the robustness of mathematical reasoning on macro scale. A CoLeG-R value deviating from signifies substantial decrease in performance from one round to the next. With the belief that model capable of "truly" solving problem should be able to so regardless of changes in context (Srivastava al. 2024]]
Qian et al. 2024 Ahn et al. 2024), there is still considerable room for improvement in developing model with higher CoLeG-E and CoLeG-R value close to
The accuracy in original GSM8K (Acc,) is insufficient for evaluating LLMs, while CoLeG-E and CoLeG-R offer additional assessment across other dimensions. Traditional leaderboards evaluate LLMs based solely accuracy with the original GSM8K, providing only limited view of model capability and often failing to effectively differentiate between LLMs. For instance, although Gem n achieves higher Acco than G 5utbbo under O-CoT (83.70% 80.89%) it consistently exhibits lower accuracy later rounds, leading to lower CoLeGGR (75.69% vs. 79.63%) compared GPT 5-turbo. This suggests that Gemini - is less robust when contexts become longer. On the other hand, DeminiiPree has higher CoLeG-E than GPT-3. 3-rbo under 0-CoT (69.16% v.s. 46.63%), implying that Gemini- Pro more effective at solving long MWPs but less robust (a lower CoLeG-R). Combining CoLeG-E with CoLeG-R, we can provide more in-depth analysis of the evaluation of LLMs on math reasoning than the original accuracy alone.
CoRe achieves the best performance on E-GSM across almost all proprietary LLMs compared to zero-shot prompting baselines. The improvements are particularly evident in higher rounds of E-GSM. Take GPT-3. 5-turbo for example, CoRe achieves improvement over 0-CoT by 2.51% in Acco and 5.34% in Acc,, Case study (Appendix has shown that our approach solve long MWPs involves initially extracting useful given conditions and the final goal while disregarding unimportant information. This is followed by sophisticated mathematical reasoning based on the information carefully extracted to solve the problem. Furthermore. CoRe has the highest CoLeG-E and CoLeG-R among all proprictary LLMs evaluated, indicating that our method not only enhances the efficacy of solving long MWPs, but also increases robustness when encountering long contexts.

---

Published as a conference paper at ICLR 2025
0.970
D-coT Odile
0-CoT Colla
edtention
500
02200
0.960
octendin
0..50
03.60
0.540
2.240
0.
0.940
6.70
0..00
0.900
0.020
0.800
6.650
9-CoT
0.88
0:0uT
0.600
extersion
0.000
Miving Stop
0888
Seteting step
0.
0.000
0.540
3.50
0.865
0.530
9.683
MM0
0.530
Gemini-Pro
(b) GPT-4o-mini
(c) LLaMA-2-7B
(d) Mistral-7B
Figure 4: Informativeness and missing step values of 4 representative LLMs
SFT with extension consistently and significantly enhances CoLeG on open-source LLMs, as reflected in metrics CoLeG-E and CoLeG-R as well as Ace. On average, SFT with extension (on D) leads to improvement of 5.9% in CoLeG-E and 9. 9.12% in CoLeG-R over SFT without extension (on Da). In particular, for LLaMA-2-7B, there is substantial absolute enhancement of 7.87% in CoLeG-E and 14.33% in CoLeG-R with our extension. This indicates that LLMs fine-tuned with extension demonstrate heightened effectiveness in solving MWPs with lengthy contexts and exhibit increased resilience against performance degradation such scenario, compared to naive SFT.
4.3 FINE-GRAINED ANALYSIS ON SEMANTIC UNDERSTANDING AND MATH REASONING
To deeply understand the underlying factors contributing to the improvements of our methods across various LLMs, we conduct more fine- pee-graiddd analysis on semantic understanding and math reasoning abilities of LLMs in solving long MWPs. We leverage informativeness and missing step metrics derived from (Golovneva et al., 2023) to capture these two facets, respectively (see also Section 2.3
The results are presented in Figure 4
From Figure we observe a performance decline both informativeness and missing step values across all LLMs from one round to the next, indicating that both their problem understanding and mathematical reasoning abilities are negatively impacted by longer contexts, with a more pronounced drop in understanding, collectively leading the accuracy degradation in solving longer MWPs.
Furthermore, both CoRe and extension enhance the math reasoning ability (shown by the missing step value) of proprietary and open-source LLMs, respectively. The improved math reasoning ability accounts for the improvement of problem solving accuracy in Table 2 which underscores the validity of our motivations and the effectiveness of our methods.
In contrast, we observe that the semantic understanding of both proprietary and open-source LLMs remains generally unchanged after applying CoRe extension, suggesting that neither prompting techniques nor supervised fine-tuning significantly impact the semantic understanding capabilities of LLMs. Based on this observation, we hypothesize that the language understanding ability of LLMs is predominantly established during the pre- pre-training stage and is minimally influenced by post- post-training adjustments or prompts, which differs from reasoning skills, which can be optimized by employing specific reasoning patterns in prompts through fine-tuning with additional data.
4.4 EXTENSION WITH SPECIALIZED MATHEMATICAL LLMS
In Figure 5 (Left, we have shown the comparison of SFT with extension on LLaMA-2 with MetaMath Yu al. 2023) across different model sizes. Note that "SFT" in the legend Figure to SFT with extension. Interestingly, the accuracy of SFT with extension on LLaMA-2 can be comparable to pre-SFT MetaMath, despite the fact that our D consists of only around 65K CoT data compared to 400K that MetaMath has been trained on. Furthermore, extension can also improve MetaMath in terms of accuracy in all rounds. These observations suggest that extending the questions included
8

---

Published as a conference paper at ICLR 2025
0.9
80
13B
Before-SFT
After-SFT
70
Accuracy (in %)
0.8
60
Accurscy
0.7
Lhuma-2 wis SET
1Jmma2 SFT
MetaMath SFT
MetnMat SFT
0.6
20
Acc 0
Avc_
2
Acc_
0.5
Short
Medium (66 ~ 82)
Long (83 ~ 203)
(40 ~ 65)
Figure 5: Left: Acc; varying over rounds in E-GSM of LLaMA-2 and MetaMath families. "w" and "w/o" stand for "with" and "without" respectively. Right: Accuracy on GSM8K with short, medium, and long length. The range of tokens within each category is in parenthesis.
in the training set could serve as an effective data augmentation strategy to improve the mathematical reasoning abilities of LLMs.
Extension as an auxiliary task can further improve MetaMath (Yu al. 2023) solving longer questions GSM8K. While MetaMath outperforms other math-related LLMs GSM8K, it still faces challenges with lengthy MWPs (Yu et al., 2023). To investigate whether extension would help, the GSM8K test set is divided into three subsets of equal size based question length and the accuracy is calculated over each subset. As shown in Figure 5 (right), extension not only benefits questions in all subsets but also yields the most significant performance improvement within the long-length group. Scaling up augmented data of extension by adding more rounds of extension to the training set (i.e., getting D, for larger r)) can be beneficial for LLMs to solve long MWPs as well (see Appendix C.2). In addition, We believe that extension can be applied orthorgonally to other data augmentation methods provided by (Yu al. 2023) to further mitigate the challenge of longer MWPs and we leave these for future work.
4.5 GENERALIZATION TO OTHER BENCHMARKS
To evaluate generalizability, we assess CoRe and fine-tuned LLMs (without further SFT on corre- sponding training set) other benchmarks including MAWPS (Koncel- Kedziorsk et al., 2016), SVAMP (Patel et al. (00T), and GSM-IC (Shi et al. 2023). Detailed experimental setups are given in Appendix B.7 The results are shown in Table
Both our CoRe and extension lead to superior performance across all evaluated benchmarks. These benchmarks consist of MWPs with concise descriptions, making the findings particularly notable: Despite being specifically tailored for long MWPs, these approaches are also effective for shorter MWPs. Remarkably, CoRe results in a 4.5% absolute increase accuracy on MAWPS with CoT and 4.15% on GSM-IC with PS+. Additionally, without further finetuning on the corresponding training set, our models (SFT on D) yield an average accuracy increase of 15.28% across different model sizes and benchmarks. This demonstrates the generalizability of our approaches in enhancing the performance of LLMs on MWPs.
RELATED WORK
Prompting for Mathematical Reasoning. Mathematical reasoning (Bobrow et al. 1964 Cobbe
et al. 2021: Hendrycks et al. 1021 is recognized system-2 task (Kahneman, 2011 Bengio et al. 2021), ) attracting significant attention in the research community. Recent LLM advancements (Brown al., 2020: OpenAl 2023 Ouyang al.) 2022 Jiang et al.] 2024a) have led to diverse prompting strategies designed to enhance their ability to perform mathematical reasoning (Ahn et al.
2024h. Notable among these is the Chain-of Thouugh (CoT) prompting (Wei et al. 2022). which has

---

Published as a conference paper at ICLR 2025
Table Solving accuracy (in %) of proprietary and open-source LLMs on other MWP benchmarks. The average number of tokens of MAWPS, SVAMP, and GSM-IC are 52, 54, 80, respectively.
GPr-3,5-turbo
LLaMA-2
Dataset
PS
PS+
0-CoT
CoRe
extension
7B
13B
70B
71.22
74.93
87.57
MAWPS
90.77
91.78
91.40
92.67
v
72.52
78.21
87.74
X 63.40
70.30
83. 10
SVAMP
71.90
75.70
71.80
76.30
V
64.90
74.00
83.80
x 62,45
74.40
84.77
GSM-IC
85,38
87.48
88.35
89.60
V
66.48
76.68
85.22
significantly improved reasoning capabilities by encouraging the model to generate intermediate steps. While various few-shot techniques such as Re-reading (Xu et al. (023) and Stepback prompting (Zheng et al. 2023) have been developed, there a growing interest in methodologies that enable zero-shot reasoning. These methods, such the two-stage CoT prompting by (Kojima et al. 2022) Plan-and-Solve (PS) prompting (Wang et al. 2023a), and the self-discovery framework by (Zhou et al. 2024). aim to equip LLMs to handle mathematical problems without few-shot demonstrations. Concurrently, Zhong et al. (2024) put forward DUP prompting to extract and answer core problems. Unlike their three-stage prompting, CoRe requires only one additional instruction before O-CoT.
CoT Extension. Extensions to CoT encompass demonstration selection (Zhang et al. 2023 Diao
et al. 2023 Fu et al. 2023), advancements in decoding (Wang et al., 2023bb Wang & Zhou 2024).
and developments for more intricate tasks (Yao et al., 2023b a Besta et al. [024]] Ding et al. 2023
Xue et al. 2024a b). Further explorations within this domain have probed areas such as incorrect
answer detection (Xu ct al. 2024), investigations of failure modes (Berglund et al. 2023 Shi et al.
2023 Chen et al.
2024]| Li et al. alla|[[[.00 (2024). other math-related topics (Gao et al. 2002a Song et al. 2024
Trinh et al. 2024:
Yan et al.) (2025), ), and application in broader domains (Jiang et al. (0044b] Xu et al.
2025a; Liu al.
2025] Gao et al. 2025 Phan et al. 2025). Our work pioneers the investigation of
I.I.Ms Col in mathematical reasoning, employing a zero-shot prompting to isolate the influence
of few-shot examplars.
Specialized LLMs for Mathematics. Despite general-purpose LLMs, there remains persistent interest in maximizing the performace of domain-specific LLMs like mathematics, through strategies such as SFT Yu et al. 2023: Yan et al.) 2024) and continued pretraining. Supervised fine-tuning methods, such as WizardMath's ' (Luo et al., 2023) combination of PPO training and MAmmoTH
(Yue al. 2023) knowledge distillation of integrating CoT with Program-of Thought(Cher et al.)
ofogram-of.Though(((C
2022; Gao et al. 2023b Xu et al. 2025b). Additionally, MetaMath (Yu et al., (023) rewrites mathematical questions in multiple formats to augment the training dataset. On the other hand, a distinct strand of research is dedicated to continuing pretraining of base LLMs on extensive mathematical corpora, exemplified by Minerva (Lewkowyez et al. 2022). Llemma (Azerbayev) et al., 2023), InternLM-MATH (Ying et al., 2024) and MeepSeekMath (Shao et al. 2024). Our work explores to what extent CoLeG of these specialized mathematical LLMs will be improved by fine-tuning with our newly proposed auxiliary task extension.
6 CONCLUSION
Our study explored LLMs' ability to solve longer MWPs, i.c CoLeG, We introduced groundbreak- ing dataset, E-GSM, designed to test CoLeG of LLMs, along with two metries to evaluate the efficacy and resilience of LLMs in this setting. Our investigation highlighted notable CoLeG deficiency in existing zero-shot prompting techniques and open-source LLMs. A new instructional prompt, CoRe, and novel auxiliary task, extension, not only significantly strengthen CoLeG but also show superior performance on GSMSK, and generalized well to other MWP benchmarks. By illuminating previously underexplored aspect of LLMs' reasoning and offering practical solutions to improve it, our work carved out new pathways for using LLMs in complex moblem-solving and set the stage for future research on model generalizability and advanced SFT paradigms.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was partially supported by grant from the Research Grants Council of the Hong Kong Special Administrative Region, China (Project Reference Number: AoE/E-601/24-NN) the National Natural Science Foundation of China (No.62477044) and the Fundamental Research Funds for the Central Universities (No. WK21501 10038). We would like to thank reviewers for their helpful suggestions in improving this paper.
REFERENCES
Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. Large language models for mathematical reasoning: Progresses and challenges. In Neele Falk, Sara Papi, and Mike Zhang (eds.), Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop, pp. 237, St. Julian's s, Malta, 2024. Association for Computational Linguistics. URL
eacl-srw.11
Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, Jian-Guang Lou, and Weizhu Chen. Learning from mistakes makes Ilm better reasoner. ArXiv preprint, abs/2310.20689, 2023. URL https
http://wwwwbbba.....aaaab/ / 20689
Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics. ArXiv preprint, abs/2310.10631, 2023. URL htttt:://axii..ot www.xiivoorg/bb...
1310.10633
Yoshua Bengio. Yann Lecun, and Geoffrey Hinton. Deep learning for ai. Communications of the ACM, 64(7):58-65, 2021.
Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The reversal curse: Lims trained on" is b" fail to learn" b is a". ArXiv preprint,
abs/2309.12288, 2023. URL htttttttss//xxii....to wbwp:////gggaaoaaa//a//oo/e/ abs /309 12288
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi. Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, and Torsten Hoefler. Graph of thoughts: Solving elaborate problems with large language models. In Michael J. Wooldridge. Jennifer G. Dy, and Sriraam Natarajan (eds.), Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pp. 17682-17690. AAAI Press, 2024. doi: 10. 10.1699/AAAI V38116.29720 URL 29720
www.://1.1..a..0...1//// v38116
Jo Boaler. The role contexts the mathematics classroom: Do they make mathematics more" real"? For the learning of mathematics, 13(2):12-17 1993,
Daniel Bobrow et al. Natural language input for a computer problem solving system. 1964.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voon Voss, Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien-Tienn Lin (eds.). Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL ht /tL/Atttppa///ppececiimmo 7457c0dde 6b cw 8b http://oocccciino..eeeeeereeeee. nww.ips.rppp... wttpsccppppppppppp/eee//ep
Abst html

---

Published as a conference paper at ICLR 2025
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompt- ing: Disentangling computation from reasoning for numerical reasoning tasks. ArXiv preprint, abs/2211.12588, 2022. https://a/xa http://rggagaaaaaa//oo/.oo.sseaa
12588
Xinyun Chen, Ryan A Chi, Xuezhi Wang, and Denny Zhou. Premise order matters in reasoning with large language models. ArXiv preprint, abs/2402.08839, 2024. URL Utl/attps://axiivoo
abs / 24002 08939
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichhro Nakano, et al. Training verifiers to solve math word problems. ArXiv preprint, abs/2110. 2021. URL https://arxiv.ot
Shizhe Diao, Pengeheng Wang, Yong Lin, and Tong Zhang. Active prompting with for large language models. ArXiv preprint, abs/2302.122446 2023. URL http::::: //arxiv o arg/abs/2307 12246
Ruomeng Ding, Chaoyun Zhang. Lu Wang, Yong Xu, Minghua Ma, Wei Zhang, Si Qin, Saravan Rajmohan, Qingwei Lin, and Dongmei Zhang. Everything of thoughts: Defying the law of penrose triangle for thought generation. ArXiv preprint, abs/2311.00 1.04244, 2023. URL https?
7/arxiv. wrg. 04254
Catherine O Fritz, Peter E Morris, and Jennifer Richler. Effect size estimates: current use, calculations, and interpretation. Journal of experimental psychology: General, 141(1):2, 2012.
Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. Complexity-based prompting for multi-step reasoning. In The Eleventh International Conference on Learning Representatiins, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL http:ttt/pppeeeiiiiiiii net bet/pdf?2 poooffiid-yy
19
Jiahui Gao, Renjie Pi, Jipeng Zhang, Jiacheng Ye, Wanjun Zhong, Yufei Wang, Lanqing Hong, Jianhua Han, Hang Xu, Zhenguo Li, al. G-llava: Solving geometric problem with multi-modal large language model. ArXiv preprint, abs/2312.11370, 2023a. URL htt.http///// /tLhttss:://artt /ttp::://rxxvvvoo.o
abs 22//2322.11 1170
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. PAL: program-aided language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 10764- 10799. PMLR, 2023b. URL press
ht httt::://proceedingg
html
Weibo Gao, Qi Liu, Linan Yue, Fangzhou Yao, Rui Lv, Zheng Zhang, Hao Wang, and Zhenya Huang. Agent4edu: Generating learner response data by generative agents for intelligent education systems. ArXiv preprint, abs/2501. 10332, 2025. URL http:::/axxxx..ccoooo http://aawww..oo/a////////////// 10332
Olga Golovneva, Moya Chen, Spencer Poff, Martin Corredor, Luke Zettlemoyer, Maryam Fazel- Zarandi, and Asli Celikyilmaz. ROSCOE: A suite of metries for scoring step-by-step reason- ing. In The Fleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. Utt.ttps::/ http::::///ppereviewwe net ndd/ ld Id=xY1JRp22ts?
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song. and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. ArXiv preprint, abs/2103.03874, 2021. URt/tttss://axxivvooo wwtpa//www.......aaaaaa/aaaaaaaa http://abs/11..... 0
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. ArXiv preprint, 1id/2330.06855, 2023. URL httlhhtps://// //ttps: org/abs.. /2310
06825
12

---

Published as a conference paper at ICLR 2025
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel. Guillaume Bour, Guillaume Lample. Lélio Renard Lavaud, Lucile Saulnier, Marie- Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts, 2024a.
Jin Jiang, Yuchen Yan. Yang Liu, Yonggang Jin, Shuai Peng, Mengdi Zhang, Xunliang Cai, Yixin Cao, Liangeai Gao, and Zhi Tang. Logiepro: Improving complex logical reasoning via program-guided learning. ArXiv preprint, (tbs/2409.12929) 2024b. URL /tRL/tttp/////xx.ooorg 12929
Daniel Kahneman. Thinking, fast and slow. macmillan, 2011.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwa- sawa. Large language models are zero-shot reasoners. In Sanmi Koyejo, S. Mo- hamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Process- ing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 December 9, nips htcc:paapr_flles/ppppppeeee -asteeelootttttttttttattttttoooo Cew.ectttooofeeeeeceeee html
2022, 2022. URL
/
Rik Kedzior-kddii Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. MAWPS: A math word problem repository. In Kevin Knight, Ani Nenkova, and Owen Rambow (eds.), Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1152-1157, San Diego, Cal- ifomiaa 2016. Association for Computational Linguistics. doi: 10. 10.18653/11/111-111 URL
https: https://aclanthen www.aaaaaao...orgg/........... org/N16-1136
Philippe Laban, Tobias Schnabel, Paul N. Bennett, and Marti A. Hearst. SummaC: Re-visiting NLI-based models for inconsistency detection in summarization. Transactions of the Association for Computational Linguistics, 10:163-177, 2022. doi:
URL https
//aclanthologe htwp:/hhhoooyy.rr/222222222 wwtp://gg//222.....o
Howard Levene. Robust tests for equality of variances. Contributions to probability and statistics, pp. 278-292, 1960.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay V. Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with language models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022. URL http:: [aa.aaeeeeaaeeetttoooooeeeeeeea.
nips oc wwwpr/aapppppeaeea files www.:eeeeeeppppeeeeee.ee / .ttm
Qintong Li, Leyang Cui, Xueliang Zhao, Lingpeng Kong, and Wei Bi. Gsm-plus: A comprehensive benchmark for evaluating the robustness of llms as mathematical problem solvers. ArXiv preprint, abs/2402.11255. 2024. URL httEps: org abs 2402. 19255
Jiayu Liu, Zhenya Huang, Tong Xiao, Jing Sha, Jinze Wu, Qi Liu, Shijin Wang, and Enhong Chen. Exploring socratic personalized teaching with large language models. Advances in Neural Information Processing Systems, 37:85693-88722 2025.
Socraticlm:
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. ArXiv preprint, abs/2308.09583, 2023.
arxiv.. org. abs /300 09583
Henry B Mann and Donald R Whitney. On a test of whether one of two random variables is stochastically larger than the other. The annals of mathematical statistics, pp. 50-60, 1947.
13

---

Published as a conference paper at ICLR 2025
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis. Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 12076-12100, Singapore, 2023. Association for Computational Linguisties. doi: 18.18v53/v1/ 2023.emnlp-main. 741. URL
emnlp main. 741
John Neter, Michael H Kutner, Christopher J Nachtsheim, William Wasserman, et al. Applied linear statistical models. 1996.
OpenAL. Gpt-4 technical report. ArXiv preprint, abs/2303.08777, 2023. URL https:///ax: org/abs/2303. 08774
OpenAI. Hello gpt-4o.
2024. 2024
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin. Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances Neural Information Processing Systems 35: Anmual Conference on Neural Information Processing Systems 2022. NeurlPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022. URL http///tt///eeeeerr.pp.p//ppp/pp b
01133391455805000 ni niss. http:///ppppeelll//s/pppeeeeeeee wttpp//ppppppp///22222 hatpapprr/22//haapeee
html
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NL.P models really able to solve sim- ple math word problems? In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani- Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies, pp. 2080-2094, Online, 2021. Association for Computational Linguistics. doi: 10.18653/v 120221 .naacl- main.1m8. URL
www.:.rgg//e 2021. naacl ma nn. 1...166 168
Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Sean Shi, Michael Choi, Anish Agrawal, Arna Chopra, et al. Humanity's 's last exam. ArXiv preprint, abs/2501. 14249, 2025. URL httphttt///rrattt.
org abs////////// 25001. /bw.////////.....a/////// 14249
Kun Qian, Shunji Wan, Claudia Tang, Youzhi Wang, Xuanming Zhang, Maximillian Chen, and Zhou Yu. Varbench: Robust language model benchmarking through dynamic variable perturbation. ArXiv preprint, abs/2406. 2024. httphhtts:/aaaa wwwp:/ivvorrgaaeomm wwwp/bs/////.oooo.o///.../t..aet abw.////.... /2006 211.214066778777111 17681
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Mingchuan Zhang, YK Li, Y Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. ArXiv preprint, abs/2402.03300, 2024. URL 03300
Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H. Chi, Nathanael Schärli, and Denny Zhou. Large language models can be casily distracted by irrelevant context. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning. ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learming Research, pp. 31210- 31227. PMLR, 2023. URL
.ml. wtt.://wprrrss/eemmmmmmmmm www.pprssss222. shi 23a html
Zezheng Song, Jiaxin Yuan, and Haizhao Yang. Fmint: Bridging human designed and data pretrained models for differential equation foundation model. ArXiv preprint, abs/2404. 14688, 2024. URL
kittps://azz /rg/ abs///b///// 2404 14688
Saurabh Srivastava, Anto PV, Shashank Menon, Ajay Sukumar, Alan Philipose, Stevin Prince, Sooraj Thomas, et al. Functional benchmarks for robust evaluation of reasoning performance, and the reasoning gap. ArXiv preprint, Abs/2402.119450 2024. URL https: aaxxiv.og/abb.. 2402. 19450)
14

---

Published as a conference paper at ICLR 2025
John Sweller, Jeroen JG Van Merrienboer, and Fred GWC Paas. Cognitive architecture and instruc- tional design. Educational psychology review, 10:251-290 199,, 1998.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean- can-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. ArXiv preprint, abs/2312. 1105, 2023. URL https:///rxxt
org/ ab///b///a 2312. 11805
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric- Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. ArXiv preprint, abs/2302.13971 2023a. URL https
//aaxiv.err//abeeb /20.a//3222222 13971
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. ArXiv preprint, 02//2307.09288 2023b. URL http::h/ta/aatt
arg/abs/ /2077. 09288
Trieu H Trinh, Yuhuai Wu, Quoc V Le, He He, and Thang Luong. Solving olympiad geometry without human demonstrations. Nature, 625(7995):476 482, 2024.
Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-We Lee, and Ee-Peng Lim. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2609-2634, Toronto, Canada, 2023a. Association for Computational Linguistics. doi: 10.18653/ v1/2023.acl-Iong. 147. URL (RRLtttts////cattaaaae
org 2tgg/023222000000 2023. acl Long.long... 147
Xuezhi Wang and Denny Zhou. Chain-of-thoughtt reasoning without prompting. ArXiv preprint, abs/2402.10200, 2024. [CLp:p/////axxxw.0/////////
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consisenccy improves chain thought reasoning in language models. In The Eleventh International Conference Learning Representationss ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023b. URL pdf 1atd-1pd-1PMMMP-
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances Neural Information Processing Systems 35: Anmual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022. URL http::ttt:///tte ni cc wwwpc//appr__feee http://ppprrrfllepspppppee/////p 9d560961 01w.0000005522211110110100000000
htg.::/ce.....aaaaaaa..aaaaaaaaa patlee/ppppr/20220 2222/hash//h
html]
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Marilyn Walker, Heng Ji, and Amanda Stent (eds.). Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Papers), pp. 1112-112,, New Orleans, Louisiana, 2018 Association for Computational Linguistics. doi: 10.18653/v1/118-111 1101. URL. htt.hhttts://tttt..tt / oot.hhlooo..ooo//11111-1111
Xiaohan Xu, Chongyang Tao, Tao Shen, Can Xu, Hongbo Xu, Guodong Long, and Jian-guang Lou. Re-reading improves reasoning in language models. ArXiv preprint, abs/2309.06275. 2023. URL
ftttps://azee www.xxivv....aaa..
wrg. abs 2309. 06275
Xin Xu, Shizhe Diao, Can Yang, and Yang Wang. Can we verify step by step for incorrect answer detection? ArXiv preprint, abs/2402.10528,. 2024. URL ptttps://arxia org abs ab//24////222222 2402 10528
Xin Xu, Qiyun Xu, Tong Xiao, Tianhao Chen, Yuchen Yan, Jiaxin Zhang, Shizhe Diao, Can Yang, and Yang Wang. Ugphysics: A comprehensive benchmark for undergraduate physics reasoning with large language models. ArXiv preprint, abs/2502.00334, 2025a. URL https
7/axxiv. wttp://wwrr//bb////...o.oo
15

---

Published as a conference paper at ICLR 2025
Xin Xu, Yan Xu, Tianhao Chen, Yuchen Yan, Chengwu Liu, Zaoyu Chen, Yufei Wang, Yichun Yin, Yasheng Wang, Lifeng Shang, et al. Teaching Ilms according to their aptitude: Adaptive reasoning for mathematical problem solving. ArXiv preprint, abs/2502.12022, 2025b. URL
attts:://arxx
12022
Xin Xu, Jiaxin Zhang. Tianhao Chen, Zitong Chao, Jishan Hu, and Can Yang. Ugmathbench: A diverse and dynamic benchmark for undergraduate--evel mathematical reasoning with large language models. ArXiv preprint, abs/2501.1376, 2025c.
org. /
2501. 13766
Shangzi Xue, Zhenya Huang, Xin Lin, Jiayu Liu, Longhu Qin, Tianhuang Su, Haifeng Liu, and Qi Liu. Enhancing the completeness of rationales for multi-step question answering. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pp. 2753-2763, 2024a.
Shangzi Xue, Zhenya Huang, Jiayu Liu, Xin Lin, Yuting Ning, Binbin Jin, Xin Li, and Qi Liu. Decompose, analyze and rethink: Solving intricate problems with human-like reasoning cycle. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024b. URL net/
htwpr//iiwieeeeeeeoueeeeeeeee Eorum?id NP 1WD j2
Yuchen Yan, Jin Jiang, Yang Liu, Yixin Cao, Xin Xu, Xunliang Cai, Jian Shao, et al. S c-math: Spontaneous step-level self-correction makes large language models better mathematical reasoners. ArXiv preprint, abs/2409.01 5244, 2024. URL httpt::////// //aa:i//////i///// wwepeiiiiorrrooooooooo abs 2409 01524
Yuchen Yan, Yongliang Shen, Yang Liu, Jin Jiang, Xin Xu, Mengdi Zhang. Jian Shao, and Yueting Zhuang. Mathfimer: Enhancing mathematical reasoning by expanding reasoning steps through fill tillin-the-mmidlle task. ArXiv preprint, abs/2502.11684 2025. URL
abs/ 11684
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In Alice Oh, Tris- Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Ad- vances in Neural Information Processing Systems 36: Annual Conference on Neural Infor- mation Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16. 2023, 2023a. URL http:://pppprsste nips. wwtpc//ppppppeeeeeeeeeeeeee files//le/// httpeeeepappee2233/// h2sh///////hh////// Abtt033-bbbsttttt Cew.rracctt-oooeeneeeeeeee html
Shunyu Yao, Jeffreyy Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Confer- ence on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023b. URL htll:htts://ppeeeeeet
net /pdf//dfff
Huaiyuan Ying, Shuo Zhang, Linyang Li, Zhejian Zhou, Yunfan Shao, Zhaoye Fei, Yichuan Ma, Jiawei Hong, Kuikun Liu, Ziyi Wang, al. Intermlm-math: Open math large language models toward verifiable reasoning. ArXiv preprint, abs/2402.06332. 2024. URL https:
org/abs/22 24tp.2402..0222 06332
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. ArXiv preprint, abs/2309. 2023. URL https
7/arxiv. org abs 2bt..///....220222
Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning. ArXiv preprint, abs/2309.05653, 2023. URL URLlhtts:///axxiv.coo
05653
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompt- ing in large language models. In The Eleventh International Conference Learning Rep- resentations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL
https://openrevieey wttp://wwwwweee//e.eeeeeem
JUHkr
16

---

Published as a conference paper at ICLR 2025
Huaixiu Steven Zheng, Swaroop Mishra, Xinyun Chen, Heng-Tze Cheng, Ed H Chi, Quoc V Le. and Denny Zhou. Take a step back: Evoking reasoning via abstraction in large language models. ArXiv preprint, abs/2310.061 17, 2023. / abs
Qihuang Zhong, Kang Wang, Ziyang Xu, Juhua Liu, Liang Ding, Bo Du, and Dacheng Tao. Achiev- ing> 97% on gsm8k: Deeply understanding the problems makes Ilms perfect reasoners. ArXiv preprint, abs/2404. 14963, 2024. /rL/ abs 2404. 14963
Pei Zhou, Jay Pujara, Xiang Ren, Xinyun Chen, Heng-Tze Cheng. Quoc Le, Ed Chi, Denny Zhou, Swaroop Mishra, and Huaixiu Steven Zheng. Self-discover: Large language models self-compose reasoning structures. ArXiv preprint, abs/2402.03620, 2024. URL https: // //ariiv.orgg/////
abs/ / 2402. 03620
17

---

Published as a conference paper at ICLR 2025
A E-GSM DETAILS
A.1 EXAMPLES IN E-GSM
In E-GSM dataset, we extend original set of problems from GSM8K to longer ones round- round-by-round by-round. GSM8K (Cobbe et al. 2021) is a benchmark of grade-school math word problems, containing training set 7473 examples, and a test set of 1319 problems. In table we show one particular
under MIT License and can
be accessible at original problem in GSM8K and its corresponding extension problems in the first two rounds. In total, our E-GSM consists of around 4.5K MWPs with extensive naratives, divided into four rounds (Table D}. E-GSM will be released under MIT License for future research.
Table 4: Examples from E-GSM.
Qo
A mother gaes shopping. She buys cocoa at $4.20, laundry at $9.45 and a package of pasta at $1.35. She pays S20. How much change does the cashier give back?
Q1
On a bright Saturday morning, a mother decided to take advantage of the weekend sales at her local supermarket. With a shopping list in hand, she navigated through the aisles, picking up items she needed for the week. Among her finds were rich, dark cocoa powder priced at $4.20, essential for her famous chocolate cake. Next, she grabhed bottle of laundry detergent, necessity for the upcoming week's laundry, priced at $9.45. Lastly, she couldn't 't resist adding package of pasta to her cart, a steal at just $1.35, perfect for Wednesday night's dinner. After browsing through the aisles and picking a few more items, she made her way to the cashier. Handing over crisp $20 bill to pay for her purchases, she waited for her change. How much change did the cashier hand back to her after her purchases?
Q2
On bright and sunny Saturday morning, the local supermarket was bustling with shoppers cager to take mother, who had meticulously prepared shopping
advantage of the weekend sales. Among them list to ensure she got everything needed for the upcoming week. As she entered the supermarket, her eyes scanned the aisles for the items on her list. Her first find was a rich, dark cocoa powder, priced at $4.20, essential for baking her famous chocolate cake that her family adored. Next, she picked up a bottle of laundry detergent, priced at $9.45, a necessity for tackling the week's 's laundry pile. Lastly, she spotted package of pasta, priced at just $1.35, perfect for the family's Wednesday night dinner. With her cart filled with these items and a few more essentials, she confidently made her way the cashier. After unloading watching the cashier scan each item, she handed over a crisp $20 bill to cover the cost of her purchases. As the cashier processed the transaction, she anticipated the change she would receive, knowing it would be just enough for small treat for her children on the way home. How much change did the cashier hand back her after her purchases?
cart and
A.2 HUMAN EVALUATION OF E-GSM
To ensure the quality of E-GSM, we carried out a human evaluation on 50 selected seed questions and their corresponding extension questions in each round. Three well-trained annotators majoring in mathematics, among whom two are graduate students and one is undergraduate, are asked to answer the following questions (Yes/No):
1. Does the order of the question descriptions (i.e., the conditions) remain the same after extension?
2. Are the conditions clear and unambiguous after extension?
3. Is the question solvable after extension (i.e., can an answer be derived from the conditions)?
4. Does the ground-truth answer remain the same after extension?
Then we can categorized these questions into three quality levels according to the answers of the previous questions: excellent, good, and poor.
Excellent. Affirmative responses to all questions.
Good. At least three affirmative responses, including a mandatory "Yes" to Question 3 and Question 4.
18

---

Published as a conference paper at ICLR 2025
Poor. Otherwise.
The inter-annotator agreement, as measured by Cohen's K, is given in Table The results indicate almost substantial agreement between the different annotators.
In Appendix A. we have shown an example at an excellent quality level. We show two examples of good and poor quality levels in table
Table 5: Cohen's 's scores of our human evaluation. 3-class refers to selecting among poor, good, and excellent; 2-class combines good and excellent and selects between poor and non-poor.
Annotator 2
Annotator 1,3
Annotator 2,
Average
2-class
0.7014 0.6307
0.6455 0.5543
0,8616 0.7675
0.7361
3-class
0.6508
Table 6: Examples of extended problems of different quality. The example in good quality changes the original condition "Each dog takes .5 hours a day walk and take care of their business" to "He takes each of his 10 dogs on half-hour walk." " The new condition does not include the time of "taking care of their business". The answer to questions is "No". The poor quality problem disregards the original condition that "he only uses his car to go work." The answer to question is "No". The changes in conditions are highlighted by red.
An Example Under Good Level
Original: John takes care of 10 dogs. Each dog takes .5 hours a day to walk and take care of their business. How many hours week does he spend taking care of dogs?
Round-4: In picturesque neighborhood, where the gentle whispers of nature hlend harmoniously with the serene ambiance, lives man named John. His love for dogs is not just hobby but a profound passion that has transformed his spacious backyard into a haven for lively dogs. Each dog, with its unique personality and zest for life, thrives under John's 's care, enjoying the lush, green expanse as their playground. John, whose heart is as expansive as the open spaces he provides, starts his day with the first light of dawn, ready to embark a journey of love and dedication with his canine family. John's 's daily routine unwavering commitment to the well-being of his dogs. He takes cach of his 10 dogs on a half-hour walk, exploring the scenic trails that meander through their neighborhood. These walks are more than just exereise; they are adventures filled with exploration, play, and moments of joy, allowing each dog to connect with the essence of nature. For John, these moments are sacred, an opportunity to deepen the bond with his dogs, ensuring their happiness and well-heing fulfilling life his dogs, how many hours does he devote each week walking and caring for his beloved companions, ensuring they experience the joy, health, and exercise they deserve?
testament to his
always the forefront. Given John's 's dedication providing
An Example Under Poor Levell
Original: Andy's 's car fuel efficiency 10 MPG (miles per gallon). the current price for regular gas S3/gallon, how much money is Andy's car consuming per week if he only uses his car to go to work from Monday to Friday and the one-way distance between his home and office is 5 miles?
Round-1: Andy, dedicated employee at bustling downtown firm, has daily routine that involves driving his reliable car work. His car, known for decent fuel efficiency of 10 miles per gallon (MPG), is his chosen mode of transportation for the 5-mile journey from his cozy suburban home to the high-rise office building where he works. As the week begins on Monday, Andy prepares for his usual commute, aware that the current price for regular gasoline stands at $3 per gallon. Throughout the week, from Monday to Friday, Andy uses his exclusively for his work commute, making the round trip each day with the intention of maximizing his time and efficiency, Given these circumstances, how much money Andy's 's car consuming per week for his work-related travels?
A.3 HEURISTIC FOR QUALITY CONTROL
To minimize human labor, we adopt a heuristic approach powered by an entailment model to automatically eliminate undesired examples after extension. Specifically, score S = (N gi) calculated to evaluate the informational equivalence between the extended and its corresponding seed questions.
For this purpose, we utilize the SAUMMAC zero-shot model (Laban et al. 2022), an entailment model trained on MNLI (Williams al. 2018). This model represents consistency detection technique
19

---

Published as a conference paper at ICLR 2025
in text summarization, employing off-the-shelf natural language inference model to determine the pairwise entailment score between texts. To detect all potential informational mismatches in any sentence, we aggregate sentence- levell entailment with the min operation:
ENT %) min max ent(ax: by),
where Gk; by are sentences of of dó respectively, ent represents the probability of entailment, the min taken over all sentences of %.. and maz is taken over all sentences of 9.. A small St suggest potentially unsatisfactory extension, leading the exclusion of the question qf. if f < 0.2, for anyj Σ r. Ultimately, human evaluation (see Appendix A.2 has shown that this approach effectively filters out all unsatisfactory questions.
A.4 DETAILED QUALITY CONTROL PROCESS
Fifty seed questions are randomly selected from GSM8K and all questions derived from these selected questions are manually inspected (200 in lotal).Three annotators assess the quality of the extended questions based on specific criteria and assign a quality level to each question (see Appendix (A.)). The final quality level is determined through majority voting. The judging criteria for each quality level and several examples are provided in Appendix A.2] Our evaluation finds that 185 of the 200 extended problems are of excellent quality, are good, and 11 are poor.
Due to resource constraints, it is impractical to conduct human evaluations across the entire dataset. Instead, we employ two heuristics to eliminate problematic instances. We guarantee that these methods will effectively detect all substandard examples within our selected questions and then apply them to the full dataset to ensure its quality.
First, we ensure that the extended variant faithfully retains information from the seed question through computing entailment score between the two. For the i-th seed question of and its associated extended variants 7/ g we derive score S = ENT q.) to evaluate the informational equivalence. More specifically, we adopt the out-of-box entailment model (Laban et al.) (022) to calculate S (see Appendix A.3)). A small ST can suggest potentially unsatistactory extension, leading to the exclusion of the question of if S < TI, for any j I r. Obviously, Qr S Qr-1 after filtration.
Second, consider extended variant unsolvable and filter it out if it cannot be addressed by proficient LLMS using various prompting methods. More concretely, GPT-4o-ming Claude opus, and Gemini-Pre with 4 different prompting methods are used to generate solutions for each question (see Section 4.2). and questions whose accuracy across 12 solutions is less than 72 are discarded.
From human evaluation of a subset, we can see that most of our extended questions are of excellent quality. We adjust the thresholds T1 and 72 based on human evaluation results, successfully filtering out examples of poor quality by setting = 0.2. The number of questions retained in each round is shown in T
B EXPERIMENTAL SETUP
B.1 LLMS
variety of proprictary LLMs used our study of zero-shot prompting methods are listed below:
GPT-3.5-turbo:
This represents refined version within the GPT-3.5 series, equipped with an enhanced understanding and generation of both natural language and coding content. We employed the opt-3.5-turb-0125 engine for experiments.
GPT-4o (OpenAl 2024): GPT-4o is multimodal, and has the same high intelligence as GPT-4 Turbo but much more efficient.
Developed by Google, this model is touted for its unmatched scalabil- ity across broad spectrum of tasks. Further information about the API is available at httpss//cloud.googg// wttglle..om//ettee.. wwwom//eeeeexa//dc///eeeeeee hww.///s//enneativeaa/imeaeee
Gemini-Pro:
Claude-3: This model is released by Anthropic recently, which is claimed to be comparable surpass GPT-4. We use C1 alaude- tolde-3-oppe -opus-20240222
20

---

Published as a conference paper at ICLR 2025
Access to all OpenAl APIs can be found at https://platfomm. htt.pppaattooooopoaaa..om www.emi...oooooos/ooooo//eveeeee Furthermore, the creation our new E-GSM dataset (Section22 2.2 utilizes GPT-4-turbo. To acquire
SFT data, GPT-3.5-turbo is leveraged generate CoT chains (Section 4.1p GPT-3.5-turbo is also
instrumental in extracting answers from the responses generated by various L.Ms (Section B.4).
To explore SFT, our study employs array of open-source LLMs and specialized mathematical LLMs.
LLaMA-2 families: We engage with models of varying scales 7B, 13B, and 70B specifically leveraging the instruction-tune versions. The instruction-tuned LLaMA- 2 accessible https://uuginngfccecommmm llaaaalaaaaaaaaaaaaaaaa----a-
They are under
Llama2 License)
Mistral-7B: The model has undergone instruction-tuning and is available at htt.:/ggggngaaac.co/msssaaaaaaaa
IB-II-tuctv--- It is under under/Apache AAache License 2.0
MetaMath (Yu al. 2023): This approach fine-tunes LLaMA-2 and Mistral-7B on MetaMathQA, a dataset augmented by rewriting original training questions from mul- tiple perspectives. We also incorporate MetaMath-LLlemma. All models are avail- able at http://ptttul.ooommmmmmmmmaaattt -lab-readme-ov-iile
The license for each model can be found at
https://gitub..com/mitta math/Metaaaall
B.2 INFORMATIVENESS AND MISSING STEP EXPLAINED
Both informativeness and missing step metrics are derived from the work of ROSCOE Goooovevaa ct al. 2023). Formally, given the problem descriptions as source context = {si, ..., ST } of T- sentences, LLM generated solutions hypothesis h {hi...., "i.....,,,,, hx } of N-steps and ground truth solutions as references r = {ri...., TK } of K -tstpps, informativeness and missing step are computed by measuring the semantics similarity among s, h and
We select the Informativeness- Chain as the informativeness metric. This metric embeds the reasoning chain and source context whole is calculated by measuring the cosine similarity: Info-Chain = + cos((s, /,, /) 2, where we employ all wwwmmttbaastteeeee v2 as the sentence embedding model.
For missing step, compute the similarity and those generated by the LLMs to determine whether any steps are missing in the latter. Given each alignment value a, r-align(r; + h) = [1 + max/_1 (cos(r;, hy))] [0, is the normalized cosine similarity between reference step and most similar step in the hypothesis, the missing step is defined as the minimal of similarity between each reference step and the hypothesis chain: Missing- Step = min,=1..K (r-align(r; - h.). Here we also employ all ali-mpnet-oo wompoet-bastee base-va as the sentence embedding model to embed all the reasoning steps.
between the reasoning steps from the ground truth
Furthermore, we conduct error analysis on 50 randomly chosen bad cases in the fourth round of E-GSM and find that 46% (23/50) samples failed due to the incorrect extraction of known conditions and the remains are due to flawed reasoning paths.
B.3 SFT DATA EXAMPLES
Our entire training set includes 64,929 CoT data, with 38,507 from the original CoT augmented training set Do and 26,422 COT data for extended questions Dj. Following YYu et al. 2023). the training prompt in Table 7 where the instruction is replaced by training example from Do or D1- For scaling up extended questions, there are 24,147 training examples in D: (see Section ?? and Appendix C.2 We showcase one concrete example from Dg. D1, and D2 in TablePJand one can find more examples our released SFT dataset D. D includes a total of approximately 89K CoT data, covering all examples within Do, Dj, D2. This dataset will be made available under MIT License Note that D. is only used Figure and Appendix C.2
B.4 ANSWER EXTRACTION
Unlike previous works Luo al. 2023 Yu et al.; 2023) that derive the final answer by extracting strings following the phrase "The answer is", we use GPT-3.5-tune for answer extraction as given in Table TO, This strategy is adept at handling atypical cases that cannot be readily resolved using
21

---

Published as a conference paper at ICLR 2025
Table 7: Training prompt for Open-source LLMs.
Table 8: Evaluation prompt for Open-source LLMs.
Below is an instruction that describes a task. Write response that appropriately completes the re- quest.
Below is an instruction that describes task. Write response that appropriately completes the re- quest.
##f Instruction:
### Instruction:
(instruction)
[instruction]
### Response:
### Response: Lct's think step by step
Table 9: Training Examples.
An example from Do
Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May? Response: Natalia sold 48 clips in April. In May, she sold half as many, which is 48/2 24 clips. Altogether, she sold 48+ + 24 72 clips in April and May.
Instruction:
An example from D1
Instruction: In the bustling town of Willow Creek, Natalia had become quite the entrepreneur with her homemade clip business. In the month of April, her creativity and hard work paid off when she managed to her beautifully crafted clips As the flowers began bloom and May rolled around, Natalia continued her efforts, though she found that the demand had slightly decreased. By the end of May, she realized she had sold exactly half the numher of clips she did in April. With these two months of sales combined, how many clips did Natalia sell altogether in April and May?
of her friends, each one more eager than the to support her venture.
Response: In April, Natalia sold clips. In May, she sold half the number she sold in April, which 48/2 = 24 clips. Therefore, Natalia sold 48 + 24 = 72 clips altogether in April and May.
An example from D2
the charming town Willow Creek, nestled among rolling hills and vibrant flower fields, Natalia had carved out a niche for herself as a budding entrepreneur with a unique talent for crafting homemade clips. Her designs, infused with crcativity and a personal touch, quickly caught the eye of her friends and neighbors. In the blossoming month of April, as the community came alive with the sights and sounds of spring, Natalia's 's business flourished. She found herself busily fulfilling orders for 48 of her friends, each one more enthusiastic than the last to support her burgeoning venture and adorn their outfits with her unique creations. As the calendar turned to May. the fresh blooms of April gave way to the full splendor of spring. Natalia, ever dedicated to her craft, continued to pour her heart and soul into each clip she made. However, she noticed slight shift in the air. The initial surge of excitement that had greeted her April launch had tempered somewhat, and the pace of orders had slowed. By the time the final days of May had drawn a close, Natalia took stock of her efforts and realized that the number of clips she had sold this month was exactly half of what she had managed in April. Reflecting on these two months of hard work and the support of her community, Natalia wondered: How many clips had she sold altogether in April and May, marking the beginning of her journey as entrepreneur in Willow Creek? Response: April, Natalia sold 48 clips to her friends.
Instruction:
May. she sold half what she sold in April, which means she sold 48 2 = 24 clips in May. To find out how many clips she sold altogether in April and May, we add the number of clips sold in April and May: 48 (April) + 24 (May)= = 72 clips. Therefore, Natalia sold 72 clips altogether in April and May, marking the beginning of her journey an entrepreneur in Willow Creek.
trigger words like "The answer is", which appears with various LLMs each presenting distinct output formats in zero-shot manner. Preliminary extraction has shown that over 99.8% of responses have been accurately extracted, Following (Yu et al.) 2023), for the SFT data mentioned in Section B.5] append "The answer is" to the answers extracted from GPT-3.5-turbe at the end of responses. Nevertheless, to ensure fairness, we also employ our innovative answer extraction technique for our fine-tuned L.MMs.
22

---

Published as a conference paper at ICLR 2025
Table 10: Prompt for answer extraction.
Given the last 'Quesion' and Answerr, your goal is extract the final numerical result from the 'Answer' part, and put the numerical result after 'Result' part. must not solve the problem by yourself, all you need to do is give me just number extracted from 'Answer'.
B.5 SFT DETAILS
In our experiments with the LLaMA-2 backbone, the learning rates chosen based on the model scale. For LLMs with parameters of 7B and 13B, the learning rate is set to 0.00002. For 70B LLMss, the learning rate is adjusted 0.00001. For the Mistral-7B base model, the learning rate is further reduced to 0.000005 to maintain the stability of the training. The learning rate is set to 0.00002 for LLaMA-3-8B and 0.00001 for LLaMA-3-70B.
Batch sizes are also tailored to LLMs parameters to maximize the utilization of computational resources. Specifically, for the 70B model, we select batch size of 24 per device. For models with 7B and 13B parameters, a larger batch size of 36 per device is chosen.
All models undergo training for epochs with AdamW optimizer with a 3% learning rate warmup. Experiments for LLMs with sizes 7B and 13B are conducted on 4 x H800 GPUs. For the larger 70B model, necessitating more computational power, the experiments are carried out on 8 x H800 GPUs (80G). The most GPU hours. The entire experiment takes around 1,600 GPU hours (excluding preliminary or failed experiments).
experiment (fine-tune 70B model) takes around 190
During inference, we apply greedy decoding with a temperature of 0.0 and the maximum generation length is set 512. Following (Yu et al. 2023), use the zero-shot evaluation prompting, as shown in Table where the instruction is replaced by the testing question.
All our SFT models will be made available for reproduction and future research endeavors. The license for these models will adhere to the same license applicable to the models prior to fine-tuning.
Multiple open-source LLMs, differing in scale and base models, fine-tuned D Do- The fine- tuning is carried out over 3 epochs and the results are reported using 0-CoT with the vLLM library All evaluations adhere to the same 0-CoT instruction set to ensure consistency. To account for the potential influence of random variations during the training process, we include the performance trends throughout the entire training period in Appendix C.1 Furthermore, comprehensive details regarding the training and inference parameters can be found in Appendix B.5
B.6 CONTRAST COEFFICIENTS
=== [C1, Cg...,, is the contrast coefficients, where each C2 is assigned based the hypothe- sized trend. The contrast C' is used to test the significance of the hypothesized trend across levels of 3. In case, the contrast coefficients that use centered linear trend coefficients. For example,
[-2. 1, 1,2] is used for [C..C2..... C5). Note that the absolute difference C² - C,-1 does not test C
matter because
and divide both sides by any number except O.
B.7 GENERALIZABILITY SETUP
Here we give a brief introduction to the benchmark datasets we used Scction 4.5
MAWPS Cronnee-Kedziorske et al. 2016) is a benchmark MWPs, incorporating 238 test examples. It is under MIT License and can be found at https://ithub..om/LLYY
YF/MWPToolki
SVAMP (Patel et al. 2021) includes 1,000 simple MWPs, which is available at is under MIT License
https://github.com/YYY YF/MWPToolkit htt.:////w/...YYYYY/P/T......o
23

---

Published as a conference paper at ICLR 2025
GSM-IC (Shi et al. 2023 is variant of GSM8K, including MWPs with irrelevant resegrcceeeeacchheee www.errcc-aatsettteetteeee datasets/GGMMIIC Following (Shi et al. 2023), randomly sampled 4K questions test set for evaluation.
sentence. The dataset is available at
C FURTHER RESULTS
C.I
TRAINING CURVES
the CoLeG-E and CoLeG-R for both LLaMA-2-7B and Mistral-7B throughout 1,200 fine-tuning steps, equivalent to three complete epochs. Initially, the performance of both models improves rapidly before stabilizing a later stage. This trend underscores the effectiveness of SFT in improving CoLeG.
40
CoLoG-E %)
CoLeG tin %)
30
20
10
65-
Mistral788
Misral-78
LLaMA-2-78
LLaMA-2-78
200
400
600
800
1000
1200
200
400
600
800
1000
1200
The performance curves of LLaMA-2-7 2-7B and Mistral-7B during 1,200 fine-tuning steps. Left: CoLeG-E; Right: CoLeG-R. The x-axis is the training steps during fine-tuning.
Figure 6:
C.2 SCALING UP SFT DATA
During the construction of the E-GSM (refer to Section 2.2
round-by-round extension strategy, which can also be applied to augment the original GSM8K training set. The primary findings utilize D Do D1 for SFT, and we can study the effect of enlarging SFT dataset through this specific extension strategy based on Dj. Consequently, we examine the effects of SFT on CoLeG in four distinct scenarios: without SFT, SFT on Do. SFT on Do, D1, SFT on Du. Dr, D2- Figure have shown the performance in different settings, and the additional outcomes are presented in Figure 7 and
C.3 MORE RESULTS
We include additional resutls in Table 12 The results have shown the efficacy our SFT method on specialized math LLMs.
D PROMPTS IN EXPERIMENTS
The 8-shot CoT experiment GSM8K (see Section 2.1 with GPT-3.5-turbo (see Section B.1 prompts provided by (Wei et al.) 2022). The full prompt is given Table [3]
For the construction of the E-GSM (see Section 2.2 provide the template for extension in Figure 2 Table shows the complete prompts for the first two rounds of extension, and all prompts will be released in our code repository.
24

---

Published as a conference paper at ICLR 2025
no SFT
80
SFT Pg.
SFT D,,,,,D,
70
CoLeG-E (%)
CoLeG-R (%)
30-
40
20
no SFT
10
10-
Ds
SFT on Py, Dy
LLaMA-2-78
LLaMA-2-138
LLaMA-2-708
LLaMA-2-78
LLaMA-2-138
LLaMA-2-708
Figure 7: CoL.eG-E (left) and CoLeG-R (right) of scaling up SFT dataset across various model scales in the LLaMA-2 family. The results suggest scaling up model scales and SFT dataset can further improve CoLeG.
50
20
Accuracy (8a
Accuraty
30
30
56
20
Figure 8: The changes in Acc; when LLMs fine-tuned different SFT dataset. Left: LLaMA-2- 7B; Middle: LLaMA-2-13B; right: LLaMA-2-70B.
Moreover, all zero-shot prompting including our proposed prompt methods along with their format are presented in Table 15 Our investigated zero-shot prompting including zero-shot CoT (Kojima)
al. 2022), Plan-and-Soe
(Wang al., 2023a) and its variant PS+
Zero-shot CoT (Kojima et al., 2022 a variant of CoT (Wei et al. 2022). prompts the model directly with the problem of interest followed by "Let's think step by step." without providing any demonstration examples.
Plan-end-Solve (Wang et al. 2023a) enhances 0-CoT by initially prompting LLMs create plan that breaks down the overarching task into subtasks, subsequently executing these subtasks in accordance with the devised plan. Additionally, we investigate a variant of PS, identified as PS+, designed to mitigate calculation inaccuracies.
E
CASE STUDY
We provide 2 solving cases of GPT-3.5-turbo with/without applying CoRe in Table 16 We could notice that faced with long context MWPs, (-CoT fails to extract all the necessary conditions from the problems (with the missed condition highlighted in red), thus finally lead to wrong answers. In contrast, our CoRe enables LLMs to first focus on extracting conditions from the problem and then perform math reasoning based on these conditions, avoiding distractions from lengthy story details.
25

---

Published as a conference paper at ICLR 2025
Table 11: Complete results of the effect scaling up SFT dataset on CoLeG, All figures are in %. Differences between the SFT results and the original are shown in parentheses. The best results within each model scale are shown in bold. In general, scaling up SFT dataset can improve CoLeG.
Model
CoLeG-E
CoLeG-R
Acco
Acc1
Acc2
Ace;
Acc₁
I.LaM-2-7B
4.31
66.71
27.37
23.93
22.83
18.97
18.26
20.22
66.64
58.45
49.62
42.96
40.93
38.95
+ SFT on Do
(+15.91)
(-0.07)
(431.08)
(+25.69)
(+20.13)
(+22.96)
(+20.69)
28.09
80.97
59.44
57.57
50.92
49.46
48.13
on Po, D)
(+23.78)
(+14,26)
(+32.07)
(+33.64)
(+28.09)
(+30.49)
(+29,87)
30.34
83.83
60.65
58.91
53.89
50.91
50,84
+ SFT on Do, D1, D2
(+26.03)
(+17.12)
(+33.28)
(+35.98)
(+31.06)
(+32.14)
(+32.18)
LLaMA-2-13B
8.61
70.18
37.76
36.32
29.83
27.95
26.50
32.40
73,91
67.02
63.10
56.87
51.09
49.53
SFT Do
(423.79)
(43.73)
(429.26)
(426.78)
(+27.04)
(+23.14)
(+23.03)
37.27
84.78
66.49
66.03
61.42
58.62
56.37
+ SFT on Do, D1
(+28.66)
(+14.60)
(+28.73)
(+29.71)
(+31.59)
(+30.67)
(+29.87)
38,86
80.05
69.60
70.63
62.99
59,44
55.71
SFT on Do. D1, D2
(+30.25)
(+9.87)
(+31.84)
(+34.31)
(+33.16)
(+31.49)
(+29.21)
I.AMA-2-70B
26.50
81.19
59.74
57.82
51.18
49.55
48,50
45.32
81.76
76.27
74.90
69.12
66.42
62,36
+ SFT on Do
(+18.82)
(+0,57)
(+16.53)
(+17.08)
(+17.94)
(+16.87)
(+13,86)
49.81
84.57
78.17
76.23
71.30
67.15
66.10
SFT on Do. D1
(+23.31)
(43.38)
(+18.43)
(+18.41)
(+20.12)
(+17.60)
(+17.60)
50.66
83.37
78.39
75.98
71.74
68.51
65.36
SFT on Do, D1, D2
(+24.16)
(+2.18)
(+18.65)
(+18.16)
(+20.56)
(+18.96)
(+16.86)
200
180
Number Tokens
140
120
80
60
40
False (Go)
True(G1)
Figure 9: GPT-40- -oommiinniiii.mi also struggles to solve longer MWPs.
F ADDITIONAL ANALYSIS
As one reviewer suggested, we use GPT-4o-mini to do the same analysis in Section 2.1 GPT-4o-mini achieves over 93% accuracy on GSM8K dataset, which strong L.M.. The results illustrated in Figure 9 which have shown very similar trend Figure I Following Section 2. there is significant evidence indicating that the number of tokens in G1 is less than in Go with U 60462.5, P 0.034.
We also add the same analysis for GPT-4o and OpenAl ol, as shown in Figure 10 and 11 Following Section 2.1 for GPT-4o there is significant evidence indicating that the number of tokens in G, is less than in Gg with 45185, = 0.0284; for OpenAl-ol, the problem remains, although is not that significant (U 47068.5, 0.1002).
26

---

Published as a conference paper at ICLR 2025
Model
CoLeG-E
CoLeG-R
Acco
Acc1
Acc₂
Acc3
Acc4
LLaMA-22 2-7B Models
MAmmoTTH-7B SFT on
11.52
66.09
47.46
45.02
34.91
32.67
31.37
30.15
80.25
62.77
60.00
54.42
52.18
50.37
(+18.63)
(+14,16)
(+15.31)
(+14.98)
(+19.51)
(+19.51)
(+19.00)
WizardMath-7B SFT on D
11.89
59.72
52.99
43.77
38.41
33.58
31.65
31.37
76.77
63.91
58.83
58.09
52.45
49.06
(+19.48)
(+17.05)
(+10.92)
(+15.06)
(+19,68)
(+18,87)
(+17.41)
MetaMath-7B
31.74
74.13
66.57
62.76
54.51
51.27
49.34
SFT on D
38.01
79.17
69.07
65.36
61.77
S7.62
54.68
(+6.27)
(+5.04)
(+2.50)
(+2.60)
(+7.26)
(+6.35)
(+5.34)
Mistral-7B Models
WizardMath-Mitral--7
50.94
79.13
82.71
76.90
70.78
68.78
65.45
SFT on
52.81
82.72
81.96
80.33
74.28
71.87
67.79
(+1.87)
(+3.59)
(-0.75)
(+3,43)
(+3,50)
(+3.09)
(+2.34)
MetaaMat-Mistral778
43.63
75.69
77.56
72.80
66.75
60.89
58.71
SFT on D
49.72
82.33
78.24
77.32
71.13
68.78
64.42
(+6.09)
(+6.64)
(+0.68)
(+4.52)
(+4.38)
(+7.89)
(+5.71)
Other Models
Ilemma_1 7b
2.06
66.63
28.81
27.64
23.10
20.05
19.19
SFT on
33.05
80.71
67.17
62.59
58.09
54.54
54.21
(430,99)
(+ (+14.08)
(+38.36)
(434.95)
(434.99)
(+34.49)
(+35.02)
meapseek-math-7b
42.98
69.07
81.88
76.57
66.67
62.79
56.55
SFT D
51.31
80.34
81.35
77.91
72.35
67.88
65.36
(+8.33)
(+11.27)
(-0.53)
(+1.34)
(45.68)
(+5.09)
(48.81)
DotaMath-Liemmar
28.18
64.77
68.23
62.34
53.81
49.55
44,19
+ SFT on D
38.86
77.32
70.96
67.78
62.73
$7.62
54.87
(+10.68)
(+12.55)
(+2.73)
(45.44)
(48.92)
(48.07)
(+10.68)
(LaMA-2-13BB Models
MAmmoTTH-138
18.63
74.07
55.12
52.97
45.76
45.01
40.82
SFT D
42.32
85.54
71.04
69.12
65.62
62.25
60.77
(+23.69)
(+11.47)
(+15.92)
(+16.15)
(+19.86)
(+17.24)
(+19.95)
WizardMath-133 138 + -FT on D
19.29
72.12
57.77
54.90
45.49
42,29
41.67
41.76
84.29
71.65
69.71
63.87
59.98
60.39
(+22.47)
(+12.17)
(+13.88)
(+14.81)
(+18.38)
(+17.69)
(+18.72)
MctaMath-133
36.14
75.68
71.27
68.20
60.45
55.99
53,93
+ SFT on D
44.10
79.38
73.84
71.38
65.97
63.07
58.61
(+7.96)
(+3.70)
(+2.57)
(+3.18)
(+5.52)
(+7.08)
(+4.68)
LLaMA-2-70B Models
WizardMath- 7OB
45.97
78.75
80.14
76.49
69.29
65.34
63.11
+ SFT on D
54.68
82.60
82.41
80.59
74.89
71.87
68.07
(+8.71)
(+3.85)
(+2.27)
(+4.10)
(45.60)
(+6.53)
(+4.96)
MAmmo 70B
42,98
85.24
72.93
74.48
69.20
64.52
62.17
SFT on D
55.15
83.25
81.88
80.67
74.37
71.05
68.16
(+12.17)
(-1.99)
(48.95)
(+6.19)
(+5.17)
(+6.53)
(+5.99)
MetaMath-70B
52.81
80.86
82,34
77.57
71.92
67.79
66.57
SFT on 2>
57.12
84.55
83.17
80,92
75.59
72.05
70.32
(+4.31)
(+3.69)
(+0.83)
(+3.35)
(+3.67)
(+4.26)
(+3.75)
Table 12: Additional results of the effect of SFT on CoLeG, All figures in %, and the SFT results are shaded. Differences between SFT results and original ones are shown in parentheses.
27

---

Published as a conference paper at ICLR 2025
200
180
Number Tokens
120
100
60
40
False (Go)
True(Gy)
Figure 10: GPT-4o also struggles to solve longer MWPs.
180
Number Tokens
160
120
100
80
40
False (Go)
True(G,)
Figure 11: OpenAI-o1 also struggles solve longer MWPs, although the problem is not that significant than other models.
28

---

Published as a conference paper at ICLR 2025
Table 13: 8-shot CoT Prompt.
Q: There are 15 trees in the grove. Grove workers will plant trees in the grove today. After they are done, there will he 21 trees. How many did the grove workers plant today?
A: There are trees originally. Then there were 21 trees after Grove workers planted some more. So there must have heen 21 15 6 trees that were planted. The answer is 6.
Q: If there are cars in the parking lot and more cars arrive, how many cars are in the parking lot?
A: There are originally 3 cars. Then more cars arrive. Now + cars are in the parking lot. The 5.
Q: Leah had 32 chocolates and her sister had 42. If they atc 35, how many pieces do they have left in total?
A: Originally, Leah had 32 chocolates and her sister had 42. So in total they had 32 42 74, After cating
35, they had 74 35 39 pieces left in total. The answer is 39.
Q: Jason had 20 lollipops. He gave Denny some lollipops. Now Jason has 12 lollipops. How many lollipops
did Jason give to Denny?
A: Jason had 20 lollipops originally. Then he had 12 after giving some to Denny. So he gave Denny 20 - 12
= 8 lollipops. The answer is 8.
Q: Shawn has five toys. For Christmas, he got two toys each from his mom and dad. How many toys does he have now?
A: Shawn started with toys. He then got 2 toys each from his mom and dad, So got Now he has 5 + 4 toys. The answer is 9.
more toys. Q: There were nine computers in the server room. Five more computers were installed each day, from now in the server room?
monday thursday. How many computers
A: There were originally 9 computers. For each day from monday thursday, more computers were installed. So 4 20 computers were added. Now 9 + 20 = 29 computers are now in the server room. The answer is 29.
Q: Michael had 58 golf balls. On tuesday, he lost 23 golf balls. On wednesday, he lost more. How many golf balls did he have at the end wednesday?
A: Michael started with 58 golf balls. He lost 23 on Tuesday, and lost more on wednesday. So he
23 = 35 at the end of Tuesday, and 35 - 2 33 at the end of wednesday. The answer is 33.
Q: Olivia has 23, She bought five bagels for 3 each. How much money does she have left?
A: Olivia had 23 dollars. She bought 5 bagels for dollars each. So she spent * 3 15 dollars. Now she
- 15 dollars left. The answer Q: [instruction]
A:
G BROADER IMPACTS
Our release of E-GSM presents a challenging benchmark test the ability of LLMS to solve long MWPs. This aids future research by highlighting the intricacies involved in extended MWPs. Our innovative prompting method and the SFT technique have improved LLMs performance these problems. potentially enhancing their generalization capabilities for other benchmarks. Note that both our E-GSM creation process and our methods can be easily adapted to other datasets or domains. This advancement not only broadens LLMs' mathematical reasoning capacities, but also has implications for educational tools. However, despite progress, significant challenges in solving extended MWPs remain, calling for future refinements. Additionally, there is risk of overreliance which might impede the development of critical thinking skill and problem-solving skills for human learners. Therefore, we point out the need for balanced use of LLM technologies, combining their benefits with critical human oversight.
29

---

Published as a conference paper at ICLR 2025
Table 14: Prompts to gpt-4-turbo-previie for extension.
Round-1
Please expand the following math question into a story-like question with long context. Your modification of the question cannot change the meaning and the answer of the original question, and you should reply the new question without any problem solving steps and logical reasoning. You can not change the order of the original description.
Original question: there are 3 cars in the parking lot and 2 more cars arrive, how many cars are in the parking lot?
At 8:00 AM on sunny Monday morning, the parking lot of the shopping mall was filling up with cars. Three sleek sedans were already parked in the designated spots. Among them are red and green. Their engines still warm from the short drive. hours later, the sound of revving engines and honking horns could be heard as two more cars zoomed into the lot, eager to find a spot. The drivers quickly found a spot and joined the other three cars in the lot. With the addition of these two new arrivals, the number of cars in the lot had increased. How many cars were now parked in the lot, ready to be driven home by their owners after long day of shopping?
New question:
Original question: There are 15 trees in the grove. Grove workers will plant trees in the grove today. After they are done, there will be 21 trees. How many trees did the grove workers plant today?
New question: At 10am, in the tranquil grove located in the outskirts of the bustling city. 5 workers arrived plant trees. As they began their task, they noticed that there were currently 15 trees in the grove. Among them are 5 apple trees and 10 peach trees. With the sun shining down on them, they worked tirclessly for hours, diligently planting new trees. By the time they finished, the grove was transformed with a total of 21 trees. How many trees were planted by the workers today? Original question: (question to extend] New question:
Round-2
Please expand the following math question into story-like question with longer context. Your modification of the question cannot change the meaning and the answer of the original question, and you should reply the new question without any problem solving steps and logical reasoning. You can not change the order of the original description.
Original question: At 8:00 AM sunny Monday morning, the parking lot of the shopping mall was filling up with cars. Three sleek sedans were already parked the designated spots. Among them are 1 red and green. Their engines still warm from the short drive. 3 hours later. the sound of revving engines and honking horns could be heard as two more cars zoomed into the lot, eager to find a spot. The drivers quickly found spot and joined the other three cars in the lot. With the addition of these two new arrivals, the number of cars in the lot had increased. How many were now parked in the lot, ready to be driven home by their owners after a long day shopping?
New question: At 11:00 AM, the shopping mall parking lot was bustling with activity. The sun was shining brightly, and the temperature was perfect for a day of shopping. As the clock struck 11 11:00, three sleck sedans were already parked in the designated spots, their engines still warm from the short drive. Among them were 1 red, 2 green. The owners of these cars had arrived early to beat the crowds and secure a goad parking spot. However, as the morning went on. more and more cars began to fill up the lot. At 1 11:30 AM, family in minivan pulled into the lot, searching for a spot to park. They were followed shortly by a group of friends in a convertible, eager to start their day of shopping. With the addition of these two new arrivals, the number of cars in the lot had increased. How many cars were now parked in the lot, ready to be driven home by their owners after long day of shopping?
Original question: At 10m, in the tranquil grove located in the outskirts of the bustling city, 5 workers arrived to plant trees. As they began their task, they noticed that there were currently 15 trees in the grove. Among them are 5 apple trees and 10 peach trees. With the sun shining down on them, they worked tirelessly for 4 hours, diligently planting new trees. By the time they finished, the grove was transformed with total of 21 trees. How many trees were planted by the workers today?
New question: On a bright and sunny morning at 10:00 AM, in a serene grove situated on the outskirts of a lively city, a group of 5 dedicated workers gathered with a mission to enhance the greenery of the area. The grove, already home to 15 trees, boasted mix of apple trees and 10 peach trees, standing tall and providing shade and beauty to the surroundings. The workers, motivated by the pleasant weather and the peaceful environment, set out to plant additional trees, aiming to contribute to the grove's diversity and vibrancy. As the sun continued its journey across the sky, the workers labored for 4 hours, their hands working the soil and their spirits lifted by the task at hand. By 2:00 PM. their efforts had visibly paid off, and the grove now featured total of 21 trees, each promising growth and life. How many trees were planted by the workers on this productive day?
Original question: (question to extend)
New question:
30

---

Published as a conference paper at ICLR 2025
Table 15: All Zero-shot prompting methods and formats.
Plan-and-Solve
Q: | Question
Let's first understand problem and devise plan solve the problem. Then, let's carry out the plan to solve the problem step by step:
Plan-and-Solve+
Q: [ Question
Let's understand problem, extract relevant variables and their corresponding numerals, devise
a plan. Then, let's carry out the plan, calculate intermediate variables (pay attention to correct numeral calculation and commonsense), solve the problem step by step, and show the answer:
Zero-shot CoT
Q: Question
A: Let's think step by step:
Condition-Retries Retrieving Instruction
Q: [ Question
A: Let's first understand the problem, then list all the known conditions which are formed by numbers or quantitative relationships along with their contexts from problem text, and identify the final goal of the problem. lett's think step by step:
31

---

Published as a conference paper at ICLR 2025
Table 16: Math problems solving cases of 5--T-b.5-tubb. before after applying CoRe.
Problem Solving Case from Q1
Henry set out early in the morning on his trusty bicycle, determined to complete a 60-mile journey through the picturesque countryside. The sun was just beginning to rise, casting golden glow over the fields and hills that lay ahead. After pedaling steadily for 20 miles, Henry decided to take his first break, stopping by a small stream to refill his water bottle and enjoy the peaceful sounds of nature. The journey was not yet halfway done, but he felt invigorated by the fresh air and the exercise. He knew that his second stop would be 15 miles before he reached his final destination. quaint little café known for its homemade pies and fresh coffee. With this thought in mind, Henry resumed his journey, cager discover how many miles lay between his first rest stop and the next. How many miles did Henry travel between his first and second stops during his scenic 60-mile bike trip?
Question:
0-CoTT Response: Sure, let's break it down step by step: 1. Henry traveled 20 miles before taking his first break. 2. His second stop would be 15 miles before reaching his final destination. 3. To find out how many miles Henry traveled between his first and second stops, we need subtract the distance of the second stop from the distance the first stop. So, to calculate the distance between Henry's first and second stops: Distance between first and second stops = Distance of first stop Distance of second stop. Distance between first and second stops = 20 miles- 15 miles. Distance between first and second stops 5 miles. Therefore, Henry traveled 5 miles between his first and second stops.
0-CoT CoRe Response: Sure, let's break down the problem and solve it step by step: Known conditions:
Henry set out on a 60-mile journey. He took his first break after pedaling 20 miles. His second stop would be 15 miles before reaching his final destination. The total distance of the journey is 60 miles. 2. Final goal: Determine how many miles Henry traveled between his first and second stops. 3. Solution: Henry took his first break after pedaling 20 miles, which means he has 60 20 40 miles left to travel. His second stop would be 15 miles before reaching his final destination, so he traveled 40 15 =52 miles from his first stop to his second stop. Therefore, Henry traveled 25 miles between his first and second stops during his scenic 60-mile bike trip.
Problem Solving Casc 2 from Q1
Question: a world where magic still whispers in wind and mythical creatures roam the hidden corners of the Earth, there exists a rare and beautiful species known as unicorns. It is said that only 27 of these majestic creatures are left, safeguarding the last remnants of magic. Among these, special group has made the misty Scottish Highlands their sanctuary. living amidst the rolling green hills and ancient forests. This secluded haven is home to one third the world's 's remaining unicorns, where they roam freely, their manes glistening in the sunlight and their hooves barely touching the soft, mossy ground. Within this group, a remarkable two thirds are female, embodying the grace and strength of their kind. How many female unicorns are there among the Scottish unicorns, preserving the magic and mystery of their species for future generations marvel at?
0-CoT Response: Sure. let's break it down step by step. We know that two thirds of unicorns in the Scottish Highlands are female. 2. Let's denote the total number of unicorns in the Scottish Highlands as "x". 3. Since two thirds of the unicorns are female, we can calculate the number of female unicorns as (2/3) x. 4. Therefore, the number of femalc unicorns among the Scottish unicorns is (2/3) 9 x. If you have specific number for the total unicorns the Scottish Highlands, we can use that to calculate the number of female unicorss.
0-CoT + CoRe Response: Problem Understanding: There are 27 unicoms in total in the world. -Oe third of the world's unicorns live in the misty Scottish Highlands. Among the unicorns in the Scottish Highlands, two thirds are female. Known Conditions:- Total number of unicorns in the world: 27. Number of unicorns the Scottish Highlands: 27 3 = 9. Percentage of female unicorns in the Scottish Highlands: 2/3. Final Goal: - Find the number of female unicorns among the Scottish unicorns. Solution: 1. Calculate the number of female unicorns in the Scottish Highlands: Number of female unicorns = Total number of unicorns in the Scottish Highlands Percentage of female unicorns. Number of female unicorns = 9 2/3. Number of female unicorns = 6. Therefore, there are female unicorns among the Scottish unicors the misty Scottish Highlands.
32


Reviews:
[{"review_id": "QJWBoNzZHV", "reviewer": "Reviewer_eQsN", "summary": "This work examines the effect of extended contexts on mathematical reasoning and introduces the Extended Grade-School Math (E-GSM) dataset, featuring math problems with lengthy narratives. Analysis reveals that current LLMs struggle with E-GSM, prompting the authors to propose new methods to address these challenges. \n\nFor proprietary LLMs, they introduce a new instructional prompt, while for open-source LLMs, they develop a novel auxiliary fine-tuning task. These approaches aim to enhance model performance in handling extended-context MWPs.", "questions": "- How is E-GSM different from GSM-IC[1]? \n\n[1] Large Language Models Can Be Easily Distracted by Irrelevant Context. ICML 2023. https://arxiv.org/abs/2302.00093", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper introduces E-GSM, a dataset with lengthy, distracting sentences that make it considerably more challenging than the original GSM. This dataset offers a valuable tool for evaluating the robustness of LLMs.\n\n- The approach used to create E-GSM can also be applied to expand existing math training datasets, providing new supervised fine-tuning (SFT) data in the math domain.", "weaknesses": "- The augmented math questions may include contradicting sentences. The augmented math questions  may become unsolvable or yield answers that differ from the original ones. \nAlthough human evaluations on 200 samples suggest that “94.5% of questions meet acceptable quality,” this accuracy may still be inadequate, particularly given that the labels in the GSM8K test set might contain errors.\nAn alternative could be to release these 200 samples as a verified subset of the E-GSM dataset. Reporting CoLeG-E and CoLeG-R results on the 200 samples, both with and without verification, would also be helpful.\n\n- In Table 2, the higher results w/ $\\mathcal{D}$ (compared to w/ $\\mathcal{D_0}$) may because the size of  $\\mathcal{D}$ is larger than $\\mathcal{D_0}$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TEWokgzSB8", "reviewer": "Reviewer_Dh39", "summary": "In this paper, the authors investigated the performance of LLMs in solving long math problems. They first examined the performance discrepancy of ChatGPT 3.5 in solving two versions (i.e. long form v.s. short form) of the same questions and concluded that LLMs struggle to answer math word problems with longer context. \n\nThen, they propose an automatic approach to extend the GSM questions into their long versions (named the obtained dataset E-GSM), with the same computation logic remaining, as far as they could. \nAfter that, the paper presented a method called CoRe to help proprietary LLMs better handle these long-form questions. \nFor the open source LLMs, the authors fine-tuned them with a fine-tuning dataset comprising 65K CoT data, created by the authors. \n\n\nThe paper introduced E-GSM containing artificial long math problems, but in real cases, there are seldom questions written in the way that the authors presented, i.e. very verbose questions talking about a relatively simple math problem. Therefore, it is unknown whether the conduct here can help in solving real-world long math problems where although the question is quite long, it already describes the problem in a succinct way that it could. Better solving them is our goal, rather than solving the artificial verbose problems that are less likely to exist in the real world. Although they show the same characteristic length-wise, the capability of solving the latter is not necessarily helpful for solving the former.", "questions": "1. According to “Evaluation results shows that 94.5% questions possess accepatable quality”, the total questions from rounds 1 to 4 should be about 5K. But in Table 1, it is only 4.5K.\n\n2. As shown in Table 1, different rounds have different numbers of questions, what’s the impact on the defined metrics? namely CoLeG-E and CoLeG-R?\n\n3. In Table 2, were the fine-tuned models evaluated with the CoRe method? can they be tested in the same way as those proprietary models? \n\n4. “Apart from 7,473 annotated examples available in GSM8K training set, we get D0 that incorporate 38,507 valid CoT data points …”, the numbers here confused me. If the authors generated five reasoning paths for each question in the training set, at most, D0 can have 7,473*5 questions, less than 38,507.\n\n5. In Section C.2, “The results suggest scaling up model scales and SFT dataset can further improve CoLeG.”, this conclusion may not be valid. Under CoLeG-R, after the SFT on D0, D1, and D2, the performance is not improved.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "The paper explored the impact of question length on LLMs’ performance and proposed a method to extend the length of GSM questions. The paper presented a method called CoRe to help proprietary LLMs better handle these long-form questions. For the open source LLMs, the authors fine-tuned them with a fine-tuning dataset comprising 65K CoT data, created by the authors.", "weaknesses": "1. The paper explores the artificial long math problems, but in real cases, there are seldom questions written in the way that the authors presented, i.e. very verbose questions talking about a relatively simple math problem. Therefore, it is unknown whether the conduct here can help in solving real-world long math problems where although the question is quite long, it already describes the problem in a succinct way that it could. Better solving them is our ultimate goal, rather than solving the artificial verbose problems that are less likely to exist in the real world.\n\n2. Besides the above major point, there are more points:\n- In Section 2.1, the authors examined the performance discrepancy of ChatGPT 3.5 in solving two versions (i.e. long form v.s. short form) of the same questions and concluded that LLMs struggle to answer math word problems with longer context. However, ChatGPT 3.5 is a relatively weak model now, I would suggest the authors do the same analysis with stronger open-source and proprietary LLMs. \n- Still in Section 2.1, the analysis here is based on real math questions, but the long questions in E-GSM are artificial. Therefore, it is not convincing to me that the conclusion in Section 2.1 can provide a solid foundation for the subsequent conduct. \n\n3. Many parts are not clear, see the questions section.\n\n4. The writing needs a thorough improvement:\n- “Human evaluation details are provided in Appendix A.4.” has a wrong reference. \n- In the first paragraph of Section 3, the subsections should be introduced in order. \n- The second sentence of Section 3.1 has redundancy. \n- The first two sentences of Section 3.2 are not about open-source LLMs, therefore, they cannot help develop this section. The third sentence is redundant. In the fourth sentence, “their generated reasoning paths” should be referred to the place that telling how it is done. The loss function has a typo, should be “ (q, e, a)”.\n- Section 3.3, “To negate the influence of few-shot demonstrations”, should be specific, what is the influence? \n- Repeated sentences in the third paragraph of Section 4.1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WVy2Gs2iN9", "reviewer": "Reviewer_jmci", "summary": "This paper investigates the ability of LLMs to solve math word problems (MWPs) with longer contexts, introducing the concept of Context Length Generalizability (CoLeG). The key contributions are:(1) Creating Extended Grade-School Math (E-GSM), a dataset of MWPs with extended narratives. (2) Proposing two metrics to evaluate LLMs' efficacy and resilience on E-GSM. (3) Developing tailored prompts for proprietary LLMs to improve CoLeG. (4) Using extension as an auxiliary fine-tuning task for open-source LLMs. (5) Analyzing the impact on semantic understanding vs reasoning efficacy.", "questions": "see weaknesses", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Strong motivation through rigorous statistical analysis shows LLMs struggle with longer MWPs (Section 2.1)\n\nProposes creative solutions (CoRe prompting and extension fine-tuning) to address identified limitations\n\nWell-designed metrics (CoLeG-E and CoLeG-R) that capture both efficacy and robustness of LLMs on long MWPs\n\nSufficient experiments have proven the effectiveness of the method", "weaknesses": "The paper focuses on LLMs tackling longer math word problems, rather than genuinely difficult ones. Addressing truly challenging problems would likely yield more impactful and valuable research insights.\n\nA deeper analysis of the types of errors LLMs make on extended MWPs would strengthen the paper. This could shed light on whether mistakes stem from misinterpreting context, losing track of key information, or actual computational errors.\n\nThe authors don't explore whether breaking down problems into atomic facts could help solve extended MWPs. It would be worthwhile to compare their methods against a baseline that first extracts crucial information from the lengthy context before attempting a solution. The techniques discussed in https://arxiv.org/abs/2305.14251 could be relevant here.\n\nThe table captions should be placed above the tables, not below, to comply with ICLR's official template guidelines.\n\nThe \"Experimental Setup\" section doesn't belong under Methodology. It should be moved to the Experiments section, alongside the results analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0sEQxr37gj", "reviewer": "Reviewer_cYhv", "summary": "This paper investigates the performance of LLMs on Math Word Problems with extended narratives, introducing the concept of Context Length Generalizability (CoLeG). The authors created a new dataset, Extended Grade-School Math (E-GSM), by iteratively extending problems from GSM8K. They propose two novel metrics, CoLeG-E and CoLeG-R, to evaluate efficacy and robustness respectively. The study reveals that existing LLMs struggle with longer MWPs, showing a consistent performance decline as context length increases. To address this, the authors introduce Condition-Retrieving Instruction (CoRe) for proprietary LLMs and an extension-based fine-tuning approach for open-source LLMs. These methods demonstrate improvements in CoLeG across various LLM types and generalize well to other MWP benchmarks as well.", "questions": "1. How does the performance degradation on longer MWPs correlate with specific architectural features of different LLMs, such as context window size or attention mechanisms?\n\n2. The extension approach shows promise for open-source LLMs. Have you considered how this might be adapted for extremely long MWPs or multi-step reasoning problems that span multiple pages?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper addresses a gap in current research by focusing on LLMs' ability to handle longer MWPs, which is more reflective of real-world mathematical reasoning tasks. The focus on CoLeG provides insights into the limitations of current LLMs and pathways for improvement.\n\n2. The creation of the E-GSM dataset through a systematic extension process is another contribution. By maintaining problem difficulty while increasing context length, the authors have developed a framework for evaluating LLM performance on longer MWPs.\n\n3. The introduction of CoLeG-E and CoLeG-R metrics offers a more comprehensive evaluation framework than traditional accuracy measures. These metrics provide insights into both the consistency and robustness of LLM performance across varying context lengths.\n\n4. The proposed methods, CoRe and extension-based fine-tuning, show consistent improvements across different LLM types and generalize well to other benchmarks.", "weaknesses": "1. The paper lacks a detailed exploration of why longer contexts impact LLM performance. While the authors mention potential working memory limitations, a deeper analysis could provide valuable insights. For instance, examining how performance correlates with the models' context window sizes or investigating the behavior of attention patterns in different layers could shed light on where breakdowns occur. Additionally, analyzing how different positional encoding schemes (e.g., rotary position embeddings vs. absolute position embeddings) affect performance on longer MWPs could offer insights into architectural considerations for improving CoLeG.\n\n2. The E-GSM dataset creation process, while systematic, may introduce biases that aren't adequately addressed. Using GPT-4 for extensions could potentially lead to biases in language style, problem structure, or even subtle cues that GPT-4 uses for reasoning. For example, GPT-4 might consistently use certain phrases or sentence structures that inadvertently serve as hints for other GPT models. Additionally, there's a risk of amplifying any biases present in the original GSM8K dataset. The authors should consider analyzing the distribution of problem types, linguistic patterns, and solution strategies in E-GSM compared to the original dataset to identify any systematic biases introduced during extension.\n\n3. The evaluation of open-source LLMs is limited to LLaMA-2 and Mistral-7B families. To provide a more comprehensive assessment, the authors should consider including models specifically designed for mathematical reasoning, such as MathGPT, GPT-f, or MetaMath. Additionally, evaluating performance on models with different architectural choices, like PaLM or BLOOM, could offer insights into how various model designs handle longer MWPs. This broader evaluation would strengthen the claims about the generalizability of the proposed methods.\n\n4. While the paper shows improvements on other MWP benchmarks, it doesn't explore how the proposed methods perform on problems significantly longer than those in E-GSM. This leaves questions about the scalability of the approaches to even more complex, multi-page word problems. The authors could consider creating a small set of extremely long MWPs (e.g., 1000+ tokens) to test the limits of their methods and provide insights into scaling challenges.\n\n5. The use of GPT-3.5-turbo for answer extraction in the evaluation process introduces a potential confounding factor. The paper doesn't adequately address how this might impact results, especially for non-OpenAI models. The authors should consider comparing this extraction method with simpler rule-based approaches or using model-specific output parsing to ensure fair comparison across different LLM families.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)