GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models
Authors: Seyed Iman Mirzadeh
Keivan Alizadeh
Hooman Shahrokhi
Oncel Tuzel
Samy Bengio
Mehrdad Farajtabar
Venue: ICLR 2025 Poster (2025)
Keywords: Reasoning
Large Language Models
Mathematical Reasoning
Datasets
Primary area: foundation or frontier models, including LLMs
TL;DR: We study the mathematical reasoning capabilities of large language models (LLMs) and introduce a new benchmark called GSM-Symbolic that enables more controlled and nuanced evaluations.

Abstract: Recent advancements in Large Language Models (LLMs) have sparked interest in their mathematical reasoning capabilities. While performance on the widely popular GSM8K benchmark has improved, questions remain about whether reported evaluation metrics are reliable, and reasoning abilities of LLMs have advanced. To overcome the limitations of existing evaluations, we introduce GSM-Symbolic, an improved benchmark created from symbolic templates that allow for the generation of a diverse set of questions. GSM-Symbolic enables more controllable evaluations, providing key insights and more reliable metrics for measuring the reasoning capabilities of models. 
Our findings reveal that LLMs exhibit noticeable variance when responding to different instantiations of the same question. Specifically, the performance of models declines when only the numerical values in the question are altered in the GSM-Symbolic benchmark. Furthermore, we investigate the fragility of mathematical reasoning in these models and demonstrate that their performance significantly deteriorates as the number of clauses in a question increases. We hypothesize that this decline is due to the fact that current LLMs are not capable of genuine logical reasoning; instead, they attempt to replicate the reasoning steps observed in their training data. When we add a single clause that appears relevant to the question, we observe significant performance drops (up to 65%) across all state-of-the-art models, even though the added clause does not contribute to the reasoning chain needed to reach the final answer. Overall, our work provides a more nuanced understanding of LLMs' capabilities and limitations in mathematical reasoning.

Published as a conference paper at ICLR 2025
GSM-SYMBOLIC: UNDERSTANDING THE LIMITA- TIONS OF MATHEMATICAL REASONING IN LARGE LANGUAGE MODELS
Iman Mirzadeh
Keivan Alizadeh*
Hooman Shahrokhi Mehrdad Farajtabar'!
Oncel Tuzel
Samy Bengio'
Apple Washington State University
ABSTRACT
Recent advancements in Large Language Models (LLMs) have sparked interest in their mathematical reasoning capabilities. While performance on the widely pop- ular GSM8K benchmark has improved, questions remain about whether reported evaluation metrics are reliable, and reasoning abilities of LLMs have advanced. To overcome the limitations of existing evaluations, we introduce GSM-Symbolic, an improved benchmark created from symbolic templates that allow for the gen- eration of diverse set of questions. evaluations, providing key insights and more reliable metrics for measuring the reasoning capabilities of models. Our findings reveal that LLMs exhibit notice- able variance when responding to different instantiations of the same question. Specifically, the performance of models declines when only the numerical values in the question are altered in the GSM-Symbolic benchmark. Furthermore, we investigate the fragility of mathematical reasoning in these models and demon- strate that their performance significantly deteriorates as the number of clauses in a question increases. We hypothesize that this decline is due to the fact that cur- rent LLMs are not capable of genuine logical reasoning; instead, they attempt replicate the reasoning steps observed in their training data. When we add a single clause that appears relevant the question, observe significant performance drops (up to 65%) across all state-of-the-ar models, even though the added clause does not contribute the reasoning chain needed reach the final answer. Over- all, our work provides more nuanced understanding of LLMss capabilities and limitations in mathematical reasoning
SSM-blybolic enables more controllable
INTRODUCTION
Large Language Models (LLMs) have demonstrated remarkable capabilities across various domains. including natural language processing, question answering, and creative tasks (Gunter et al. 2024
OpenAl 2023 Dubey et al. 2024 Anil et al. 2023 Abdin et al.) 2024 Rivière et al. 2024). Their potential to perform complex reasoning tasks, particularly in coding and mathematics, has garnered significant attention from researchers and practitioners.
However, the question of whether current LLMs are genuinely capable of true logical reasoning remains an important research focus. While some studies highlight impressive capabilities, closer examination reveals substantial limitations. Literature suggests that the reasoning process in LLMs probabilistic pattern-matchin rather than formal reasoning (Jiang et al.) 2024). Although LLMs can match more abstract reasoning patterns, they fall short of true logical reasoming. Small changes in input tokens can drastically alter model outputs, indicating strong token bias and suggesting that these models are highly sensitive and fragile (Jiang et al.) 2024: Shi et al. (202)). Additionally, in tasks requiring the correct selection of multiple tokens, the probability of arriving at an accurate answer decreases exponentially with the number of tokens or steps involved, underscoring their inherent unreliability complex reasoning scenarios (Schaeffeer et al. 2023).
WWrk done during an internship at Apple. CCorespondencc to ftc (arprrdech..aaaaaaarraaaaaa (GSM-Symbolic templates and generated data are available http:///ggtaiicccmammmmm///mmmmm

---

Published as a conference paper at ICLR 2025
GSM8K
GSM Symbolic Template
Sophic vatches nopheu, abe gets out variety of toya for hin. The bag of building The
when (name) vatches her (family). she gets out varisty of toys
bag building has GO) blocks in it. The bin of stuffed aninala bas (y) etuffed aninale inside. The tower of stacking rings has [2] multicolored ringa it.[nan)) recently bought a tube of bouncy balls, bringing her total nunber of toys bought her (family) up (total), many bouncy balls came in the tube?
blocka bin of stuffed aninals has tover of stacking ringe has multicolored rings tt.Soohii recently bought tube of bouncy halls, bringing her total number toya for nephee up How nany bouncy balls cane in the
blocka
#variables:
sample (names)
femily semple ("pephev", "cousin", "brother"])
rango (5, 100)
range 100)
range (5, 100)
range 500)
and
range (86, 200)
#conditions:
aBS
total
the balls in the tube.
bouncy
Let T be the number of bouncy balls in the tabe. After buying tube
buying Sophia has 31+8-9+ T 48 =62 toys for her nephen.
balls,
(asue) + (total) toys for her (family).
T
Thue, -82-48 < 62-48-14>>11 bouncy balls came in the tube.
Thus, T (total) + z (ans) {ans] bouncy balls came in the tubo.
(tstal))
Figure I: Illustration of the GSM STM--Smbolls lic template creation process. This dataset serves tool investigate the presumed reasoning capabilities of LLMs, enabling the design of controllable mathematical reasoning evaluations with more reliable metrics, Our results reveal that all state-of- the-art LLMs exhibit significant performance variations, suggesting the fragility or lack of reasoning.
Mathematical reasoning is a crucial cognitive skill that supports problem-solving in numerous scien- tific and practical applications. Consequently, the ability of large language models (LLMs) to effec- tively perform mathematical reasoning tasks is key to advancing artificial intelligence and its real- world applications. The GSMSK (Grade School Math 8K) dataset Cobbe et (202T) has emerged as popular benchmark for evaluating the mathematical reasoning capabilities of L.LMs. While it includes simple math questions with detailed solutions, making it suitable for techniques like Chain- of-Thought (CoT) prompting, provides only single metric on a fixed set of questions. This limi- tation restricts comprehensive insights into the models' mathematical reasoning. Moreover, the pop- ularity and prevalence of GSM8K
increase the risk of inadvertent data contamination. Finally, the static nature of GSM8K does not allow for controllable experiments to understand model limita- tions, such behavior under varied conditions or changes in question aspects and difficulty levels.
To address these limitations, a more versatile and adaptive evaluation framework is needed one that generate diverse question variants and adjust complexity levels to better explore the robustness and reasoning abilities of LLMs. This would facilitate a deeper understanding of the strengths and weaknesses of these models in mathematical reasoning tasks. We make the following contributions:
We introduce SSM-Symbolic, an enhanced benchmark that generates diverse variants of GSM8K questions using symbolic templates (Sec. 33. as shown in Fig. []] This enables a more nuanced and reliable evaluation of LLMs' performance across various setups, moving beyond single-point accuracy metrics. Our large-scale study on 25 state-of sfte-oo--the-ar open and closed mod- els provides significant insights into LLMs' behavior mathemattial reasoning tasks.
We question the reliability of currently reported results on GSMSK and demonstrate that the performance of LLMS can be viewed as a distribution with unwarranted variance across differ- ent instantiations of the same question. We show that the performance of all models drops on GSM-Symbolic (Sec. 4.1), hinting at potential data contamination.
We show that LLMs exhibit more robustness to changes in superficial elements like proper names but are very sensitive to changes in numerical values (Sec. 4.2h. We show that performance degra- dation and variance increase as the number of clauses increases, indicating that LLMs' reasoning capabilities struggle with increased complexity (Sec. 4.3
2

---

Published as a conference paper at ICLR 2025
Finally, we further question the reasoning abilities of LLMs and introduce the GSM-NoOp dataset. By adding seemingly relevant but ultimately irrelevant information to problems, we demonstrate substantial performance drops (up to 65%) across all state-of-the-art models (Sec. 4.4). This reveals a critical flaw in the models' ability to discern relevant information for problem-solving, likely because their reasoning is not formal not formal in the conventional sense and is mostly based on pattern matching. We show that even when provided with multiple examples of the same question or examples containing similar irrelevant information, LLMs struggle overcome the challenges posed by GSM-NoOp. This suggests deeper issues in their reasoning processes that cannot be alleviated by in-context shots and needs further investigation.
Overall, our work provides a comprehensive understanding of the limitations of LLMs in mathe- matical reasoning. Our results emphasize the need for more reliable evaluation methodologies and further research into the reasoning capabilities of large language models.
2 BACKGROUND: REASONING & LANGUAGE MODELS
Logical reasoning is a critical trait of intelligent systems, and building intelligent systems capable of such reasoning has long been an important goal in artificial intelligence (Newell & Simon [956
Marcus, [2033; Brachman & Levesque| 2004; Legg & Hutter [007] Pearl [214]] Chollet 2019).
Recent advancements in Large Language Models (LLMM)) have demonstrated significant potential across various domains; however, their reasoning abilities remain uncertain and inconsistent. Many
works have investigated whether LLMs are truly capable of reasoning by examining how these models solve tasks that require logical reasoning.
We define logical reasoning as the process by which an agent (a human or machine) employs logical steps to achieve "novel" goal. The emphasis on "novelty" is crucial because it helps distinguish genuine reasoning from mere memorization of solutions or responses, or mimicking the logical steps previously encountered (Gignac & Szodorai 2024). This distinction aligns with ideas akin to the Chinese Room Argument (Cole, 2024), underscoring the difference between true comprehension and pattern-matching. Furthermore, this definition connects to several established definitions in AI literature. For instance, the "length-generalizatton"" refers to the ability to apply known logical steps on larger inputs. Moreover, in order successfully apply such logical steps, an agent often requires other skills such "decomposing" a problem into smaller sub- cuub-roollmss, and "composing" logical steps from these sub-problems in order to solve another problem.
One interesting direction focuses on modeling the computation performed by transformers. For ex- ample, parallels have been drawn between components such as attention and feed-forward modules and simple computational primitives (Weiss et al. 2021] Zhou et al.] 2024). [Deéttang et al. (2023) demonstrated that transformers fail to generalize on non-regular tasks and showed that structured memory (e.g.. memory tape) is necessary for handling complex tasks. This is related to the effec- tiveness of Mhaai--of-Toughh (CoT) prompting eWei al. 2022) and using scratchpads for LLMs as additional memory for intermediate computations. Moreover, Zubic et al. (0255) show theoretically and empirically that Structured State Space Models (SSMs) and Transformers face fundamental lim- itations in performing function composition and complex reasoning tasks. Overall, current results suggest that while the transformer architecture has limitations and lacks the required expressiveness for solving problems across several complexity classes, these limitations can be alleviated with ad- ditional memory (e.g,, scratchpads) (Liu et al.] (2024). However, this still requires generating vast amounts of tokens solve problem (Peng al. 22024; OpenAl 2024). While these works pro- vide insights into the theoretical computational complexity of transformers, in practice, it remains unclear whether these LLMs can perform formal logical reasoning to solve tasks.
There is considerable body of work suggesting that the reasoning process in LLMs is fragile and not formal (Kambhampati 2024 Valmeekam et al.) 2022, 2024, Nezhurina et al.) 2024] McCoy et al.) 2023) IZhang et al.) 2023). even though it appears that these models understand symbols and can work with them to some limited degree (Boix-Adserà et al. 2024). Instead, LLMs likely perform a form of probabilistic pattern-matching and searching to find closest seen data during training without proper understanding of concepts. While this process goes beyond naive memorization of words and the models are capable of searching and matching more abstract reasoning steps, it still falls short of true formal reasoning. For instance, Jiang et al. (2024) show, with statistical guarantees, that most LLMs still struggle with logical reasoning due to strong token
3

---

Published as a conference paper at ICLR 2025
bias, where the reasoning output of the model changes when single token of input changes. This aligns with our results, which indicate that the performance of models on different instances of the same mathematical question can vary greatly from one instance to another. [i et al. (2024b) prove that a single transformer layer learns a one-nearest neighbor, which could explain why the reasoning of models is highly sensitive input tokens. [Schaeffer et al. (2023) argue that when task requires emitting multiple tokens correctly, the probability of answering correctly decreases exponentially with the number of tokens. Dziri et al. (2023) represent reasoning tasks as computation graphs and find that full computation subgraphs appear much more frequently in training data for correct predictions than incorrect ones. |Razeghi et al. (2022) show correlation between frequency in training and test performance, supporting the pattern matching hypothesis, while Nezhurina et al.) 2024) and McCoy et al. (2023) demonstrate the fragility of reasoning LLMs on simple tasks.
Our work builds upon these findings by introducing GSM-Symbolice ic,, an improved benchmark using symbolic templates to generate diverse question variants. This allows us to study math- ematical reasoning ability beyond single performance metric. By evaluating performance on different instantiations and difficulty levels, we draw comprehensive picture of LLMs' reasoning capabilities. Related this direction, Stolfo et al. (2023) evaluate LLMs math word problems by measuring sensitivity and robustness to different imput interventions and report that models often rely surface-level cues rather than genuine reasoning. while Hong al. (2024) developed an ontology and used it to perturb the GSM8K dataset with the aid of LLMs, and [Srivastaval al. (2024) purposes functional variant of MATH dataset. Our findings in this work support the hypothesis that current LLMs are not capable of performing formal mathematical reasoning and pave the way for further research.
3 GSM-SYMBOLIC
The GSMBK dataset (Cobbe et al. 2021) includes over 8000 grade school math questions and an- swers, divided into 7473 training and 1319 test examples. As shown in Fig. [ the questions relatively simple, requiring knowledge of only the four main arithmetic operations. However, since GSM8K is single, popular test set, there is a risk of data contamination, and performance may change significantly with minor modifications to the questions. These limitations have led to efforts to generate new datasets and variants. 1GSM Yee et al. 2024) math dataset created through synthetic pipeline that captures parameter dependencies in hierarchical and graph structure. GSM-IC (Shi et al., 2023) shows that irrelevant context can impair LLM performance, focusing on prompting techniques. Our work, however, suggests more fundamental issue: LLMs struggle even when given multiple shots the same question, indicating deeper challenges in problem- solving that cannot be resolved with few-shot prompting or fine-tuning on unseen distractions or variations of the same or different difficulty levels. GSM-Plut (Li al. 2024a) introduces variants of M&K questions but lacks symbolic templates and has fixed size and difficulty. GSM1 K (Zhang et al. 2024) mirrors the style and complexity of GSM8K to identify systematic overfitting in existing models, but it is not publicly available for researchers.
While the mentioned benchmarks offer a single performance metric on a fixed number of questions, argue that viewing LLM performance as a distribution across various problem instances provides deeper insights. The design of GSM- Symbol ic enables the generation of numerous instances and allows for finer control over question difficulty. We believe our paper contributes to this direction by offering a reliable evaluation framework that underscores the importance of generating multi- ple instances to assess LLMs' mathematical capabilities and their robustness to diverse problem difficulties and augmentations.
3.1 TEMPLATE GENERATION
Given specific example from the test set of GSM8K, we create parsable templates as shown in Fig, (right). The annotation process involves identifying variables, their domains, and necessary conditions to ensure the correctness of both the question and the answer. For instance, since the questions are grade-school level, a common condition is divisibility to ensure the answer is a whole number. We common proper names (e.g., persons, foods, currencies) to streamline creation. After creating the templates, we apply several automated checks to ensure the annotation process is correct. For example, verify that none of the original variable values appear in the template.
4

---

Published as a conference paper at ICLR 2025
We also check that the original values satisfy all conditions and that the final answer matches the original question's 's answer. Once data are generated, 10 random samples per template are reviewed manually. As a final automated check, after evaluating all models, verify that at least two models answer each question correctly; otherwise, the question reviewed manually again.
When constructing symbolic templates, we deliberately select numerical ranges that closely align with those in the original GSMBKtest set. This decision reflects our focus on assessing logical rea- soning capabilities rather than arithmetic skills. Our analysis in Appendix [A.6] confirms that the expanded ranges remain within boundaries where models maintain their arithmetic accuracy.
3.2 EXPERIMENTAL SETUP
While we provide further details on our experimental setup and evaluation in the Appendix, we briefly review the important aspects here:
Models. Throughout this work, we report on more than 20 open models of various sizes, ranging from 2B to 27B. Additionally, we include state-of-the-ard closed models such as GPT-4o-mini, GPT- 4o, ol-mini, and ol -preview.. To conserve space, present results for few selected models in each experiment, but the full results for all models are available Tab. of the Appendix A.2
Evaluation Setup Overall, for this work, we conducted nearly 500 total evaluations on various setups. To this end, we maintained a manageable dataset size by using 100 templates and generating 50 samples per template, resulting in 5000 total examples for each benchmark. Therefore, we have 50 datasets of 100 examples each, where each example is a mutation of one of the original 100 examples from GSM8K. Unless stated otherwise, and other math benchmarks that includes Chain-of- Thoughh (CoT) prompting with 8-shots with greedy decoding. However, we note that in preliminary experiments, the number of shots did not significantly change the performance and conclusions. We provide our prompt template in Fig.
follow common evaluation setup on GSM8K
EXPERIMENTS & RESULTS
In this section, we present our main results and postpone complementary findings to the Appendix. We begin our experiments by addressing an important question regarding the reliability of current reported metrics on GSM8K. By studying the distribution of performance on GSM-Symbol ic, we demonstrate notable performance variation. More importantly, we observe that the performance of models drops on SybboSy (Sec. 4.1).
Next, we investigate the fragility of reasoning in LLMs by comparing performance distributions when only proper names are changed versus when values and numbers are altered. Our findings indicate that the original GSM8K performance of models is much closer to the performance distribu- tion when only names are changed. However, performance drops more significantly when values are changed, with this trend continuing both changes are applied simultancously (Sec. 4.2). We then examine the impact of question difficulty, as indicated by the number of clauses added to or removed from the questions. Our results show that the number of clauses increases, average performance drops, and the variance in performance increases consistently across all models (Sec. 4.3
Finally, in Sec. 4.4 tackle more fundamental question: whether the models truly understand the mathematical concepts. We show that, likely due to potential pattern matching and the fact that the training distribution of models included only necessary information for solving questions, adding seemingly relevant clauses to the question that do not impact the reasoning process required to solve significantly drops the performance of all models.
4.1 How RELIABLE ARE THE CURRENT GSMSK RESULTS?
As our first experiment, we evaluate the performance of several state-of-the-art models on GSM Son--Smboliee The number of samples and difficulty can be adjusted by modifying variable domains, as we will see in subsequent sections. Fig. 2 shows the empirical distribution of the perfor- mance of models GSM-Symbo ic computed on these 50 datasets. As shown, all models exhibit a non non-negligible variance across different sets. For instance, for the Gemma2-99 the gap between the worst performance and the best performance is more than 12%, while for Phi-3.5-m 5-ni, this gap
5

---

Published as a conference paper at ICLR 2025
IJamn in-ttrreet
GPT-4o
GSMSK 87.0
GSM8KK 74.0
GSMSK 95.0
Gym-oly 7911 (23.0)
Symbolld 74.6 (42.9)
ASM-Symbbllic 919
Prequeney
Hespomayy
Proquemey
GSM Symbolic Accuracy (%6 (be CVT)
GSM mymooe Accuracy (56) (86 CoT)
GSM Symbolic Accuracy (%6 (8s COT)
Phii---mddium- 125 instruct
Phi-3.5-mici-inea mimmmii-insttuuet
Mattastml-7b-ol..
GSM8K
GSMBK
GSM8K
GSM-SSSyy SSSM-Suubllic 82.5 (42.9)
GSM Sym-hyyullli 82.1 (43.4)
GSM-Syymbllt 74.0 (43.5)
15
13
Eoouancy
Foquuncy
Prequncc
70
50
GSM Symbolic Accuracy (%) CUT)
GSM Symboli Accuracy
CoT)
GSM Symbolic Accurcy (%) CoTy
Figure 2: 8-shot CoT performance across 50 sets generated from GSM-Symbo 1 templates. All state-of-the-are models exhibit notable variance accuracy. interesting that the majority of the models, the performance GSM8K (represented by dashed line) falls on the right side of the distribution, which statistically speaking, should have a very low likelihood.
Models
GSMSK - CSM Symbblli Acrmany Drop (%)
Pli-s S-mnini-
GPT So min
Miltral 70-x01
GPT-30
Phi Phi----min D..---mmi
ol-min
Pli- coneediam
Genumo
Gomam 27
Lloaat s0
Plu smaall
Misscol 7b +0.3
Madi tostrrs 7hw-0011
30
00-it
Gonmand Genmand 30--
Mistral 7b.tt vil.
7.4
7.4
Figure 3: The performance of all state-of-the art models on GSM-Symbo 1 i a drops compared to GSM8K. Later, we investigate the factors that impact the performance drops more depth.
around 1.5%. It is interesting that this variation exists, the only differences across different instances of each question are the changes in names and values, while the overall reasoning steps needed solve a question remain the same.
Another noteworthy observation is that the performance (represented by the dashed line in Fig. 2 on the original questions from the 100 examples of GSM8K used templates is often than one standard deviation away from the center of the GSM-Symbor ic performance distribution, frequently on the right side of the distribution (this holds for 21 out 25 models). One explanation for this could be data contamination, where some of the test examples from GSMBK inadvertently ended up the training set of these models, leading to an optimistic bias in performance. Fig. 3 shows the performance drop from GSM8K to GSM-S ic for several models. We can see that for models such as Gemma2-9B, Phi-3, Phi-3.5, and Mathstral-7BB the dashed line in Fig. 2 lies on the right side, and the drop in performance is higher than for models such Llama3-8b and GPT-4o, where the performance on GSM8K is close to the center of the GSM- GSM-Symbo Symbo c distribution and the drop in performance is negligible. In Appendix A.3 present further results to support this claim for other models such as Phi-2 and Mistral-7B 7B. These results lead us to investigate the fragility of the reasoning abilities of LLMSs in the next section.

---

Published as a conference paper at ICLR 2025
Gemma3--0hitt
LJoma3 3-b-innetrrct
Phi-3-mediim-- 2Skeiinetmet
(SMMK 87.0
CSSMBK
CSSMBK 89.0
Namas (2211)
Namas
15
Nambor
15
Namber
Rrepmeyy
Both
Freqwaay
20-
20
Systlele Accransy (55) (5e OZT)
GSM Symbelic Accuncy (5) (5s CUT)
CSN Sysbhlic Accmncy (8)- CoT)
Phil3-small 128k instruct
P.s-335 mini instrrcct
Mathetral 7b-v0.1
GSMSS 2013
GS3SSK 86.1
GSMISE 8010
Namme 85.4 (#1.))
Namso 80.1 (#1.8)
Stamer 8ED (41.))
Nimber 50 (1220)
Numihess sta (12.4)
Nambers 72.3 (1300))
Toquene
Both 83.7 (1244
Foqqusec
Beth 82.1 [15
Foqueees
Boek 740 (1333)
85
55
10
70
75
80
CS3 Sysalolle Actuany (5)-(5i (5a OdT)
CSS Synzlele Accuncy (55 (5s CcT)
GSM Syszhlli Acmuayy (5) 1& CoT)
Figure 4: How sensitive are LLMs when we change only names, only proper numbers, or both names and numbers? Overall, models have noticeable performance variation even if we only change names, but even more when we change numbers or combine these changes.
4.2 How FRAGILE IS MATHEMATICAL REASONING IN LARGE LANGUAGE MODELS?
In the previous sub-section, observed high performance variation across different sets generated from the same templates, along with a performance degradation compared to the original GSM8K accuracy. This suggests that the perceived reasoning process of language models may not be formal and is hence susceptible to changes. One explanation is that these models attempt to perform a kind of in-distribution pattern-matching. aligning given questions and solution steps with similar ones seen in the training data. As no formal reasoning is involved in this process, it could lead to high variance across different instances of the same question. In this sub-section and the next one, we investigate these observations further and we show that several factors contribute to the performance variation of the models. First, we investigate the impact of the type of change to understand the difference between changing names (e.g., person names, places, foods, currencies, etc.) versus changing numbers (i.c., the values of variables).
Figure 4 demonstrates that while performance variation persists, the variance is lower when chang- ing names compared to numbers. Notably, the original GSM8K accuracy of models now much closer the center of the changed proper names distribution, in contrast changed numbers or both. Furthermore, a gradual shift in the means of distributions from right to left, along with an increase in variance, is evident across almost all models. It is both striking and concerning that such performance variance exists when only changing proper names, as this level of variability would not be expected from grade-school student with genuine mathematical understanding.
From the results in this section, we observe that by increasing the difficulty of changes (from names numbers). the performance drops and the variance increases, overall suggesting that the reason- ing capabilities of state-of-the-arr LLMs are fragile for the aforementioned reasons. Assuming that LLMs are not performing formal reasoning, how important the question difficulty on the distribu- tion of performance? In the next section, we study this question further.
4.3 HOw DOES QUESTION DIFFICULTY AFFECT PERFORMANCE DISTRIBUTION?
The results in the previous subsection motivate us to study the impact of question difficulty on the mean and variance of the performance distribution. To this end, generate several new tem- plates from the Sym,, as illustrated in Fig. 5 First, by removing one clause, we obtain add GSM-Symbol Mi nus-1 GSM-M1 for short. Similarly,
clauses
7

---

Published as a conference paper at ICLR 2025
Different Levels of GSM-Symbolic Difficulty
68M-Syabolic-Mi: To nake call from phone booth, you muat $0.6 for each ninute of your call. aéter mimbtee, thot price deopo per minute How would 60-minute call
OSM OSM-Synbolic: To make call from a phone booth, you must pay $0.6 for each ninute of your call. After 10 minuten, that price dropa $0.5 per minuta.
would 60-ninute ceat?
To make call from hotel roon phone, must $0.5 for each ninute of your call After 10 minutes, that price drops to 80.5 per ninute. After 25 ninutes from the start of the call, the price drops
GBM-Synbolic-PI:
$0.3 per minute Nox much would 60-minute call cost?
CSM-Synbolic-P2: To make a call from hotel roon phone, you must pay $0.6 for each ninute of your call. After ninutes, price drops 80.5 per ninute. of the call, the price drops even nore to $0.3 per minute. If your total bill is more than $10, you get 25% discount. How much sould 60-minute call coat?
25 ninutes the
Figure 5: Modifying the difficulty level of GSM-Symbolic by modifying the number of clauses.
Gemma2-9b-it "GSM-Mii' st.(222
Phi-3. 5-mini-insttre
Phi---mediin-1129--nnttttt
GSMM-X 87.6(12.0))
GSMM-Mi 89 60.6(117))
GSSM-Symb 79.1(=3.0)
CSM-Symb 82.(+334)
GSM-Symb 823(+229)
Frequency
CSM-P] 68.1 68.1(44488
Frequency
GSM-PI 64.((45.4)
Frequency
CSMPPI 75.8(+3.9)
GSM-P2 (1.8(16.0)
GSM-P2 (1.8(16.3)
GSM-P2 53.1(21.8)
10
10
to
40
60
80
40
60
601
70
80
50
Accuracy (%) (8s CoT)
Accuracy (%) (8s Co
Accuracy (96) (8s C6T)
GPT-4o-mini GSM--1 $2,5(116)
GPT-4o
o1-mini
GSM-Xi 51 11166)
GSM--M 919()1)))
GSM--Symb 01.7(422.0)
GSM-Symb 96.9(11.)
GSM-Symm 94.5(11))
Frequency
15
GSM-P1 (1:31)11)
Frequency
15
GSM-PP 98.9(=2.))
Frequency
15
GSM-P1 913
GSM-P2 72.44+466)
GSM-PP 8800--34)
GSM-P2 8n 1(+3.6)
to
10
100
Accuracy (%) (8s CoTT
Accuracy (96) (8s CoT)
Accuracy (96 (8s CoT)
Figure 6: The impact of increasing the number of clauses on performance: As the difficulty increases from GSM-M1 GSM- Symb-+ GSM-P1-> GSM-P2, the distribution of performance shifts the left (i.e., accuracy decreases), and the variance increases.
to the questions to increase the difficulty, resulting in CSM-Symbolic-Plus-1 (GSM-P1) and ESM--Smmblicc-Pe
(GSM-P2), respectively
As shown in Fig. 6 the trend of the evolution of the performance distribution is very consistent across all models: as the difficulty increases, the performance decreases and the variance increases. Note that overall, the rate of accuracy drop also increases as the difficulty increases. This is in line with the hypothesis that models are not performing formal reasoning, as the number of required reasoning steps increases linearly, but the rate of drop seems to be faster. Moreover, considering the pattern-matching hypothesis, the increase in variance suggests that searching and pattern-matchine become significantly harder for models as the difficulty increases.
Note that adding or removing a clause does not necessarily correspond to increasing or decreasing the number of required reasoning steps by exactly one. However, our main focus in this section is to understand the evolution of the performance distribution rather than the precise performance numbers.
8

---

Published as a conference paper at ICLR 2025
GSM-NoOp
Oliver picks 44 kiwis on Friday. Then be picke 58 kivie on Saturday. Sunday, be picks double the number of kivis did on Priday, but five of then bit snaller than average Hou many kivis does Oliver have?
o1-mini:
Oliver picks kivis over three days follous:
Priday: kivis
Saturday: 58 kivis
Sunday: Double the number he picked on Friday, which is 2 88 kivis
subtract the Sunday total: 88 (Sunday's kivis) - (smaller kikis) - 83 kivis Now, suminng up the kivis fron days: (Priday) (Saturday) (Sunday) kivis Oliver total kivis.
Hovever, Sunday, of
snaller average.
Mana3-8E BE: Let's break dovn step by step:
Oliver picks 44 kivis on Priday. He picks 68 kivis on Saturday. On Sunday, be picks double the 44 kivis.
numbar of kivia did Friday, which
Five of the kivis picked on Sunday are bit smaller than average, so subtract 5 from the total up
numbar
picked an Sunday:
kieta. Nou,
Oliver has: 44 (Friday)
(Saturday) (Sunday) 185 kiwis
So. Oliver has 185 kivis total.
Figure 7: An example from the GSM- dataset: We add seemingly relevant statements the questions that are, in fact, irrelevant to the reasoning and conclusion. However, the majority of models fail ignore these statements and blindly convert them into operations, leading mistakes.
4.4 CAN LLMS REALLY UNDERSTAND MATHEMATICAL CONCEPTS?
In the previous sections, we studied the impact of type change and difficulty the performance distribution. In this section, we demonstrate that models are susceptible to catastrophic perfor- mance drops on instances not part of the training distribution, potentially due their reliance on in-distribution maathhnn. We introduce GSM-NoOp, a dataset designed to challenge the rea- soning capabilities language models. To create the templates, we add seemingly relevant but ultimately inconsequential statements to GSM-Symbolic templates. Since these statements carry no operational significance, we refer to them as "No-Op".
Fig. 7 iillustrates example from GSM-NoOp. An interesting observation is that models tend to blindly subtract the number of smaller fruits, potentially because their training datasets included similar examples that required conversion subtraction operations. In the Appendix, additional failure cases from -o00. Overall, we find that models tend to convert statements to operations without truly understanding their meaning. For instance, common case observe is that models interpret statements about "discount" as "multiplication" regardless of the context. This raises the question of whether these models have truly understood the mathematical concepts well enough. Consequently, as shown in Fig. Sa there is a catastrophic performance decline across all tested models, with the Phi-3-mini model experiencing over 65% drop, and even stronger models such as ol -preview showing significant declines.
include
To better understand this performance drop, we conducted another experiment. While our previous evaluations on GSM-P2 used the original 8-shots of GSM8K, here we explore two new scenarios where we change the source of the 8-shots. We report the results in Figures 8h and Sc
NoOp-Symb (Using GSM-Symbol ic shots of the same question): During evaluation, we include 8 different shots of the same question coming from GSM-Symbbolic. Hence, each shot provides the required reasoning steps. The target question from GSM-NoOp then presents yet another variation of the same question that different only in values and the added clause that inconsequential. This setup should simplify the task by making clear that the extra information in the target question is irrelevant. However, as shown Fig. 8b) the performance remains within the standard deviation, with 8 shots of the same question providing the reasoning chain. Interestingly. Fig. &c shows that some models can perform significantly better, even though they don't 't perform nearly as well on GSM8K and SSMMSymbolic. We believe this is very notable observation.
NoOp-NoOp (Using GSM-NoOp shots of different questions): Here, we provide 8 shots chosen randomly from different questions of GSM-NoOp in the context. These questions share the common

---

Published as a conference paper at ICLR 2025
Pitskennduu- (8l--nnstrrct
Gemma20
al --ppooaee
177.5
20055
S-llet Accurees (86)
Acconsee(())
Sisteal 20-11-1-33-300
Mistral 7a 11.1
28.0
Leetnnd-22ee 2lwi
-11
-220
Quaetiie
Quartiiia
Mistril-7o 3
10
Kordy
Mistral-7hesk1
44.0
Mothor Accuracy
AccuracyURi 2
-5.7
Geamno2-27h-11
Geontoz 0b-g
43.0
Semnoe2th
43.0
N:dy Seab
GR2
Doutie
Sirrce
2000p
GSMBK - GSM NoOp Accuracy Drop(X)
(a)
(b)
(c)
Figure 8: (a) The performance of models drops significantly GSM-NoOp, with more recent mod- els experiencing greater decline than older ones. (b) As previously demonstrated, performance on GSM-Symboll
very close that GSM8K. However, GSM-NcOp, the significant drop in performance cannot be recovered, even when using the exact same question's variation as shots (NoOp-Symbl when using different questions with different GSM-NoOpthat contain No-Op operations -o0ppNNOp) as shots. (c) Notably, some models that perform significantly worse than show much better performance on NoOp- Symb.
those (b) GSM8K and GSM-Symbon
fact that the correct answer should ignore the No-Op statement. We observe that for the Llama-3-8B model, the performance remains the same compared to the original No-Op model, while for the Phi-3 model, performance slightly decreases.
5 CONCLUSION
In this work, we have investigated the reasoning capabilities of large language models (LLMM) and the limitations of current evaluations on GSM8K. We introduced GSM-Symbolic, novel bench- mark with multiple variants designed to provide deeper insights into the mathematical reasoning abilities of LLMs. Our extensive study reveals significant performance variability across different instantiations of the same question, challenging the reliability of current GSM8K results that rely on single hingle-point accuracy metrics. We found that while LLMs exhibit some robustness to changes in proper names, they are more sensitive to variations in numerical values. We have also observed the performance of LLMS deteriorating question complexity increases.
The introduction of GSM-NoOp exposes a critical flaw in LLMs' ability to genuinely understand mathematical concepts and discern relevant information for problem-solving. Adding seemingly relevant but ultimately inconsequential information to the logical reasoning of the problem led to substantial performance drops across models. Importantly, even when provided with multiple examples of the same question or examples containing similar irrelevant information. This suggests deeper issues in their reasoning processes that cannot easily mitigated through few-shot learning or fine-tuning.
demonstrate that LLMs struggle
Ultimately, our work underscores significant limitations in LLMS' ability to perform genuine math- ematical reasoning. The LLMs' high performance variance on different instances of the same ques- tion, their significant drop in performance with slight increase difficulty, and their sensitivity to inconsequential information indicate that their reasoning is fragile and may be more akin to sophis- ticated pattern matching rather than true logical reasoning. We believe further research is needed to develop Al systems capable of formal reasoning, moving beyond probabilistic pattern matching to achieve more robust and generalizable problem-sol -olving skills. This remains a critical challenge for the field as we strive create systems with human-like cognitive abilities general intelligence.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
The authors would like to thank Max Horton, Fartash Faghri, Moin Nabi, Devi Krishna, and Arsalan Farooq for the valuable feedback and support.
REFERENCES
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat S. Behl, Alon Benhaim, Misha Bilenko, Johan Bjorek, Sébastien Bubeck, Martin Cai, Caio César Teodoro Mendes, Weizhu Chen. Vishrav Chaudhary, Parul Chopra, Allie Del Giorno, Gustavo de Rosa, Matthew Dixon, Ronen Eldan, Dan Iter, Amit Garg, Abhishek Goswami. Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Jamie Huynh, Mojan Javaheripi, Xin Jin, Piero Kauffmann, Nikos Karampatziakis, Dongwoo Kim, Mahoud Khademi, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Chen Liang. Weishung Liu, Eric Lin, Zeqi Lin, Piyush Madan, Arindam Mi- tra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez-Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Corby Rosset, Sambudha Roy, Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Xia Song, Masahiro Tanaka, Xin Wang, Rachel Ward. Guanhua Wang, Philipp Witte, Michael Wyatt, Can Xu, Jiahang Xu, Sonali Yadav, Fan Yang, Ziyi Yang, Donghan Yu, Chengruidong Zhang. Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou. Phi-3 technical report: A highly capable language model lo- cally on your phone. CoRR, 0is/2404.142199 2024. doi: 4044850/ARXXV:2444..M0 lttpss:/ddi..rr//2 115//110488888
URL
48550/ http//riiiiv.. 2404. 2404.14219
Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean rean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Jo- han Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler. Tim- othy Lillicrap. Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Isard, Paul Ronald Barham, Tom Hemnigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, George Tucker, Enrique Piqueras, Maxim Krikun, lain Barr, Nikolay Savinov, Ivo Dani- helka, Becca Roclofs, Anaïs White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Mehran Kazemi, Lucas Gonzalez, Misha Khalman, Jakub Sygnowski, and et al. Gemini: A fam- ily of highly capable multimodal models. CORR, abs/2312. 1885, 2023. doi: 10.48550/ARXIV. 2312. 18805. URL/httss:// ate...org/11 48544 /at.0//axxvvv.. 2312. 1380.11805 11805.
Enric Boix-Adserà, Omid Saremi, Emmanuel Abbe, Samy Bengio, Etai Littwin, and Joshua M. Susskind. When transformers reason with abstract symbols? In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenRe- view.net, 2024. URL htt.htttts///peeeeiiiit
net n
Ronald Brachman and Hector Levesque. Knowledge representation and reasoning. Elsevier, 2004.
François Chollet. On the measure of intelligence. arXiv preprint 1911v:191 2019.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.144 14168, 2021.
David Cole. The Chinese Room Argument. In Edward N. Zalta and Uri Nodelman (eds.), The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Winter 2024 edition, 2024.
Grégoire Delétang, Anian Ruoss, Jordi Grau-Moya, Tim Genewein, Li Kevin Wenliang, Elliot Catt, Chris Cundy, Marcus Hutter, Shane Legg, Joel Veness, and Pedro A. Ortega. Neural networks and the chomsky hierarchy. In The Eleventh International Conference Learning Representations, ICLR 2023. Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https://cpenrevieew net Forum http://ooummmddoooaeeeeeeee

---

Published as a conference paper at ICLR 2025
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Amirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song. Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Carcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guan Pang. Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, and et al. The llama 3 herd of models. CoRR, abs/2407.21783, 2024. doi: 10.48550/ARXI.2247.2211 21783
URL https: /tap::://do.... /oii.i..oo. org 48150/00..5550///111/
Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D. Hwang, Soumya Sanyal, Sean Welleck, Xi- ang Ren, Allyson Ettinger. Zaid Harchaoui, and Yejin Choi. Faith and fate: Limits of transformers on compositionality. 2023.
Gilles E. Gignac and Eva T. Szodorai. Defining intelligence: Bridging the gap between human and artificial perspectives. Intelligence, 2024. URL https://app. https///aap..eennnccccchch...aa. orgg
Corpus1D:26901500
Tom Gunter, Zirui Wang. Chong Wang, Ruoming Pang, Andy Narayanan, Aonan Zhang, Bowen Zhang, Chen Chen, Chung--Chngg Chiu, David Qiu, Deepak Gopinath, Dian Ang Yap. Dong Yin, Feng Nan, Floris Weers, Guoli Yin, Haoshuo Huang. Jianyu Wang, Jiarui Lu, John Pee- bles, Ke Ye, Mark Lee, Nan Du, Qibin Chen, Quentin Keunebroek, Sam Wiseman, Syd Evans, Tao Lei, Vivek Rathod, Xiang Kong, Xianzhi Du Yanghao Li, Yongqiang Wang, Yuan Gao, Zaid Ahmed, Zhaoyang Xu, Zhiyun Lu, Al Rashid, Albin Madappally Jose, Alec Doane, Al- fredo Bencomo, Allison Vanderby, Andrew Hansen, Ankur Jain, Anupama Mann Anupama, Areeba Kamal, Bugu Wu, Carolina Brum, Charlie Maalouf, Chinguun Erdenebileg, Chris Dul- hanty, Dominik Moritz, Doug Kang, Eduardo Jimenez, Evan Ladd, Fangping Shi, Felix Bai, Frank Chu, Fred Hohman, Hadas Kotek, Hannah Gillis Coleman, Jane Li, Jeffrey P. Bigham, Jeffery Cao, Jeff Lai, Jessica Cheung, Jiulong Shan, Joe Zhou, John Li, Jun Qin, Karanjeet Singh, Karla Vega, Kelvin Zou, Laura Heckman, Lauren Gardiner, Margit Bowler, Maria Cordell, Meng Cao, Nicole Hay, Nilesh Shahdadpuri, Otto Godwin, Pranay Dighe, Pushyami Racha- pudi, Ramsey Tantawi, Roman Frigg, Sam Davarnia, Sanskruti Shah, Saptarshi Guha, Sasha Sirovica, Shen Ma, Shuang Ma, Simon Wang, Sulgi Kim, Suma Jayaram, Vaishaal Shankar, Varsha Paidi, Vivek Kumar, Xin Wang, Xin Zheng, and Walker Cheng. Apple intelligence foun- dation language models. CoRR, abs/2407.21077, 2024. doi: (0-48500/ARXIV.440.. 240752 URL. https: //doi www.or//1 /0/.000.40 48550885//aaa. w1t.550arriv.2202
21075
Pengfei Hong, Navonil Majumder, Deepanway Ghosal, Somak Aditya, Rada Mihalcea, and Sou- janya Poria. Evaluating ilms' mathematical and coding competency through ontology-guided interventions. arXiv preprini arXiv:2401.09395. 2024.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Ant Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. CoRR, abs/2310.06825 2023. doi: 1044550/ARXXV.231000068 URL 06825
/ttp:///...0011...555/00//////// 2310.
Bowen Jiang, Yangxinyu Xie, Zhuoqun Hao, Xiaomeng Wang, Tanwi Mallick, Weijie J. Su, Camillo J. Taylor, and Dan Roth. A peek into token bias: Large language models are not yet genuine reasoners. CoRR, abs/2406. 1050, 2024. doi: 10.48550/ARXVV220666 10550. URL
https:://ol http:///ooo..r/11....0
2406.
Subbarao Kambhampati. Can large language models reason and plan? Annals of the New York www.olrarroooroooooooooooo
Academy of Sciences, 1534:15 18, 2024. hRLhttps::/ /app./appp..///
CorpusTD:26824996:
12

---

Published as a conference paper at ICLR 2025
Shane Legg and Marcus Hutter. Universal intelligence: A definition of machine intelligence. Minds and machines, 17:391-444) 2007.
Qintong Li, Leyang Cui, Xueliang Zhao, Lingpeng Kong, and Wei Bi. Gsm-plus: A comprehensive benchmark for evaluating the robustness of Ilms as mathematical problem solvers. In Lun-Wet Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACL 2024, Bangkok, Thai- Association for Computational Linguistics, 2024a.
land, August 11-16, 2024, pp.
URL https::/aacaatthheeeet
2024 aww-lcllnnggo 163
Zihao Li, Yuan Cao, Cheng Gao, Yihan He, Han Liu, Jason M. Klusowski, Jianqing Fan, and Mengdi Wang. One-layer transformer provably learns mec-nearest neighbor in context. 2024b. URL
https://api wew.nttescaaaaceeeaeaoooaoa ort//oorpus
Zhiyuan Liu, Hong Liu, Denny Zhou, and Tengyu Ma. Chain of thought empowers transform- ers to solve inherently serial problems. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL net/ Fo
htttp::/
Gary F Marcus. The algebraic mind: Integrating connectionism and cognitive science. MIT press, 2003.
R. Thomas McCoy, Shunyu Yao, Dan Friedman, Matthew Hardy, and Thomas L. Griffiths. Embers of autoregression: Understanding large language models through the problem they are trained to solve, 2023. URL /nttts::/arxtie org/ 2tpp://bs//309 2309. 13638
Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette-Choo, Clément Crepy, Daniel Cer, Daphne Ippolito, David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, Crorge-Cristian Muraru, Grigory Rozhdestvenskly. Henryk Michalewski, Ian Tenney, Ivan Gr- ishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean- can--Bptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, and et al. Gemma: Open models based on gemini research and technology. CoRR, abs/2403.08295, 2024. doi: 10.48350/ARXIV.2403.. 08295. URL pettps://dol:. org.ooo..rg.g/ 400..48550///// www.xxiv..400..00 08295
Allen Newell and Herbert Simon. The logic theory machine-a complex information processing system. IRE Transactions on information theory, 2(3):61-79, 1956.
Marianna Nezhurina, Lucia Cipolina- Mehdi Cherti, and Jenia Jitsev. Alice in wonderland: Simple tasks showing complete reasoning breakdown in tfate-of-the-at large language models. arXiv preprint arXiv:24466.006// 2024.
OpenAl. URL learmiigg-to-ressoo-ithtt
report. CoRR, htt.:aa3..//3377..4...4 2023. doi: 10.4555/AARXV22230007744
OpenAl. Learning to reason with large language models.
https: /wwp//.pnaaaa../////// iiteena.ccommndiii cwmm/idex/
2024. Accessed: 2024-099-2.
Judea Pearl. Probabilistic reasoning intelligent systems: networks of plausible inference. Elsevier, 2014.
Binghui Peng, Srini Narayanan, and Christos H. Papadimitriou. On limitations of the transformer
architecture. CoRR, abs/2402.08164, 2024. doi: 10:48550/AXX12240200666 URL https:
// wttg//10.4550/arra/
2402. 240.0.61642200.....4. 08164]
Yasaman Razeghi, Adam Roberts, Colin Raffel, and Ariel Herbert- Voss. Impact pretraining term frequencies few-shot reasoning. arXiv preprint 2200:22002.08004 2022.
13

---

Published as a conference paper at ICLR 2025
Morgane Rivière, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piotr Stanczyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Jean-Bastien Grill, Behnam Neyshabur, Olivier Bachem, Alanna Walton, Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchison, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Perry, Chris Welty, Christopher A. Choquette- Choo, Danila Sinopalnikov, David Weinberger, Dimple Vi- jaykumar, Dominika Rogozinska, Dustin Herbison, Elisa Bandy, Emma Wang, Eric Noland, Er- ica Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Klimczak-Plucinska Harleen Batra, Harsh Dhand, Ivan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Peng Zhou, Joana Carrasqueira, Joana Njjazi, Jocelyn Becker, Joe Fernandez, Joost van Amersfoort, Josh Gordon, Josh Lipschultz, Josh Newlan, Ju-yeong Ji, Kareem Mohamed, Kartikeya Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen, Kiranbir Sodhia, Kish Greene, Lars Lowe Sjösund, Lauren Usui, Laurent Sifre,, Lena Heuermann, Leticia Lago, and Lilly Mc- Nealus. Gemma 2: Improving open language models practical size. CoRR, abs/2448.001 18, 2024. doi: 40.48550/ARXIV.24088.. 0.08.00118
URL https: http://c..oooroo wttp:///.....r.......0 http:///....55///X....
Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models a mirage?, 2023.
Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H. Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant con- text. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023, 23- 29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Re- search, pp. 31210-31227. PMLR, 2023. URL 1222/shi23a.1o
mml p.rsss/
Saurabh Srivastava, Annarose M B, Anto P V, Shashank Menon, Ajay Sukumar, Adwaith Samod T, Alan Philipose, Stevin Prince, and Sooraj Thomas. Functional benchmarks for robust evaluation of reasoning performance, and the reasoning gap, 2024. URL
2402. 19450
Alessandro Stolfo, Zhijing Jin, Kumar Shridhar, Bernhard Schoelkopf, and Mrinmaya Sachan. A causal framework quantify the robustness of mathematical reasoning with language models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 545-561, acl-long 32/
2023. URL http:p/////aaanchooooooooo/////
Karthik Valmeekam, Alberto Olmo Hernandez, Sarath Sreedharan, and Subbarao Kambhampati. Large language models still can't plan (A benchmark for Ilms on planning and reasoning about change). CORR, abs/2206. 10498, 2022. doi: 10.48550/ARXIV2220666 10498. URL. https: 2206 10498
//doi. aecii.org/11 10.
Karthik Valmeekam, Kaya Stechly, and Subbarao Kambhampati. Llms still can't plan; can Irms? a preliminary evaluation of openai's ol on planbench. 2024. URL httpps://pp..
semanticscholase www.oooaaa.rggoooouuuuuuuua 22:
Jason Wei, Xuezhi Wang, Dale Schuurmans, et al. Chain-of-thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201. 11903, 2022.
Gail Weiss, Yoav Goldberg, and Eran Yahav. Thinking like transformers. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Re- search, pp. 1 11080-1100 11080-11090 11900 PMLR, 2021. URLhttp: http::::/ppooceedngg..oo mlr prtss//1/s// we ss21 html
14

---

Published as a conference paper at ICLR 2025
Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. Physies of language models: Part 2.1, grade-school math and the hidden reasoning process. arXiv preprint arXiv:2407.20311, 2024.
Honghua Zhang, Liunian Harold Li, Tao Meng, Kai-Wei Chang, and Guy Van den Broeck. On the paradox of learning to reason from data. In Edith Elkind (ed.), Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23, pp. 3365-3373. International Joint Conferences on Artificial Intelligence Organization, 8 2023. Main Track.
Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Dylan Slack, Qin Lyu, Sean Hendryx, Russell Kaplan, Michele Lunati, and Summer Yue. A careful examination of large language model performance on grade school arithmetic. CoRR. abs/2405.00333 2024. doi: 48550/arXiv 2405. 00332
URL stttps://oil org/10.
Hattie Zhou, Arwen Bradley, Etai Littwin, Noam Razin, Omid Saremi, Joshua M. Susskind, Samy Bengio, and Preetum Nakkiran. What algorithms can transformers learn? A study length gener- alization. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vi- enna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL http:tss//peeeeeiieee.cco forum? id=Ass HHnMHX
Nikola Zubic, Federico Soldà, Aurelio Sulser, and Davide Scaramuzza. Limits of deep learning: Sequence modeling through the lens of complexity theory. In The Thirteenth International Con- ference Learning Representations, 2025. URL https: wwwpreveee.eeeeeeee net forum?
id=DDddMM.3FdM)
15

---

Published as a conference paper at ICLR 2025
A
APPENDIX
In this appendix, we provide additional details to the main text, including:
A.1
Detailed experimental setups, including the prompt template.
A.2
Full results CSM8K, CSM-Symbolice and their variants.
A.3
Additional results for the distributional performance of several models, similar to the
results from Sec. in the main text.
A.4
Additional results for Sec. 4.3 where we studied the impact of question difficulty. We
show that fine-tuning easier tasks does not necessarily improve performance on more difficult tasks.
A.5 more comprehensive discussion and analysis of performance for OpenAl ol-mini and ol -preview models.
A.1 DETAILED EXPERIMENTAL SETUP
In this work, all reported evaluations results use 8-shots with chain-of- off-thutttthtt prompting, We use the following prompt format:
Evaluation Prompt Format
// preamble or system instruction
As an expert problem solver, solve step by step the following aathenstical questione.
// shot-1
((question))
A: Let'ss think step by step. ((solution)). The final ansver is (ffinal ansver)).
1/
Q: [[question]]
step by step. ((solution)).
final ansver (ffinal ansver)).
target question
([queetion))
Let's think step by step.
Figure The prompt format used for evaluations.
Except for the last experiment in Sec. 4.4 we allow the models to generate until either their context size limit is reached, they generate one of the ond-of-response tokens such as '
' or "</endoffee current question and move to generating the next question, indicated by another "Q:' generation.
we use the original shots from GSM8K. addition,
|>', or they finish answering the
Finally, we note that all experiments we use greedy decoding to generate responses from models, with exception: currently, the available APIs for "ol-mini" and "ol "ol-preview" models do not allow controlling the decoding strategy, and it seems that at the time of writing, these models do not perform greedy decoding, responses to the same prompt change.
A.2 FULL RESULTS
In Tab. present the comprchensive performance results of various models, including Gemma (Mesnard et al. 2024). Gemma2 (Rivière et al. 2024). Phi (Abdin et al. (024). Mis- tral (Jiang al. 2023 Llama3 (Dubey al. 2024), GPT-4o (OpenAl 2023), and the ol (OpenAl 2024 series, on GSM8K and its different variants, CSM-Symbolic.
We report two sets of results for GSMBK: the first column indicates the accuracy the full test set of GSM8K (comprising 1,319 examples), while the second column shows the accuracy on a subset of 100 questions from the GSM8K test set, which randomly selected to generate GSM-Symbo) ic
16

---

Published as a conference paper at ICLR 2025
templates. It is noteworthy that the performance levels across both sets very similar, with significant differences observed.
Table Full 8-shot results of all models on GSM8 different variants GSM SMMM-Symboic..
GSM8K (Full)
GSM8K (100)
Model
Symbolic-M1
Symbolic
Symbolic-Pi
Symbolic-P2
Symbolic-NoOp
Gemma2b
12.1
11.0
24.5 (- 3.85)
8.2 2.21)
(+ 2.13)
15 1.3)
4+ 1.99)
GemmaZb-1
3.28)
(- 2.21)
(L 1.49)
15 1.63)
2.48)
Gemma-7b
53.8
50.0
34,1 (- 4.41)
2-.6
26.0 5.30)
3. 1.92)
8+ 2.71)
Gomma-7b-it
29.3
(- 4,41)
25
(.388)
3+ 1,921
8.7 2.71)
Gemma2-2h 2h
47.5
46.0
57.2 3.40)
40.1 40.1 (+33.04)
(9 3,89)
1.3
(8
Gemma2-2b-it
47.5
46.0
57.2 3.40)
(+
19.5 3,89)
45 1.94)
15.7 3,97)
Gemma2-9b
85.3
87.0
71,2 2.81)
4.00 (2:29)
41.8 6.00)
22
Gomma2-9b--
85.3
84.4 2.36)
79.1 (+
68.1 4.77)
41.8 6.00)
(2.11
Gemma2-276-H
89.7
92.0
90.2 1.86)
88.3 (2
80.7 (# 4.07)
63.4 4.14)
30.0 3.39)
Phi-2
56.0
53.0
$3.0 3.10)
3.56)
23 4.07)
8.9
11.2 3.51)
Pti-3--mini-128e
128k-instruct
80.7 (#
5.63)
1el-small-128k-instte
88.5
89.0
86.4 1.95)
83.7 2.65)
72.0 3.65)
50.7
24.5 (2 4.8))
Pti--3-medium- 28k-instruutt
1.65)
(#
3.89)
4.18)
Phi 3.5 mini instruct
84.9
88.0
87.6 1.98)
82.1 3.38)
64.8 (1 5.43)
44.8 (1 6.32)
22.4 (1 4.03)
Mistral-7b-x0.1
44.5
48.0
55.4 (L 3.18)
41.1 (1.36))
17.4 4.82)
5.5
16.2 4.43)
39.7
42.0
44,9 4.29)
30.5 3.477)
13.1 (# 3,51)
4.0 (.244
10.1 (# 3.42)
Mistral-7b-v0.3
40.6
44.0
54.0 2.95)
15.6 (.L 4.02) (1
2.31)
16.7 [1 4.26)
Mistral-7h-inee Saai---biiittttttO....
56.2
62.3 2.68)
50.0 3.49)
24.5 (+ 4.34)
10.8 3.60)
15.9 (+ 4.44)
Mabbstral-7b--O.11
80.1
80.0
82.9 2.87)
74.0 (.499
57.4 (.L 5.20)
35.5 (1L 5.07)
20.4 (1L 3.58)
Llama3-8b
3.62)
(2: 2.94)
(1: 4.54)
3.86)
Llama3 Sb instruct
76.0
74.0
79.5 (a. 3.62)
74.6 (1: 2.94)
53.8 (.. 4.54)
28.3 (1. 4.37)
18.6 (:.. 3.86)
GPT GPTT-4-mim
94.2
95.0
92.5 (L 1.63)
91.7 91.7 (.022)
81.1 (1 3.05)
72.4 (1 4.57)
54.1 (1 3.85)
GPT-4o
95.2
94.4 1.62)
94.9 1.87)
93.9 2.59)
88.0 3.43)
63.1 (+ 4.53)
ul-mmiai
95.1
93.0
94.9 1.49)
94.5 1.58)
94.3 (1 2.57)
89.1 (1L 3,56)
66.0 (1 4.60)
ol-preview
94.9
93.6 1.68)
92.7 1.82)
95.4 1.72)
94.0 2.38)
77.4 3.84)
Phi-2
Mitral-7b-instrees-sd...
Gemmmn2--2--t
GSMSK 53.0
GSMSK 12.0
GSMSK 46.0
GSM-Symbolie 41.4 (43.6)
GSM-Symbolic 30.5 (43.5)
GSM--ymmolle 40.1 (=3.0)
Preqpene
P-Value <0.001
Erequeercy
V.Vuue <20.001
P. Value <0.0011
25
GSM Symbollc Accuracy (%6 (8e CVT)
GSM mybbe Accuracy (%) (86 CoT)
GSM Symbolic Accuracy (%) (8s CoT)
Gemma2-9b-it
Phi-3.5 Phi-3.5-mini-issttee
Mathstrad77-v0..
GSM8K 79.1 (=3.0)
GSMBK
GSM8K
18
GSS--Sybblii
GSM-Symbblic $2.1 (#3.4
GSM-Symbolic 74.0 (+3.5)
Eroqenry
P.Valu <0.001
Evoquncc
P. P-Value <0.001
Preeuancy
P. Valuue <0.001
75
80
85
70
50
GSM Symbolic Accuracy (%) CoT)
GSM Symboke Accuracy CoT)
GSM Symbolic Accurcc (%) CoT)
-lama3-Sb-instrron
CPT-40
GPT-4o-mini
GSMMK 74.0
GSMBK 95.0
GSSMSK 95.0
GSM-Symbolld 71.6 (=2.9)
GSM--ymbelic 9199)911
GSM-Symbolic 91.7 (#2.0)
Prequenry
P-Value 0.14
Stequnncy 5.
P-Value
Tropamec
P-Value <00.001
to
10
75.0
96
14
GSM Symbolie Accuracy (36)
(86 CVT)
GSM Symbohe Arcunay (36) 18
(cT)
GSM Symbolle Accuracy (%6 (8s CoT)
Figure 10: Additional results on performance variation on GSM- Syybolicc..
17

---

Published as a conference paper at ICLR 2025
A.3 ADDITIONAL RESULTS ON GSM-SY SYMBLIII PERFORMANCE DISTRIBUTIONS
In section 4.1 we have presented results for several models in Fig. 2 Here, we provide additional results showing the performance on GSM SoM--Symolicc for other models also have high variance. Moreover, these models correspond to highest drop.
Another important question regarding our results measuring the statistical significance In this work, we calculate statistical significance using the one-sample t-test to determine whether the 50 different performance results on GSM- Symbolic differ from the original GSM8K score (i.e., the null hypothesis [ As we can see in Fig. 10 for an overwhelming majority of models (except Llama3-88 and GPT-4o),, the results are statistically significant.
A.4 ABLATION: DOES TASKS?
ON EASIER TASKS HELP WITH MORE DIFFICULT
In Sec. 4.3 we observed that the performance on GSM-P P2 significantly lower than perfor- mance GSM-P1. We also argued that it is unlikely that additional fine-tuning including shots from GSM-P1 would be beneficial. Here, in Fig. Tal we show that including shots from GSM-P1 does not improve performance compared the results where shots come solely from GSM8K.
Moreover, in Fig. 11b demonstrate that fine-tuning Phi-3.5 GSM-P1 slightly improves perfor- mance on GSM while decreasing performance on GSM-P2. We have used set of 50 templates from GSM P1, separate from the test templates, and generated 10000 examples for finetuning train- ing set.
Overall, while this direction warrants further research, current results suggest that scaling training data will not be helpful in improving the reasoning capabilities of language models.
96
From GSMSK
44
GSM-P2 (ecuracy())
P1
6SM-P2 Accuracy
GRMPP Accurayy
40
30
I.AamoS 8b it
Ph-
Plas.5
Gammno
Matletral Tb vo.1
medimm
mini
9b
Epoche Finetamed on Pi-train
(a)
(b)
Figure 11: Using in-context shots finetuning GSM-P1 does not improve performance on GSM P2: (a) Compared to the case where 8 shots come from GSM8K, when we include shots from GSM P1 the performance on G2M-P2 does not improve. (b) Finetuning GSM- P1 can im- prove performance on GSM-P1 but not on GSM-P2.
A.5 RESULTS ON 0l OR-PREVIEW AND 01 -IMIINI
The recently released ol op-pevve and ol-mini models (OpenAI 2024) have demonstrated strong performance various reasoning and knowledge-based benchmarks. As observed in Tab. [ the mean of their performance distribution is significantly higher than that of other open models.
"Tiis very complicated topic, as the notion statistical significance depends on many assumptions about the properties of data. For instance, one could view the overall performance of models as a Bernoulli trial, with model accuracy representing the probability of success. However, this requires an i.idd. assumption the questions and the model accuracy for each question, which may not necessarily hold and needs further investigation.
'In general, the t-test assumes normality, which may not necessarily hold for evaluation results. How- ever, using standard normality test based on the skewness and kurtosis of samples, we have verified that all distributions in Fig. pass the normality (p-value of the normality test = 0.1).
18

---

Published as a conference paper at ICLR 2025
In Fig. 12 (top), illustrate that both models exhibit -enligglii performance variation. When the difficulty level altered, ol oi-mini follows a similar pattern to other open models: as the difficulty increases, performance decreases and variance increases.
ol-mini
ol-preview
GSM8K 93.0
GSM8K 96.0
GSM-Symbolic 04.5 (+1.6)
GSMM-Smmbliic 92.7 (+1.8
Frequency
P-Value <0.001
Frequency
P-Value <0.001
10
10
50
92
94
86
88
90
92
94
96
GSM Symbolic Accuracy (%) (8s CoT)
GSM Symbolic Accuracy (%) (8s CoT)
ol-mini
ol-preview
20
dSM-MM (4.5)
20
GSM-MI 93.6 11.7
GSM-Symb 94.5(=16)
GSM-Symb 92.
15
Frequency
GSM-PI 94.3(126).
Frequency
GSM-P1 95.4 95.4(_1.7)
GSM-P2 89.1(=3.6)
GSM-P2 94
10
0
87.5
100.0
Accuracy (%) (8s CoT)
Accuracy (%) (8s CoT)
Results on ol -min and ol-preview: both models mostly follow the same trend we pre- sented in the main text. However, ol or-preve shows very strong results on all levels of difficulty as all distributions are close to each other.
Figure 12:
The op-preveew model demonstrates robust performance across all levels of difficulty, as indicated by the closeness of all distributions. However, important note that both ol o1-preview and ol-mini experience significant performance drop on GSM-NoOp In Fig. 13 we illustrate that ol ol-preview struggles with understanding mathematical concepts, naively applying the 10% inflation discussed in the question, despite it being irrelevant since the prices pertain to this year. Additionally, in Fig. [4]
we present another example highlighting this issue.
Overall, while ol -p-pevweww and ol-mini exhibit significantly stronger results compared to current open models- potenialpo due improved training data and post-training procedures- still share similar limitations with the open models.
19

---

Published as a conference paper at ICLR 2025
GSM-NoOp
Lian wanta to buy some achool suppliea. He buya 24 erasers that nov coat $6. each, 10 notebooks $11.0 each, now, assuring that due to inflation, prices vere 10% cheaper last year?
pager
should pay
01-preview: To solve the problem, ve need to calculate hov much Lian should pay, considering that inflation. means calculate prices
year's prices reducing the current prices by 10%.
cheaper
Step Calculate last year's prices by reducing the current prices 10%.
Braser price last year: $6.75 0.90 $6.075
Notebook
year: $11.00 0.90 89.90
Bond paper price last year: 819.00 0.90 = $17.10
Step 2:
Calculate the total cost using last year's prices.
Total cost for erasers: 24 erasers 86.075 = $145.80
Total for notaboaks:
notebooks $9.90 $99.00 Total cost. for bond paper: I rean x 817.10 $17.10
Step 3:
Sun all the costs to get the total anount Lian thould
+ $261.90
$281..90
Sample response from op-prvve example from GSM-NoOp: the model blindly applies the inflation rate, even though the inflation amount is irrelevant as the question clearly indi- cates the given prices for "now" and not last year.
Figure 13:
20

---

Published as a conference paper at ICLR 2025
GSM-NoOp
leaf of sourdough the aourdough and 10 mufrina, how nuch more do the sourdough loaves cost coupared to the muffine, if plan
Nuffins each. purchasa
sourdough muffins purchase?
01-preview:
solve this problen, we need to compare the total cost of the sourdough loares and muffins after considering donated
Step
kept donations. Sourdough leaves kept: 10 leaves donated = loaves Huffine žept: 10 muffine donated muffine
Step 2:
Calculate the total cost of the itans kept. sourdough
7 loaves Total cost of muffine: 8 muffins x $3 = 824
$63
Step 3:
Find the difference in total cost.
Difference: $63 $21 = $39
Ansver:639
o1-mini:
stop by step doternine nore sourdough compared to the miffins after accounting for the donations.
Initial Purchase Costs
Sourdough Loaves:
Price per $9
Quantity purchased: 10
10
Naffins:
Price per muffin: $3
Quantity purchased:
Total Coet: 10 muffine $3 $30
Value
Donated Sourdough Loaves:
Quantity donated:
Value: loaves 39 $27
Donated Murfina:
Quantity donated:
Value: 2 muffins $3 86
Costa Donation
find the effective cost of what you're actually keeping (1.e., not donating):
Sourdough
Quantity kept: 10 purchased 3 donatedd =
Cost: (total) 827 (donated)
Nuffins:
Quantity kept: 10 purchased donsted
Net Cost: $30 (total) (donated) $24
4. Difference in Xet Coste Sourdough Loaves: $63
Maffins: $24
Difference: $63 $24 839
Conclusion accounting
sourdough
muffins.
Figure 14: Sample response from ol ol-preveew and ol-mini example from GSM-NoOp: while the donation amount is irrelevant to the price difference, the models subtract the amount we donate.
21

---

Published as a conference paper at ICLR 2025
A.6 ABLATION: THE IMPACT ARITHMETIC ACCURACY
An important question regarding the design of GSM-Symbolic is: when designing templates, "how should the numerical range of the variables be chosen?" and "how much of the performance drop can be attributed to arithmetic mistakes?".
G5 GSAISK-tee
GSM-Symboli
GSM-PI
GSM P2
B.
Perseetage (90
Prrceroeg (50)
Peccentagn
Fizal Anmer Digits
Final Azener Digits
Final Anrwe Digis
Final Anrwer Digts
Figure 15: Statistics of the number of digits in the final answer for questions in GSM8K-test, GSM- Symbolic, GSM-P1, and GSM-P2. Compared to GSM8K-test, the GSM-Symbolic set shows a slight reduction in the number of questions with I-digit 2-digit final answers, and an increase in 3-digit answers. However, overall, the range of numbers does not increase significantly, with a significant majority of the final answers containing fewer than digits.
Porsonaag (58
Civ
Piessttag aoagangg (55
div
Posestagg (36)
50b
50b
Poreutagn
5th
Intermediate Auswerr Lights
Intermediste Anseaz
Digil
Intermediatte Ansous Dighs
Lucerannnaate Ansvec Dight
Figure 16: Statistics the Number of Digits in Intermediate answers: Compared to GSM8K, GSM- Symbolic and its variants involve operations that result 3-digit numbers. However, as shown in Fig.1 17 modern LLMs well capable of performing arithmetic within this range.
Gemmaa2-hbit
Gemmn2-0biit
Gemmuu 2-
Aritimetie Accuacy
100
Soleracter E
Arithmerii Acemary
100
GSMMMI 844((=24)
Milliinn
Drissin
G5M--yymb 19.1(23.0)
Erequency
GSM-PI 68.((+4.8)
GSM-P2 (1.8(+60)
to
-0
000.7%
100000
1000 00.00%
100 00%
iin 0%
Diit 20%
00 15%
29%
106 10%
06 80%
100 100,005
Final Answer Digits
Final Answer Digits
Accuracy (%) (8e CoT)
(a)
(b)
(c)
Phi-3-medlumm-I 22keiistre
Phi-3-medium-
1e 128l-instruet
medium- 128k
Arithmetic Accuzacy
Soltrctionn 6)
Arithmetic Acemacy
Dorion [-]
(SALMM 6666(1177
MAllio
GSM-Syyub 82.5(129))
16
Mallopoctton
Frequency
GSMLPI 75.8(13.9)
CSM-P2 x3.1=448)
15
1
100003
910 10%
T0000S
11R00%
95, S-%
0:00
70
Final Answer Digits
Final Auswer Digits
Accuracy (%) (8s CoT)
(d)
Figure 17: Arithmetic accuracy of Gemma2-9B and Phi3-Medium: (a) and (d) The addition ac- curacy of the models remains nearly perfect for calculations involving up to 4 digits. Note that, as shown in Fig. 15] the majority the final answers in GSM-Symbolic and its variants are below 5 digits. (b) and (e) Although GSM-PI and GSM-P2 have a similar degit-length distribution to GSM- Symbolic, illustrated in Fig. 15 the performance drop is significant.
22

---

Published as a conference paper at ICLR 2025
Overall, when creating the symbolic templates, have chosen numerical ranges close to their original values the GSM8K test set. The rationale behind this choice is that the arithmetic ability of the models is much less important than their logical reasoning capabilities. However, we face challenge: we need to generate many instances per question. Given that not all values in the range satisfy the conditions and hence won't be selected, need slightly increase the numeric range of numbers to ensure enough instances are generated from each template.
However, we show that this adjustment does not push the numerical values into a range where models have low arithmetic accuracy. As observed in Fig. 15
Compared to GSM8K, GSM-Symbolic has more 3-digit final answers and fewer I-digit and 2-digit answers. However, shown in Fig. modern LLMs such Gemma2-98 have difficulty with up to 3-digit addition and multiplication.
Even though GSM-P1 and GSM-P2 have very similar digit distribution to GSM-Symbolic and are mostly within the range that modern LLMs can perform accurate arithmetic (Fig. the performance drop on these benchmarks is very significant (Fig. 17c).
Additionally, Fig. 16 provides statistics on the different operations involved in calculating interme- diate answers across various datasets. Here, "intermediate" refers to the result of each intermediate operation throughout the solution process. For example, if a solution step involves 12 x 6 = 72, we categorize this two-digit multiplication. We observe that, overall, the distribution of the digit lengths in intermediate answers in Fig. [6 is similar to that in the final answers in Fig. 15
Moreover, in Fig. Gemma2-9B and Phi3- Meddim are capable of nearly perfect addition and subtraction up to length of 4, as well as achieving very high accuracy in multiplication and division. To this end, for each digit length of the answer, we ask the model in zero-shot manner what the result of the operation would be. For example, for addition, we use the prompt "What is x plus y and for multiplication, ask "What is times y?", and arithmetic accuracy on every possible combination numbers. However, we ensure that for every digit length and every operation, we have least 1,000 test cases.
demonstrate that modern Language Learning Models (LLMs) such as
To reduce the computational burden, do not check
Finally, in Table 2 we report the arithmetic accuracy results Gemma2-99 -bbit on a more realistic setup where calculations extracted from the generated responses. To this end, we extracted parts of the generated responses where the equal sign (=) was generated along with numerical values, and then evaluated whether the left-hand side of the equation had the numerical value the right- hand side. As shown in the table, we observe very high arithmetic accuracy, similar to the previous results in this section.
Table 2: Ablation: Arithmetic accuracy of Gemma2-9b-it on different GSM benchmarks.
Benchmark
1-digit
2-digits
3-digits
4-digits
5-digits
All
GSM8K
99.8
99.1
99.3
96.8
95.9
98.9
GSM-Symbolic
99.7
99.2
97.6
97.2
99.1
98.3
GSM-PI
97.5
98.6
98.7
96.2
99.5
97.4
GSM-P2
96.2
98.6
97.5
96.4
99.2
97.1
Overall, we believe it is unlikely that arithmetic difficulty account for the significant performance drop on benchmarks such as GSM-P2, which has a very similar distribution to GSM-Symbolic and falls well within the range of fewer than digits that models like Gemma2-99 M-BB can handle accurately.
23


Reviews:
[{"review_id": "4wTQfanAL0", "reviewer": "Reviewer_Yt9o", "summary": "This paper creates a novel dataset (GSM-Symbolic) with several different variants to test whether LLMs are capable of generalized reasoning. GSM-Symbolic has several variants which change the numbers involved in the problem and/or removes or adds clauses that change the difficulty level of the problem but not in a way that is too different from the original problem. They benchmark across a number of models and show that many LLMs show significant drops when changing the numbers only slightly or adding irrelevant information.\n\nI have some questions that I raise in the sections below. In particular, I think some of the claims are not fully supported by the provided evidence and that it's not clear that you can make strong, broad claims about LLMs as opposed to claims about the particular models tested. Nevertheless, I believe this paper to be both high quality and important work and think that it should be accepted to the conference.\n\nNits:\n\nl84, citation format error\nL759: spacing error", "questions": "1. The abstract claims that “Specifically, the performance of all models declines when only the numerical values in the question are altered in the GSM-Symbolic benchmark.” However, if I’m reading this right, this is not what you report in Table 1 in A.2? For example, o1-mini and both Llama3 8B models seem to get a higher score on Symbolic compared to GSM8k (100).\n\n2. One potential concern I have with the paper is that. Especially given (1), how do you differentiate between results that apply to LLMs more broadly compared  claims that apply broadly to specific LLMs? In particular, several models in your evaluations are well known to have interesting results regarding overfitting on GSM8k as per works in your current related works section.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "Paper tests a very good question. The state of LLMs is very strange right now. They can clearly solve very tricky problems. At the same time, they often fail in very elementary ways as well. This paper is an exceptional analysis of this, especially testing across several different variants of GSM-Symbolic. In general, I think this paper is a fantastic contribution to the field and as a result vote accept.", "weaknesses": "Some weaknesses are referenced in the questions section. One additional question: would the authors be able to include statistical significance results in the Appendix or main results? I think this would significantly improve the paper.\n\nSecondly, while the GSM-Noop experiments are very interesting, I think there is a large difference in the claim (also made by prior work) that LLMs are bad at handling irrelenvant context and them not performing reasoning.\n\nRelated Work: I think Srivastava+ 2024 should be cited and compared to, as it is a very similar method on the MATH dataset with similar-ish findings.\n\n@misc{srivastava2024functionalbenchmarksrobustevaluation,\n      title={Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gap}, \n      author={Saurabh Srivastava and Annarose M B and Anto P V au2 and Shashank Menon and Ajay Sukumar and Adwaith Samod T and Alan Philipose and Stevin Prince and Sooraj Thomas},\n      year={2024},\n      eprint={2402.19450},\n      archivePrefix={arXiv},\n      primaryClass={cs.AI},\n      url={https://arxiv.org/abs/2402.19450}, \n}\n\n@misc{shi2023largelanguagemodelseasily,\n      title={Large Language Models Can Be Easily Distracted by Irrelevant Context}, \n      author={Freda Shi and Xinyun Chen and Kanishka Misra and Nathan Scales and David Dohan and Ed Chi and Nathanael Schärli and Denny Zhou},\n      year={2023},\n      eprint={2302.00093},\n      archivePrefix={arXiv},\n      primaryClass={cs.CL},\n      url={https://arxiv.org/abs/2302.00093}, \n}", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HASlFJdurz", "reviewer": "Reviewer_7pZc", "summary": "The paper \"GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models\" presents a thorough investigation into the mathematical reasoning capabilities of large language models (LLMs) using a newly developed benchmark, GSM-Symbolic. This benchmark, derived from the GSM8K dataset, includes modifications to test the robustness of LLMs against changes in numerical values, proper names, and question complexity.\n\n**Key Contributions:**\n\n1. **Introduction of GSM-Symbolic**: The paper introduces GSM-Symbolic, a benchmark that generates diverse and controllable mathematical question variants based on symbolic templates. This allows for more precise evaluations of LLMs by modifying elements like numbers and text within the questions to assess their impact on model performance.\n\n2. **Empirical Analysis of LLM Performance**: Through extensive testing with GSM-Symbolic, the authors demonstrate that LLMs exhibit significant performance variability and sensitivity to minor changes in question structure. Their experiments reveal that even small perturbations, such as altering numerical values or adding irrelevant clauses, can significantly degrade the performance of state-of-the-art models.\n\n3. **Exploration of Reasoning Fragility in LLMs**: The study details how the introduction of irrelevant information (via the GSM-NoOp dataset) into mathematical problems significantly impacts model accuracy, highlighting a critical flaw in the current models' ability to discern relevant from irrelevant data.\n\nOverall, the paper provides a deeper understanding of the limitations inherent in current LLMs regarding their mathematical reasoning capabilities and offers a comprehensive framework for future research to explore and mitigate these weaknesses.", "questions": "**Q1: Computational Complexity Impact**\nCould the authors provide a detailed analysis of how computational complexity affects the accuracy of the models tested? Specifically, it would be informative to understand the proportion of errors that can be attributed directly to the failure of numerical computations versus those caused by reasoning errors. This differentiation could help clarify the extent to which computational demands influence the performance of large language models on the GSM-Symbolic dataset.\n\n**Q2: Impact of Fine-Tuning on Model Robustness**\nWould the authors consider conducting experiments where some models are fine-tuned on the GSM-Symbolic dataset and its variants to assess whether this training approach enhances their robustness to the types of perturbations introduced in your study? This could provide valuable insights into whether the observed limitations are inherent to the model architectures or can be ameliorated through targeted training, potentially challenging the paper's conclusions about the fundamental nature of these limitations.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "**Strengths:**\n\n1. **Comprehensive Experimental Design**: The authors explore various perturbations in mathematical questions such as changing names or numbers, adjusting the number of clauses, and introducing irrelevant information. These detailed experiments provide robust evidence of the fragility exhibited by prominent language models under slight modifications, thereby contributing significantly to our understanding of their limitations.\n\n2. **Development of the GSM-Symbolic Dataset**: The introduction of the GSM-Symbolic dataset and its variants is a notable contribution. By coding 100 problems from the GSM8K dataset and creating corresponding modifications, the authors have developed a valuable resource that can facilitate future research into model robustness.\n\n3. **Relevance and Depth of Analysis**: The topic is highly relevant to current discussions in the field, and the paper delivers extensive empirical data concerning critical questions about the robustness of current language models when subject to various perturbations.", "weaknesses": "**Weaknesses:**\n\n1. **Oversight of Computational Complexity**: The paper does not sufficiently address the computational challenges posed by the range of parameter values (5 to 100) used in their experiments. This range suggests that computing expressions like x + y + z, particularly with all parameters as two-digit numbers, could introduce significant computational errors unrelated to the models' reasoning capabilities. The potential impact of these computational difficulties on the results, particularly evident in Figure 4, suggests that the observed distribution shifts in Figure 2 might be attributed more to computational challenges than to data contamination.\n\n2. **Ambiguity in Defining Reasoning**: The paper's definition of reasoning needs clarification. Asserting that models capable of only three-step inferences lack reasoning capabilities seems overly restrictive. Since the GSM8K focuses on the NLP aspect of reasoning rather than the length of reasoning required, it naturally tailors models to excel within these confines. Thus, introducing additional clauses or irrelevant details, as done in the experiments, effectively renders the questions out-of-distribution (OOD). This shift likely degrades performance due to the models not being trained to handle such modifications, which doesn't necessarily indicate a lack of reasoning ability but rather a limitation in the training paradigm.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gfVDW6OhVv", "reviewer": "Reviewer_wG4b", "summary": "This paper investigates the reasoning capabilities of LLMs using the GSM8K benchmark and introduces a new benchmark called GSM-Symbolic. The authors highlight the variability in model performance, particularly noting that LLMs struggle more when numerical values are altered compared to when only proper names are changed. They also explore the impact of question difficulty on performance, revealing that as complexity increases, both average performance and variance decline. The findings emphasize the need for more reliable evaluation methodologies and further research into the reasoning abilities of LLMs, suggesting that while these models show potential, their understanding of mathematical concepts remains limited", "questions": "1. For numerical variables, how did the authors choose their boundary? Why x belongs to (5, 100) in Figure 1? I think this range may also influence the performance, especially when some extreme values are selected.\n2. To add clause in GSM-NoOp, we can see results drop a lot in Figure 8, but what is the standard to add this extra clause? Because adding clause can be subjective if authors already know some patterns will make the model fail.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "+ GSM-Symbolic provides a strategy to enrich the math reasoning datasets. The symbolic template can be applied for other datasets.\n+ The NoOp design is innovative, it shows undiscussed disadvantage of current LLM on reasoning tasks.\n+ The experiment design is adequate and valid. This paper evaluates 25 models, the results provide a comprehensive overview of the current SOTA methods.", "weaknesses": "- Figure 3 shows the chatgpt-o1 and GPT-4o has tiny accuracy drop. But even without GSM-Symbolic, running any LLM models for 50 times on the same dataset can cause this variance.\n- For numerical variables, how did the authors choose their boundary? This is not explained in the paper\n- To add clause in GSM-NoOp, we can see results drop a lot in Figure 8, but what is the standard to add this extra clause? Because adding clause can be subjective if authors already know some patterns will make the model fail.\n- Part of the beginning paragraph of section 3 should belong to related work discussion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Et7In3N51T", "reviewer": "Reviewer_FK55", "summary": "The paper \"GSM-SYMBOLIC: Understanding the Limitations of Mathematical Reasoning in Large Language Models\" explores the mathematical reasoning capabilities of Large Language Models (LLMs) through the introduction of a new benchmark, GSM-Symbolic. This benchmark aims to address the limitations of existing evaluations like GSM8K by generating diverse question variants and adjusting complexity levels. The study finds that LLMs exhibit performance variance when faced with different instantiations of the same question, particularly when numerical values are altered. It also highlights the fragility of LLMs' reasoning capabilities, as performance deteriorates with increased question complexity and the introduction of irrelevant information. The paper suggests that current LLMs rely on probabilistic pattern-matching rather than robust logical reasoning.", "questions": "1. Can you clearly define \"true logical reasoning\"? What conditions need to be satisfied for a language model to demonstrate true logical reasoning?\n2. Why was the performance of o1 not investigated in certain parts of the study, such as in Figure 3?\n3. Given that some models like GPT4-o show robustness across numerical changes, do the authors believe that scaled training and model size can alleviate the identified limitations? If so, how does this impact the generalizability of the findings?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The study provides empirical evidence of the performance variance and fragility of LLMs when faced with different question instantiations and increased complexity in grade school math questions.\n- The introduction of the GSM-Symbolic benchmark allows for the creation of diverse instances of the GSM8K dataset, which helps avoid data contamination and enables a more comprehensive evaluation of LLMs' reasoning capabilities.", "weaknesses": "- The findings largely reconfirm known limitations of LLMs, such as their reliance on probabilistic pattern-matching and sensitivity to small changes in question statements, which have been explored in depth in previous works, notably in \"Are NLP Models really able to Solve Simple Math Word Problems?\"\n- The methods used to curate the dataset, such as altering numerical values and adding irrelevant information, are not novel and have been employed in other studies, such as \"PAL: Program-aided Language Models,\" which introduces GSM8k-hard.\n- The paper uses ambiguous terms like \"true logical reasoning,\" \"genuine mathematical reasoning,\" and \"formal [reasoning] in the common sense term\" without providing clear definitions of these terms.\n- The paper’s findings on performance degradation with increasing question complexity have been similarly explored in other works as compounding errors in multi-step reasoning processes.\n- The authors suggest that  performance of LLMs on GSM-Symbolic compared to their performance on GSM8k indicates significant limitations in LLMs’ ability to perform genuine mathematical reasoning. However,  performance of GPT4-o remains robust on GSM-Symbolic, as shown in figure 3, suggesting that the performance variance resulting from different numerical instantiation of the questions may be resolved with scaled training or model size and this limitations may not be inherent to all LLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)