GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models

## Abstract

Abstract Recent advancements in Large Language Models (LLMs) have sparked interest in their formal reasoning capabilities, particularly in mathematics. The GSM8K benchmark is widely used to assess the mathematical reasoning of models on grade-school-level questions. While the performance of LLMs on GSM8K has significantly improved in recent years, it remains unclear whether their mathematical reasoning capabilities have genuinely advanced, raising questions about the reliability of the reported metrics. To address these concerns, we conduct a large-scale study on several state-of-the-art open and closed models.
To overcome the limitations of existing evaluations, we introduce GSM-Symbolic, an improved benchmark created from symbolic templates that allow for the generation of a diverse set of questions. GSM-Symbolic enables more controllable evaluations, providing key insights and more reliable metrics for measuring the reasoning capabilities of models.Our findings reveal that LLMs exhibit noticeable variance when responding to different instantiations of the same question. Specifically, the performance of all models declines when only the numerical values in the question are altered in the GSM-Symbolic benchmark. Furthermore, we investigate the fragility of mathematical reasoning in these models and demonstrate that their performance significantly deteriorates as the number of clauses in a question increases. We hypothesize that this decline is due to the fact that current LLMs are not capable of genuine logical reasoning; instead, they attempt to replicate the reasoning steps observed in their training data. When we add a single clause that appears relevant to the question, we observe significant performance drops (up to 65%) across all state-of-the-art models, even though the added clause does not contribute to the reasoning chain needed to reach the final answer. Overall, our work provides a more nuanced understanding of LLMs’ capabilities and limitations in mathematical reasoning.

## 1 Introduction

Large Language Models (LLMs) have demonstrated remarkable capabilities across various domains, including natural language processing, question answering, and creative tasks (Gunter et al., ; OpenAI, ; Dubey et al., ; Anil et al., ; Abdin et al., ; Rivière et al., ). Their potential to perform complex reasoning tasks, particularly in coding and mathematics, has garnered significant attention from researchers and practitioners.

However, the question of whether current LLMs are genuinely capable of true logical reasoning remains an important research focus. While some studies highlight impressive capabilities, a closer examination reveals substantial limitations. Literature suggests that the reasoning process in LLMs is probabilistic pattern-matching rather than formal reasoning (Jiang et al., ). Although LLMs can match more abstract reasoning patterns, they fall short of true logical reasoning. Small changes in input tokens can drastically alter model outputs, indicating a strong token bias and suggesting that these models are highly sensitive and fragile (Jiang et al., ; Shi et al., ). Additionally, in tasks requiring the correct selection of multiple tokens, the probability of arriving at an accurate answer decreases exponentially with the number of tokens or steps involved, underscoring their inherent unreliability in complex reasoning scenarios (Schaeffer et al., ).

Mathematical reasoning is a crucial cognitive skill that supports problem-solving in numerous scientific and practical applications. Consequently, the ability of large language models (LLMs) to effectively perform mathematical reasoning tasks is key to advancing artificial intelligence and its real-world applications. The GSM8K (Grade School Math 8K) dataset (Cobbe et al., ) has emerged as a popular benchmark for evaluating the mathematical reasoning capabilities of LLMs. While it includes simple math questions with detailed solutions, making it suitable for techniques like Chain-of-Thought (CoT) prompting, it provides only a single metric on a fixed set of questions. This limitation restricts comprehensive insights into the models’ mathematical reasoning. Moreover, the popularity and prevalence of GSM8K can increase the risk of inadvertent data contamination. Finally, the static nature of GSM8K does not allow for controllable experiments to understand model limitations, such as behavior under varied conditions or changes in question aspects and difficulty levels.

Figure: Figure 1: Illustration of the GSM-Symbolic template creation process. This dataset serves as a tool to investigate the presumed reasoning capabilities of LLMs, enabling the design of controllable mathematical reasoning evaluations with more reliable metrics. Our results reveal that all state-of-the-art LLMs exhibit significant performance variations, suggesting the fragility or lack of reasoning.

To address these limitations, a more versatile and adaptive evaluation framework is needed—one that can generate diverse question variants and adjust complexity levels to better explore the robustness and reasoning abilities of LLMs. This would facilitate a deeper understanding of the strengths and weaknesses of these models in mathematical reasoning tasks. We make the following contributions:

- •
We introduce GSM-Symbolic, an enhanced benchmark that generates diverse variants of GSM8K questions using symbolic templates (Sec. 3), as shown in Fig. 1. This enables a more nuanced and reliable evaluation of LLMs’ performance across various setups, moving beyond single-point accuracy metrics. Our large-scale study on 25 state-of-the-art open and closed models provides significant insights into LLMs’ behavior in mathematical reasoning tasks.
- •
We question the *reliability* of currently reported results on GSM8K and demonstrate that the performance of LLMs can be viewed as a distribution with unwarranted variance across different instantiations of the same question. We show that the performance of all models drops on GSM-Symbolic (Sec. 4.1), hinting at potential data contamination.
- •
We show that LLMs exhibit more robustness to changes in superficial elements like proper names but are very sensitive to changes in numerical values (Sec. 4.2). We show that performance degradation and variance increase as the number of clauses increases, indicating that LLMs’ reasoning capabilities struggle with increased complexity (Sec. 4.3).
- •
Finally, we further question the reasoning abilities of LLMs and introduce the GSM-NoOp dataset. By adding seemingly relevant but ultimately irrelevant information to problems, we demonstrate substantial performance drops (up to 65%) across all state-of-the-art models (Sec. 4.4). This reveals a critical flaw in the models’ ability to discern relevant information for problem-solving, likely because their reasoning is not formal in the common sense term and is mostly based on pattern matching. We show that even when provided with multiple examples of the same question or examples containing similar irrelevant information, LLMs struggle to overcome the challenges posed by GSM-NoOp. This suggests deeper issues in their reasoning processes that cannot be alleviated by in-context shots and needs further investigation.

Overall, our work provides a comprehensive understanding of the limitations of LLMs in mathematical reasoning. Our results emphasize the need for more reliable evaluation methodologies and further research into the reasoning capabilities of large language models.

## 2 Related Work: Reasoning & Language Models

Logical reasoning is a critical trait of intelligent systems. Recent advancements in Large Language Models (LLMs) have demonstrated significant potential across various domains, yet their reasoning abilities remain uncertain and inconsistent. Many works have investigated whether LLMs are truly capable of reasoning by examining how these models solve tasks requiring logical reasoning.

One interesting direction focuses on modeling the computation performed by transformers. For example, parallels have been drawn between components such as attention and feed-forward modules and simple computational primitives (Weiss et al., ; Zhou et al., ). Delétang et al. () demonstrated that transformers fail to generalize on non-regular tasks and showed that structured memory (e.g., memory tape) is necessary for handling complex tasks. This is related to the effectiveness of Chain-of-Thought (CoT) prompting (Wei et al., ) and using scratchpads for LLMs as additional memory for intermediate computations. Overall, current results suggest that while the transformer architecture has limitations and lacks the required expressiveness for solving problems across several complexity classes, these limitations can be alleviated with additional memory (e.g., scratchpads) (Liu et al., ). However, this still requires generating vast amounts of tokens to solve a problem (Peng et al., ; OpenAI, ). While these works provide insights into the theoretical computational complexity of transformers, in practice, it remains unclear whether these LLMs can perform formal logical reasoning to solve tasks.

There is a considerable body of work suggesting that the reasoning process in LLMs is not formal (Kambhampati, ; Valmeekam et al., , ), even though it appears that these models understand symbols and can work with them to some limited degree (Boix-Adserà et al., ). Instead, LLMs likely perform a form of probabilistic pattern-matching and searching to find closest seen data during training without proper understanding of concepts. While this process goes beyond naive memorization of words and the models are capable of searching and matching more abstract reasoning steps, it still falls short of true formal reasoning. For instance, Jiang et al. () show, with statistical guarantees, that most LLMs still struggle with logical reasoning due to strong token bias, where the reasoning output of the model changes when a single token of input changes. This aligns with our results, which indicate that the performance of models on different instances of the same mathematical question can vary greatly from one instance to another. Li et al. () prove that a single transformer layer learns a one-nearest neighbor, which could explain why the reasoning of models is highly sensitive to input tokens. Schaeffer et al. () argue that when a task requires emitting multiple tokens correctly, the probability of answering correctly decreases exponentially with the number of tokens. Dziri et al. () represent reasoning tasks as computation graphs and find that full computation subgraphs appear much more frequently in training data for correct predictions than incorrect ones. Razeghi et al. () show a correlation between frequency in training and test performance, supporting the pattern matching hypothesis.

Our work builds upon these findings by introducing GSM-Symbolic, an improved benchmark using symbolic templates to generate diverse question variants. This allows us to study mathematical reasoning ability beyond a single performance metric. By evaluating performance on different instantiations and difficulty levels, we draw a comprehensive picture of LLMs’ reasoning capabilities. Our findings support the hypothesis that current LLMs are not capable of performing formal mathematical reasoning and pave the way for further research on this important topic.

## 3 GSM-Symbolic

The GSM8K dataset (Cobbe et al., ) includes over 8000 grade school math questions and answers, divided into 7473 training and 1319 test examples. As shown in Fig. 1, the questions are relatively simple, requiring knowledge of only the four main arithmetic operations. However, since GSM8K is a single, popular test set, there is a risk of data contamination, and performance may change significantly with minor modifications to the questions. These limitations have led to efforts to generate new datasets and variants. iGSM (Ye et al., ) is a math dataset created through a synthetic pipeline that captures parameter dependencies in a hierarchical and graph structure. GSM-IC (Shi et al., ) shows that irrelevant context can impair LLM performance, focusing on prompting techniques. Our work, however, suggests a more fundamental issue: LLMs struggle even when given multiple shots of the same question, indicating deeper challenges in problem-solving that cannot be resolved with few-shot prompting or fine-tuning on unseen distractions or variations of the same or different difficulty levels. GSM-Plus (Li et al., ) introduces variants of GSM8K questions but lacks symbolic templates and has a fixed size and difficulty. GSM1K (Zhang et al., ) mirrors the style and complexity of GSM8K to identify systematic overfitting in existing models, but has a fixed number of examples, and is not publicly available for researchers.

While the mentioned benchmarks offer a single performance metric on a fixed number of questions, we argue that viewing LLM performance as a distribution across various problem instances provides deeper insights. The design of GSM-Symbolic enables the generation of numerous instances and allows for finer control over question difficulty. We believe our paper contributes to this direction by offering a reliable evaluation framework that underscores the importance of generating multiple instances to assess LLMs’ mathematical capabilities and their robustness to diverse problem difficulties and augmentations.

### 3.1 GSM-Symbolic: Template Generation

Given a specific example from the test set of GSM8K, we create parsable templates as shown in Fig. 1 (right). The annotation process involves identifying variables, their domains, and necessary conditions to ensure the correctness of both the question and the answer. For instance, since the questions are grade-school level, a common condition is divisibility to ensure the answer is a whole number. We use common proper names (e.g., persons, foods, currencies) to streamline template creation. After creating the templates, we apply several automated checks to ensure the annotation process is correct. For example, we verify that none of the original variable values appear in the template. We also check that the original values satisfy all conditions and that the final answer matches the original question’s answer. Once data are generated, 10 random samples per template are reviewed manually. As a final automated check, after evaluating all models, we verify that at least two models answer each question correctly; otherwise, the question is reviewed manually again.

### 3.2 Experimental Setup

While we provide further details on our experimental setup and evaluation in the Appendix, we briefly review the important aspects here:

Models. Throughout this work, we report on more than 20 open models of various sizes, ranging from 2B to 27B. Additionally, we include state-of-the-art closed models such as GPT-4o-mini, GPT-4o, o1-mini, and o1-preview. To conserve space, we present results for a few selected models in each experiment, but the full results for all models are available in Tab. 1 of the Appendix A.2.

Evaluation Setup
Overall, for this work, we conducted nearly 500 total evaluations on various setups. To this end, we maintained a manageable dataset size by using 100 templates and generating 50 samples per template, resulting in 5000 total examples for each benchmark. Therefore, we have 50 datasets of 100 examples each, where each example is a mutation of one of the original 100 examples from GSM8K. Unless stated otherwise, we follow a common evaluation setup on GSM8K and other math benchmarks that includes Chain-of-Thought (CoT) prompting with 8-shots with greedy decoding. However, we note that in our preliminary experiments, the number of shots did not significantly change the performance and conclusions. We provide our prompt template in Fig. 9.

Figure: Figure 2: The distribution of 8-shot Chain-of-Thought (CoT) performance across 50 sets generated from GSM-Symbolic templates shows significant variability in accuracy among all state-of-the-art models. Furthermore, for most models, the average performance on GSM-Symbolic is lower than on GSM8K (indicated by the dashed line). Interestingly, the performance of GSM8K falls on the right side of the distribution, which, statistically speaking, should have a very low likelihood, given that GSM8K is basically a single draw from GSM-Symbolic.
Refer to caption: x1.png

## 4 Experiments & Results

In this section, we present our main results and postpone complementary findings to the Appendix. We begin our experiments by addressing an important question regarding the reliability of current reported metrics on GSM8K. By studying the *distribution* of performance on GSM-Symbolic, we demonstrate notable performance variation. More importantly, we observe that the performance of models drops on GSM-Symbolic (Sec. 4.1).

Next, we investigate the fragility of reasoning in LLMs by comparing performance distributions when only proper names are changed versus when values and numbers are altered. Our findings indicate that the original GSM8K performance of models is much closer to the performance distribution when only names are changed. However, performance drops more significantly when values are changed, with this trend continuing as both changes are applied simultaneously (Sec. 4.2). We then examine the impact of question difficulty, as indicated by the number of clauses added to or removed from the questions. Our results show that as the number of clauses increases, average performance drops, and the variance in performance increases consistently across all models (Sec. 4.3).

Finally, in Sec. 4.4, we tackle a more fundamental question: whether the models truly understand the mathematical concepts. We show that, likely due to potential pattern matching and the fact that the training distribution of models included only necessary information for solving questions, adding seemingly relevant clauses to the question that do not impact the reasoning process required to solve it significantly drops the performance of all models.

### 4.1 How Reliable Are the Current GSM8K Results?

As our first experiment, we evaluate the performance of several state-of-the-art models on GSM-Symbolic. The number of samples and difficulty can be adjusted by modifying variable domains, as we will see in subsequent sections. Fig. 2 shows the empirical distribution of the performance of models on GSM-Symbolic computed on these 50 datasets. As shown, all models exhibit a non-negligible variance across different sets. For instance, for the Gemma2-9B, the gap between the worst performance and the best performance is more than 12%, while for Phi-3.5-mini, this gap is around 15%. It is interesting that this variation even exists, as the only differences across different instances of each question are the changes in names and values, while the overall reasoning steps needed to solve a question remain the same.

Another noteworthy observation is that the performance (represented by the dashed line in Fig. 2) on the original questions from the 100 examples of GSM8K used as templates is often more than one standard deviation away from the center of the GSM-Symbolic performance distribution, frequently on the *right side* of the distribution (this holds for 21 out of 25 models). One explanation for this could be data contamination, where some of the test examples from GSM8K inadvertently ended up in the training set of these models, leading to an optimistic bias in performance. Fig. 3 shows the performance drop from GSM8K to GSM-Symbolic for several models. We can see that for models such as Gemma2-9B, Phi-3, Phi-3.5, and Mathstral-7B, the dashed line in Fig. 2 lies on the right side, and the drop in performance is higher than for models such as Llama3-8b and GPT-4o, where the performance on GSM8K is close to the center of the GSM-Symbolic distribution and the drop in performance is negligible. In Appendix A.3, we present further results to support this claim for other models such as Phi-2 and Mistral-7B.
These results lead us to investigate the fragility of the reasoning abilities of LLMs in the next section.

### 4.2 How Fragile is Mathematical Reasoning in Large Language Models?

In the previous sub-section, we observed high performance variation across different sets generated from the same templates, along with a performance degradation compared to the original GSM8K accuracy. This suggests that the perceived reasoning process of language models may not be formal and is hence susceptible to changes. One explanation is that these models attempt to perform a kind of in-distribution pattern-matching, aligning given questions and solution steps with similar ones seen in the training data. As no formal reasoning is involved in this process, it could lead to high variance across different instances of the same question. In this sub-section and the next one, we investigate these observations further and we show that several factors contribute to the performance variation of the models.

First, we investigate the impact of the *type* of change to understand the difference between changing names (e.g., person names, places, foods, currencies, etc.) versus changing numbers (i.e., the values of variables).

Figure: Figure 3: The performance of all state-of-the-art models on GSM-Symbolic drops compared to GSM8K. Later, we investigate the factors that impact the performance drops in more depth.
Refer to caption: x7.png

Figure: Figure 4: How sensitive are LLMs when we change only names, only proper numbers, or both names and numbers? Overall, models have noticeable performance variation even if we only change names, but even more when we change numbers or combine these changes.
Refer to caption: x8.png

Figure 4 demonstrates that while performance variation persists, the variance is lower when changing names compared to numbers. Notably, the original GSM8K accuracy of models is now much closer to the center of the changed proper names distribution, in contrast to changed numbers or both. Furthermore, a gradual shift in the means of distributions from right to left, along with an increase in variance, is evident across almost all models. It is both striking and concerning that such performance variance exists when only changing proper names, as this level of variability would not be expected from a grade-school student with genuine mathematical understanding.

From the results in this section, we observe that by increasing the *difficulty* of changes (from names to numbers), the performance drops and the variance increases, overall suggesting that the reasoning capabilities of state-of-the-art LLMs are fragile for the aforementioned reasons. Assuming that LLMs are not performing formal reasoning, how important is the question *difficulty* on the distribution of performance? In the next section, we study this question further.

### 4.3 How Does Question Difficulty Affect Performance Distribution?

The results in the previous subsection motivate us to study the impact of question *difficulty* on the mean and variance of the performance distribution. To this end, we generate several new templates from the GSM-Symb, as illustrated in Fig. 5. First, by removing one clause, we obtain GSM-Symbolic-Minus-1 or GSM-M1 for short. Similarly, we can add one or two clauses to the questions to increase the difficulty, resulting in GSM-Symbolic-Plus-1 (GSM-P1) and GSM-Symbolic-Plus-2 (GSM-P2), respectively(^1^11It is important to recognize that adding or removing a clause does not always result in an exact increase or decrease of one in the number of required reasoning steps. In general, the exact number of steps needed to solve a problem is not fixed, as there may be multiple valid solutions for each problem, each requiring a different number of steps. Regardless, our main focus in this section is to understand the *evolution* of the performance distribution rather than the precise performance metrics.).

Figure: Figure 5: Modifying the difficulty level of GSM-Symbolic by modifying the number of clauses.

Figure: Figure 6: The impact of increasing the number of clauses on performance: As the difficulty increases from GSM-M1$\rightarrow$ GSM-Symb$\rightarrow$ GSM-P1$\rightarrow$ GSM-P2, the distribution of performance shifts to the left (i.e., accuracy decreases), and the variance increases.
Refer to caption: x14.png

As shown in Fig. 6, the trend of the evolution of the performance distribution is very consistent across all models: as the difficulty increases, the performance decreases and the variance increases. Note that overall, the *rate of accuracy drop* also increases as the difficulty increases. This is in line with the hypothesis that models are not performing formal reasoning, as the number of required reasoning steps increases linearly, but the rate of drop seems to be faster. Moreover, considering the pattern-matching hypothesis, the increase in variance suggests that searching and pattern-matching become significantly harder for models as the difficulty increases.

### 4.4 Can LLMs Really Understand Mathematical Concepts?

In the previous sections, we studied the impact of *type of change* and *difficulty* on the performance distribution. In this section, we demonstrate that models are susceptible to catastrophic performance drops on instances not part of the training distribution, potentially due to their reliance on in-distribution pattern-matching.

Figure: Figure 7: An example from the GSM-NoOp dataset: We add seemingly relevant statements to the questions that are, in fact, irrelevant to the reasoning and conclusion. However, the majority of models fail to ignore these statements and blindly convert them into operations, leading to mistakes.

We introduce GSM-NoOp, a dataset designed to challenge the reasoning capabilities of language models. To create the templates, we add seemingly relevant but ultimately inconsequential statements to GSM-Symbolic templates. Since these statements carry no operational significance, we refer to them as *"No-Op"*. These additions do not affect the reasoning required to solve the problem.

Fig. 7 illustrates an example from GSM-NoOp. An interesting observation is that models tend to blindly subtract the number of smaller fruits, potentially because their training datasets included similar examples that required conversion to subtraction operations. In the Appendix, we include additional failure cases from GSM-NoOp. Overall, we find that models tend to convert statements to operations without truly understanding their meaning. For instance, a common case we observe is that models interpret statements about “discount” as “multiplication”, regardless of the context. This raises the question of whether these models have truly understood the mathematical concepts well enough. Consequently, as shown in Fig. 8(a), there is a catastrophic performance decline across all tested models, with the Phi-3-mini model experiencing over a 65% drop, and even stronger models such as o1-preview showing significant declines.

Figure: (a)
Refer to caption: x20.png

To better understand this performance drop, we conducted another experiment. While our previous evaluations on GSM-P2 used the original 8-shots of GSM8K, here we explore two new scenarios where we change the source of the 8-shots. We report the results in Figures 8(b) and 8(c).

- •
NoOp-Symb (Using GSM-Symbolic shots of the *same* question): During evaluation, we include 8 different shots of the *same* question coming from GSM-Symbolic. Hence, each shot provides the required reasoning steps. The target question from GSM-NoOp then presents yet another variation of the same question that is different only in values and the added clause that is inconsequential. This setup should simplify the task by making it clear that the extra information in the target question is irrelevant. However, as shown in Fig. 8(b), the performance remains within the standard deviation, even with 8 shots of the same question providing the reasoning chain. Interestingly, Fig. 8(c) shows that some models can perform significantly better, even though they don’t perform nearly as well on GSM8K and GSM-Symbolic. We believe this is a very notable observation.
- •
NoOp-NoOp (Using GSM-NoOp shots of *different* questions): Here, we provide 8 shots chosen randomly from different questions of GSM-NoOp in the context. These questions share the common fact that the correct answer should ignore the No-Op statement. We observe that for the Llama-3-8B model, the performance remains the same compared to the original No-Op model, while for the Phi-3 model, performance slightly decreases.

## 5 Conclusion

In this work, we have investigated the reasoning capabilities of large language models (LLMs) and the limitations of current evaluations on GSM8K. We introduced GSM-Symbolic, a novel benchmark with multiple variants designed to provide deeper insights into the mathematical reasoning abilities of LLMs. Our extensive study reveals significant performance variability across different instantiations of the same question, challenging the reliability of current GSM8K results that rely on single-point accuracy metrics. We found that while LLMs exhibit some robustness to changes in proper names, they are more sensitive to variations in numerical values. We have also observed the performance of LLMs deteriorating as question complexity increases.

The introduction of GSM-NoOp exposes a critical flaw in LLMs’ ability to genuinely understand mathematical concepts and discern relevant information for problem-solving. Adding seemingly relevant but ultimately inconsequential information to the logical reasoning of the problem led to substantial performance drops of up to 65% across all state-of-the-art models. Importantly, we demonstrate that LLMs struggle even when provided with multiple examples of the same question or examples containing similar irrelevant information. This suggests deeper issues in their reasoning processes that cannot be easily mitigated through few-shot learning or fine-tuning.

Ultimately, our work underscores significant limitations in the ability of LLMs to perform genuine mathematical reasoning. The high variance in LLM performance on different versions of the same question, their substantial drop in performance with a minor increase in difficulty, and their sensitivity to inconsequential information indicate that their reasoning is fragile. It may resemble sophisticated pattern matching more than true logical reasoning. We remind the reader that both GSM8K and GSM-Symbolic include relatively simple grade-school math questions, requiring only basic arithmetic operations at each step. Hence, the current limitations of these models are likely to be more pronounced in more challenging mathematical benchmarks.

We believe further research is essential to develop AI models capable of formal reasoning, moving beyond pattern recognition to achieve more robust and generalizable problem-solving skills. This remains a critical challenge for the field as we strive to create systems with human-like cognitive abilities or general intelligence.

## Acknowledgments

The authors would like to thank Max Horton, Fartash Faghri, Moin Nabi, and Devi Krishna for the valuable feedback and support.

## Appendix A Appendix

In this appendix, we provide additional details to the main text, including:

- •
A.1: Detailed experimental setups, including the prompt template.
- •
A.2: Full results on GSM8K, GSM-Symbolic, and their variants.
- •
A.3: Additional results for the distributional performance of several models, similar to the results from Sec. 4.1 in the main text.
- •
A.4: Additional results for Sec. 4.3, where we studied the impact of question difficulty. We show that fine-tuning on easier tasks does not necessarily improve performance on more difficult tasks.
- •
A.5: A more comprehensive discussion and analysis of performance for OpenAI o1-mini and o1-preview models.

### A.1 Detailed Experimental Setup

In this work, all reported evaluations results use 8-shots with chain-of-thought prompting. We use the following prompt format:

Figure: Figure 9: The prompt format used for evaluations.

Except for the last experiment in Sec. 4.4, we use the original 8 shots from GSM8K. In addition, we allow the models to generate until either their context size limit is reached, they generate one of the end-of-response tokens such as ‘
’ or ‘
’, or they finish answering the current question and move on to generating the next question, indicated by another ‘Q:’ generation.

Finally, we note that in all experiments we use greedy decoding to generate responses from models, with one exception: currently, the available APIs for "o1-mini" and "o1-preview" models do not allow controlling the decoding strategy, and it seems that at the time of writing, these models do not perform greedy decoding, as responses to the same prompt change.

### A.2 Full Results

In Tab. 1, we present the comprehensive performance results of various models, including Gemma (Mesnard et al., ), Gemma2 (Rivière et al., ), Phi (Abdin et al., ), Mistral (Jiang et al., ), Llama3 (Dubey et al., ), GPT-4o (OpenAI, ), and the o1 (OpenAI, ) series, on GSM8K and its different variants, GSM-Symbolic.

We report two sets of results for GSM8K: the first column indicates the accuracy on the *full* test set of GSM8K (comprising 1,319 examples), while the second column shows the accuracy on a subset of 100 questions from the GSM8K test set, which we randomly selected to generate GSM-Symbolic templates. It is noteworthy that the performance levels across both sets are very similar, with no significant differences observed.

**Table 1: Full 8-shot results of all models on GSM8Kand different variants of GSM-Symbolic.**
| Model | GSM8K<br>(Full) | GSM8K<br>(100) | Symbolic-M1 | Symbolic | Symbolic-P1 | Symbolic-P2 | Symbolic-NoOp |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Gemma2b | 12.1 | 11.0 | 24.5 ($\pm$ 3.85) | 8.2 ($\pm$ 2.21) | 3.6 ($\pm$ 2.13) | 1.5 ($\pm$ 1.63) | 4.7 ($\pm$ 1.99) |
| Gemma2b-it | 12.1 | 11.0 | 16.2 ($\pm$ 3.28) | 8.2 ($\pm$ 2.21) | 1.5 ($\pm$ 1.49) | 1.5 ($\pm$ 1.63) | 4.1 ($\pm$ 2.48) |
| Gemma-7b | 53.8 | 50.0 | 34.1 ($\pm$ 4.41) | 25.6 ($\pm$ 3.25) | 26.0 ($\pm$ 5.30) | 3.1 ($\pm$ 1.92) | 8.7 ($\pm$ 2.71) |
| Gemma-7b-it | 29.3 | 33.0 | 34.1 ($\pm$ 4.41) | 25.6 ($\pm$ 3.25) | 6.0 ($\pm$ 3.38) | 3.1 ($\pm$ 1.92) | 8.7 ($\pm$ 2.71) |
| Gemma2-2b | 47.5 | 46.0 | 57.2 ($\pm$ 3.40) | 40.1 ($\pm$ 3.04) | 19.5 ($\pm$ 3.89) | 1.3 ($\pm$ 1.37) | 8.8 ($\pm$ 4.12) |
| Gemma2-2b-it | 47.5 | 46.0 | 57.2 ($\pm$ 3.40) | 40.1 ($\pm$ 3.04) | 19.5 ($\pm$ 3.89) | 4.5 ($\pm$ 1.94) | 15.7 ($\pm$ 3.97) |
| Gemma2-9b | 85.3 | 87.0 | 71.2 ($\pm$ 2.81) | 79.1 ($\pm$ 2.99) | 44.0 ($\pm$ 5.69) | 41.8 ($\pm$ 6.00) | 22.3 ($\pm$ 5.11) |
| Gemma2-9b-it | 85.3 | 87.0 | 84.4 ($\pm$ 2.36) | 79.1 ($\pm$ 2.99) | 68.1 ($\pm$ 4.77) | 41.8 ($\pm$ 6.00) | 22.3 ($\pm$ 5.11) |
| Gemma2-27b-it | 89.7 | 92.0 | 90.2 ($\pm$ 1.86) | 88.3 ($\pm$ 2.56) | 80.7 ($\pm$ 4.07) | 63.4 ($\pm$ 4.14) | 30.0 ($\pm$ 3.39) |
| Phi-2 | 56.0 | 53.0 | 53.0 ($\pm$ 3.10) | 41.4 ($\pm$ 3.56) | 23.3 ($\pm$ 4.07) | 8.9 ($\pm$ 3.33) | 11.2 ($\pm$ 3.51) |
| Phi-3-mini-128k-instruct | 83.7 | 85.0 | 85.9 ($\pm$ 2.44) | 80.7 ($\pm$ 2.94) | 63.4 ($\pm$ 5.63) | 37.5 ($\pm$ 5.76) | 18.0 ($\pm$ 3.83) |
| Phi-3-small-128k-instruct | 88.5 | 89.0 | 86.4 ($\pm$ 1.95) | 83.7 ($\pm$ 2.65) | 72.0 ($\pm$ 3.65) | 50.7 ($\pm$ 4.99) | 24.5 ($\pm$ 4.81) |
| Phi-3-medium-128k-instruct | 87.3 | 89.0 | 89.6 ($\pm$ 1.65) | 82.5 ($\pm$ 2.86) | 75.8 ($\pm$ 3.89) | 53.1 ($\pm$ 4.80) | 29.4 ($\pm$ 4.18) |
| Phi-3.5-mini-instruct | 84.9 | 88.0 | 87.6 ($\pm$ 1.98) | 82.1 ($\pm$ 3.38) | 64.8 ($\pm$ 5.43) | 44.8 ($\pm$ 6.32) | 22.4 ($\pm$ 4.03) |
| Mistral-7b-v0.1 | 44.5 | 48.0 | 55.4 ($\pm$ 3.18) | 41.1 ($\pm$ 3.36) | 17.4 ($\pm$ 4.82) | 5.5 ($\pm$ 2.55) | 16.2 ($\pm$ 4.43) |
| Mistral-7b-instruct-v0.1 | 39.7 | 42.0 | 44.9 ($\pm$ 4.29) | 30.5 ($\pm$ 3.47) | 13.1 ($\pm$ 3.51) | 4.0 ($\pm$ 2.24) | 10.1 ($\pm$ 3.42) |
| Mistral-7b-v0.3 | 40.6 | 44.0 | 54.0 ($\pm$ 2.95) | 40.0 ($\pm$ 4.43) | 15.6 ($\pm$ 4.02) | 3.9 ($\pm$ 2.31) | 16.7 ($\pm$ 4.26) |
| Mistral-7b-instruct-v0.3 | 56.2 | 56.0 | 62.3 ($\pm$ 2.68) | 50.0 ($\pm$ 3.49) | 24.5 ($\pm$ 4.34) | 10.8 ($\pm$ 3.60) | 15.9 ($\pm$ 4.44) |
| Mathstral-7b-v0.1 | 80.1 | 80.0 | 82.9 ($\pm$ 2.87) | 74.0 ($\pm$ 3.49) | 57.4 ($\pm$ 5.20) | 35.5 ($\pm$ 5.07) | 20.4 ($\pm$ 3.58) |
| Llama3-8b | 55.8 | 61.0 | 79.5 ($\pm$ 3.62) | 74.6 ($\pm$ 2.94) | 53.8 ($\pm$ 4.54) | 12.3 ($\pm$ 3.43) | 18.6 ($\pm$ 3.86) |
| Llama3-8b-instruct | 76.0 | 74.0 | 79.5 ($\pm$ 3.62) | 74.6 ($\pm$ 2.94) | 53.8 ($\pm$ 4.54) | 28.3 ($\pm$ 4.37) | 18.6 ($\pm$ 3.86) |
| GPT-4o-mini | 94.2 | 95.0 | 92.5 ($\pm$ 1.63) | 91.7 ($\pm$ 2.02) | 81.1 ($\pm$ 3.05) | 72.4 ($\pm$ 4.57) | 54.1 ($\pm$ 3.85) |
| GPT-4o | 95.2 | 95.0 | 94.4 ($\pm$ 1.62) | 94.9 ($\pm$ 1.87) | 93.9 ($\pm$ 2.59) | 88.0 ($\pm$ 3.43) | 63.1 ($\pm$ 4.53) |
| o1-mini | 95.1 | 93.0 | 94.9 ($\pm$ 1.49) | 94.5 ($\pm$ 1.58) | 94.3 ($\pm$ 2.57) | 89.1 ($\pm$ 3.56) | 66.0 ($\pm$ 4.60) |
| o1-preview | 94.9 | 96.0 | 93.6 ($\pm$ 1.68) | 92.7 ($\pm$ 1.82) | 95.4 ($\pm$ 1.72) | 94.0 ($\pm$ 2.38) | 77.4 ($\pm$ 3.84) |

### A.3 Additional Results on GSM-Symbolic Performance Distributions

In section 4.1, we have presented results for several models in Fig. 2. Here, we provide additional results showing the performance on GSM-Symbolic  for other models also have high variance. Moreover, these models correspond to highest drop. We also report the results for all models in table 1.

Figure: Figure 10: Additional results on performance variation on GSM-Symbolic.
Refer to caption: x25.png

### A.4 Ablation: Does Fine-Tuning on Easier Tasks Help with More Difficult Tasks?

In Sec. 4.3, we observed that the performance on GSM-P2 is significantly lower than the performance on GSM-P1. We also argued that it is unlikely that additional fine-tuning or including shots from GSM-P1 would be beneficial. Here, in Fig. 11(a), we show that including shots from GSM-P1 does not improve performance compared to the results where shots come solely from GSM8K.

Moreover, in Fig. 11(b), we demonstrate that fine-tuning Phi-3.5 on GSM-P1 slightly improves performance on GSM-P1 while decreasing performance on GSM-P2. We have used a set of 50 templates from GSM-P1, separate from the test templates, and generated 10000 examples for finetuning training set.

Overall, while this direction warrants further research, current results suggest that scaling training data will not be helpful in improving the reasoning capabilities of language models.

Figure: (a)
Refer to caption: x28.png

### A.5 Results on o1-preview and o1-mini

The recently released o1-preview and o1-mini models (OpenAI, ) have demonstrated strong performance on various reasoning and knowledge-based benchmarks. As observed in Tab. 1, the mean of their performance distribution is significantly higher than that of other open models.

In Fig. 12 (top), we illustrate that both models exhibit non-negligible performance variation. When the difficulty level is altered, o1-mini follows a similar pattern to other open models: as the difficulty increases, performance decreases and variance increases.

Figure: Figure 12: Results on o1-mini and o1-preview: both models mostly follow the same trend we presented in the main text. However, o1-preview shows very strong results on all levels of difficulty as all distributions are close to each other.
Refer to caption: x30.png

The o1-preview model demonstrates robust performance across all levels of difficulty, as indicated by the closeness of all distributions. However, it is important to note that both o1-preview and o1-mini experience a significant performance drop on GSM-NoOp . In Fig. 13, we illustrate that o1-preview struggles with understanding mathematical concepts, naively applying the 10% inflation discussed in the question, despite it being irrelevant since the prices pertain to this year. Additionally, in Fig. 14, we present another example highlighting this issue.

Overall, while o1-preview and o1-mini exhibit significantly stronger results compared to current open models—potentially due to improved training data and post-training procedures—they still share similar limitations with the open models.

Figure: Figure 13: Sample response from o1-preview on an example from GSM-NoOp: the model blindly applies the inflation rate, even though the inflation amount is irrelevant as the question clearly indicates the given prices are for “now” and not last year.

Figure: Figure 14: Sample response from o1-preview and o1-mini on an example from GSM-NoOp: while the donation amount is irrelevant to the price difference, the models subtract the amount we donate.

## Peer Reviews

confidence: 4
contribution: 4
initial_rating: 8
presentation: 2
questions: 1. The abstract claims that “Specifically, the performance of all models declines when only the numerical values in the question are altered in the GSM-Symbolic benchmark.” However, if I’m reading this right, this is not what you report in Table 1 in A.2? For example, o1-mini and both Llama3 8B models seem to get a higher score on Symbolic compared to GSM8k (100).

2. One potential concern I have with the paper is that. Especially given (1), how do you differentiate between results that apply to LLMs more broadly compared  claims that apply broadly to specific LLMs? In particular, several models in your evaluations are well known to have interesting results regarding overfitting on GSM8k as per works in your current related works section.
soundness: 3
strengths: Paper tests a very good question. The state of LLMs is very strange right now. They can clearly solve very tricky problems. At the same time, they often fail in very elementary ways as well. This paper is an exceptional analysis of this, especially testing across several different variants of GSM-Symbolic. In general, I think this paper is a fantastic contribution to the field and as a result vote accept.
summary: This paper creates a novel dataset (GSM-Symbolic) with several different variants to test whether LLMs are capable of generalized reasoning. GSM-Symbolic has several variants which change the numbers involved in the problem and/or removes or adds clauses that change the difficulty level of the problem but not in a way that is too different from the original problem. They benchmark across a number of models and show that many LLMs show significant drops when changing the numbers only slightly or adding irrelevant information.

I have some questions that I raise in the sections below. In particular, I think some of the claims are not fully supported by the provided evidence and that it's not clear that you can make strong, broad claims about LLMs as opposed to claims about the particular models tested. Nevertheless, I believe this paper to be both high quality and important work and think that it should be accepted to the conference.

Nits:

l84, citation format error
L759: spacing error
weaknesses: Some weaknesses are referenced in the questions section. One additional question: would the authors be able to include statistical significance results in the Appendix or main results? I think this would significantly improve the paper.

Secondly, while the GSM-Noop experiments are very interesting, I think there is a large difference in the claim (also made by prior work) that LLMs are bad at handling irrelenvant context and them not performing reasoning.

Related Work: I think Srivastava+ 2024 should be cited and compared to, as it is a very similar method on the MATH dataset with similar-ish findings.

@misc{srivastava2024functionalbenchmarksrobustevaluation,
      title={Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gap}, 
      author={Saurabh Srivastava and Annarose M B and Anto P V au2 and Shashank Menon and Ajay Sukumar and Adwaith Samod T and Alan Philipose and Stevin Prince and Sooraj Thomas},
      year={2024},
      eprint={2402.19450},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2402.19450}, 
}

@misc{shi2023largelanguagemodelseasily,
      title={Large Language Models Can Be Easily Distracted by Irrelevant Context}, 
      author={Freda Shi and Xinyun Chen and Kanishka Misra and Nathan Scales and David Dohan and Ed Chi and Nathanael Schärli and Denny Zhou},
      year={2023},
      eprint={2302.00093},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2302.00093}, 
}

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: **Q1: Computational Complexity Impact**
Could the authors provide a detailed analysis of how computational complexity affects the accuracy of the models tested? Specifically, it would be informative to understand the proportion of errors that can be attributed directly to the failure of numerical computations versus those caused by reasoning errors. This differentiation could help clarify the extent to which computational demands influence the performance of large language models on the GSM-Symbolic dataset.

**Q2: Impact of Fine-Tuning on Model Robustness**
Would the authors consider conducting experiments where some models are fine-tuned on the GSM-Symbolic dataset and its variants to assess whether this training approach enhances their robustness to the types of perturbations introduced in your study? This could provide valuable insights into whether the observed limitations are inherent to the model architectures or can be ameliorated through targeted training, potentially challenging the paper's conclusions about the fundamental nature of these limitations.
soundness: 3
strengths: **Strengths:**

1. **Comprehensive Experimental Design**: The authors explore various perturbations in mathematical questions such as changing names or numbers, adjusting the number of clauses, and introducing irrelevant information. These detailed experiments provide robust evidence of the fragility exhibited by prominent language models under slight modifications, thereby contributing significantly to our understanding of their limitations.

2. **Development of the GSM-Symbolic Dataset**: The introduction of the GSM-Symbolic dataset and its variants is a notable contribution. By coding 100 problems from the GSM8K dataset and creating corresponding modifications, the authors have developed a valuable resource that can facilitate future research into model robustness.

3. **Relevance and Depth of Analysis**: The topic is highly relevant to current discussions in the field, and the paper delivers extensive empirical data concerning critical questions about the robustness of current language models when subject to various perturbations.
summary: The paper "GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models" presents a thorough investigation into the mathematical reasoning capabilities of large language models (LLMs) using a newly developed benchmark, GSM-Symbolic. This benchmark, derived from the GSM8K dataset, includes modifications to test the robustness of LLMs against changes in numerical values, proper names, and question complexity.

**Key Contributions:**

1. **Introduction of GSM-Symbolic**: The paper introduces GSM-Symbolic, a benchmark that generates diverse and controllable mathematical question variants based on symbolic templates. This allows for more precise evaluations of LLMs by modifying elements like numbers and text within the questions to assess their impact on model performance.

2. **Empirical Analysis of LLM Performance**: Through extensive testing with GSM-Symbolic, the authors demonstrate that LLMs exhibit significant performance variability and sensitivity to minor changes in question structure. Their experiments reveal that even small perturbations, such as altering numerical values or adding irrelevant clauses, can significantly degrade the performance of state-of-the-art models.

3. **Exploration of Reasoning Fragility in LLMs**: The study details how the introduction of irrelevant information (via the GSM-NoOp dataset) into mathematical problems significantly impacts model accuracy, highlighting a critical flaw in the current models' ability to discern relevant from irrelevant data.

Overall, the paper provides a deeper understanding of the limitations inherent in current LLMs regarding their mathematical reasoning capabilities and offers a comprehensive framework for future research to explore and mitigate these weaknesses.
weaknesses: **Weaknesses:**

1. **Oversight of Computational Complexity**: The paper does not sufficiently address the computational challenges posed by the range of parameter values (5 to 100) used in their experiments. This range suggests that computing expressions like x + y + z, particularly with all parameters as two-digit numbers, could introduce significant computational errors unrelated to the models' reasoning capabilities. The potential impact of these computational difficulties on the results, particularly evident in Figure 4, suggests that the observed distribution shifts in Figure 2 might be attributed more to computational challenges than to data contamination.

2. **Ambiguity in Defining Reasoning**: The paper's definition of reasoning needs clarification. Asserting that models capable of only three-step inferences lack reasoning capabilities seems overly restrictive. Since the GSM8K focuses on the NLP aspect of reasoning rather than the length of reasoning required, it naturally tailors models to excel within these confines. Thus, introducing additional clauses or irrelevant details, as done in the experiments, effectively renders the questions out-of-distribution (OOD). This shift likely degrades performance due to the models not being trained to handle such modifications, which doesn't necessarily indicate a lack of reasoning ability but rather a limitation in the training paradigm.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. For numerical variables, how did the authors choose their boundary? Why x belongs to (5, 100) in Figure 1? I think this range may also influence the performance, especially when some extreme values are selected.
2. To add clause in GSM-NoOp, we can see results drop a lot in Figure 8, but what is the standard to add this extra clause? Because adding clause can be subjective if authors already know some patterns will make the model fail.
soundness: 3
strengths: + GSM-Symbolic provides a strategy to enrich the math reasoning datasets. The symbolic template can be applied for other datasets.
+ The NoOp design is innovative, it shows undiscussed disadvantage of current LLM on reasoning tasks.
+ The experiment design is adequate and valid. This paper evaluates 25 models, the results provide a comprehensive overview of the current SOTA methods.
summary: This paper investigates the reasoning capabilities of LLMs using the GSM8K benchmark and introduces a new benchmark called GSM-Symbolic. The authors highlight the variability in model performance, particularly noting that LLMs struggle more when numerical values are altered compared to when only proper names are changed. They also explore the impact of question difficulty on performance, revealing that as complexity increases, both average performance and variance decline. The findings emphasize the need for more reliable evaluation methodologies and further research into the reasoning abilities of LLMs, suggesting that while these models show potential, their understanding of mathematical concepts remains limited
weaknesses: - Figure 3 shows the chatgpt-o1 and GPT-4o has tiny accuracy drop. But even without GSM-Symbolic, running any LLM models for 50 times on the same dataset can cause this variance.
- For numerical variables, how did the authors choose their boundary? This is not explained in the paper
- To add clause in GSM-NoOp, we can see results drop a lot in Figure 8, but what is the standard to add this extra clause? Because adding clause can be subjective if authors already know some patterns will make the model fail.
- Part of the beginning paragraph of section 3 should belong to related work discussion.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. Can you clearly define "true logical reasoning"? What conditions need to be satisfied for a language model to demonstrate true logical reasoning?
2. Why was the performance of o1 not investigated in certain parts of the study, such as in Figure 3?
3. Given that some models like GPT4-o show robustness across numerical changes, do the authors believe that scaled training and model size can alleviate the identified limitations? If so, how does this impact the generalizability of the findings?
soundness: 2
strengths: - The study provides empirical evidence of the performance variance and fragility of LLMs when faced with different question instantiations and increased complexity in grade school math questions.
- The introduction of the GSM-Symbolic benchmark allows for the creation of diverse instances of the GSM8K dataset, which helps avoid data contamination and enables a more comprehensive evaluation of LLMs' reasoning capabilities.
summary: The paper "GSM-SYMBOLIC: Understanding the Limitations of Mathematical Reasoning in Large Language Models" explores the mathematical reasoning capabilities of Large Language Models (LLMs) through the introduction of a new benchmark, GSM-Symbolic. This benchmark aims to address the limitations of existing evaluations like GSM8K by generating diverse question variants and adjusting complexity levels. The study finds that LLMs exhibit performance variance when faced with different instantiations of the same question, particularly when numerical values are altered. It also highlights the fragility of LLMs' reasoning capabilities, as performance deteriorates with increased question complexity and the introduction of irrelevant information. The paper suggests that current LLMs rely on probabilistic pattern-matching rather than robust logical reasoning.
weaknesses: - The findings largely reconfirm known limitations of LLMs, such as their reliance on probabilistic pattern-matching and sensitivity to small changes in question statements, which have been explored in depth in previous works, notably in "Are NLP Models really able to Solve Simple Math Word Problems?"
- The methods used to curate the dataset, such as altering numerical values and adding irrelevant information, are not novel and have been employed in other studies, such as "PAL: Program-aided Language Models," which introduces GSM8k-hard.
- The paper uses ambiguous terms like "true logical reasoning," "genuine mathematical reasoning," and "formal [reasoning] in the common sense term" without providing clear definitions of these terms.
- The paper’s findings on performance degradation with increasing question complexity have been similarly explored in other works as compounding errors in multi-step reasoning processes.
- The authors suggest that  performance of LLMs on GSM-Symbolic compared to their performance on GSM8k indicates significant limitations in LLMs’ ability to perform genuine mathematical reasoning. However,  performance of GPT4-o remains robust on GSM-Symbolic, as shown in figure 3, suggesting that the performance variance resulting from different numerical instantiation of the questions may be resolved with scaled training or model size and this limitations may not be inherent to all LLMs.

Meta-review: The paper introduces GSM-Symbolic, a novel benchmark designed to evaluate the mathematical reasoning capabilities of large language models (LLMs) with a focus on controlled and nuanced assessments. Building on the widely used GSM8K benchmark, GSM-Symbolic employs symbolic templates to generate diverse question variants, allowing for more reliable and comprehensive evaluations of LLMs' reasoning abilities. The authors conduct extensive experiments across multiple state-of-the-art models, revealing significant performance variance when numerical values and question complexity are altered. Notably, the study shows that adding irrelevant clauses can lead to substantial performance drops, suggesting that current LLMs struggle with genuine logical reasoning and often rely on pattern-matching from training data. The paper also discusses the fragility of LLMs' mathematical reasoning, particularly as question complexity increases, and highlights potential areas for future research.

#### Contribution
1. **GSM-Symbolic Benchmark**: The introduction of GSM-Symbolic, with its symbolic templates and various question variants (e.g., GSM-names, GSM-vars, GSM-NoOp), provides a valuable tool for assessing LLMs' robustness and reasoning capabilities in mathematical contexts.
2. **Empirical Insights**: The paper offers empirical evidence of LLMs' limitations, demonstrating performance variance and fragility when faced with slight modifications or increased complexity in mathematical problems.
3. **Discussion on Reasoning**: The study contributes to the ongoing discourse about what constitutes "true logical reasoning" in LLMs, proposing a heuristic based on performance consistency across different instantiations of problems.
4. **Broad Evaluation**: The authors evaluate a large number of models, providing a comprehensive overview of current state-of-the-art methods and their limitations.

#### Weaknesses
1. **Statistical Rigor**: The paper lacks rigorous statistical evaluations, particularly in assessing the significance of performance variations across different models and question variants.
2. **Novelty Concerns**: Some reviewers argue that the methods used (e.g., altering numerical values, adding irrelevant information) are not novel and have been explored in prior works, raising questions about the paper's originality.
3. **Ambiguity in Reasoning Definition**: The paper uses terms like "true logical reasoning" and "genuine mathematical reasoning" without providing clear definitions, leading to potential confusion about the study's claims.
4. **Limited Exploration of Computational Complexity**: The impact of computational complexity on model performance is not thoroughly addressed, potentially confounding the interpretation of reasoning limitations.
5. **Potential Overlap with Existing Datasets**: Concerns were raised about the similarity between GSM-Symbolic and the TemplateGSM dataset, although the authors argue that their work was developed independently.

Decision: Accept (Poster)