GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models

Authors: Seyed Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, Mehrdad Farajtabar
Primary area: foundation or frontier models, including LLMs
Keywords: Reasoning, Large Language Models, Mathematical Reasoning, Datasets

Abstract:
Recent advancements in Large Language Models (LLMs) have sparked interest in their mathematical reasoning capabilities. While performance on the widely popular GSM8K benchmark has improved, questions remain about whether reported evaluation metrics are reliable, and reasoning abilities of LLMs have advanced. To overcome the limitations of existing evaluations, we introduce GSM-Symbolic, an improved benchmark created from symbolic templates that allow for the generation of a diverse set of questions. GSM-Symbolic enables more controllable evaluations, providing key insights and more reliable metrics for measuring the reasoning capabilities of models. 
Our findings reveal that LLMs exhibit noticeable variance when responding to different instantiations of the same question. Specifically, the performance of models declines when only the numerical values in the question are altered in the GSM-Symbolic benchmark. Furthermore, we investigate the fragility of mathematical reasoning in these models and demonstrate that their performance significantly deteriorates as the number of clauses in a question increases. We hypothesize that this decline is due to the fact that current LLMs are not capable of genuine logical reasoning; instead, they attempt to replicate the reasoning steps observed in their training data. When we add a single clause that appears relevant to the question, we observe significant performance drops (up to 65%) across all state-of-the-art models, even though the added clause does not contribute to the reasoning chain needed to reach the final answer. Overall, our work provides a more nuanced understanding of LLMs' capabilities and limitations in mathematical reasoning.

Decision: accept (poster)
Reviewer scores: 8, 8, 5, 3

Reviews and discussion:
<thread>
creation date:
1730353712359

thread_type:
official_review

<thread_object>
creator: reviewer_fk55

<content>
summary:
The paper "GSM-SYMBOLIC: Understanding the Limitations of Mathematical Reasoning in Large Language Models" explores the mathematical reasoning capabilities of Large Language Models (LLMs) through the introduction of a new benchmark, GSM-Symbolic. This benchmark aims to address the limitations of existing evaluations like GSM8K by generating diverse question variants and adjusting complexity levels. The study finds that LLMs exhibit performance variance when faced with different instantiations of the same question, particularly when numerical values are altered. It also highlights the fragility of LLMs' reasoning capabilities, as performance deteriorates with increased question complexity and the introduction of irrelevant information. The paper suggests that current LLMs rely on probabilistic pattern-matching rather than robust logical reasoning.

strengths:
- The study provides empirical evidence of the performance variance and fragility of LLMs when faced with different question instantiations and increased complexity in grade school math questions.
- The introduction of the GSM-Symbolic benchmark allows for the creation of diverse instances of the GSM8K dataset, which helps avoid data contamination and enables a more comprehensive evaluation of LLMs' reasoning capabilities.

weaknesses:
- The findings largely reconfirm known limitations of LLMs, such as their reliance on probabilistic pattern-matching and sensitivity to small changes in question statements, which have been explored in depth in previous works, notably in "Are NLP Models really able to Solve Simple Math Word Problems?"
- The methods used to curate the dataset, such as altering numerical values and adding irrelevant information, are not novel and have been employed in other studies, such as "PAL: Program-aided Language Models," which introduces GSM8k-hard.
- The paper uses ambiguous terms like "true logical reasoning," "genuine mathematical reasoning," and "formal [reasoning] in the common sense term" without providing clear definitions of these terms.
- The paper’s findings on performance degradation with increasing question complexity have been similarly explored in other works as compounding errors in multi-step reasoning processes.
- The authors suggest that  performance of LLMs on GSM-Symbolic compared to their performance on GSM8k indicates significant limitations in LLMs’ ability to perform genuine mathematical reasoning. However,  performance of GPT4-o remains robust on GSM-Symbolic, as shown in figure 3, suggesting that the performance variance resulting from different numerical instantiation of the questions may be resolved with scaled training or model size and this limitations may not be inherent to all LLMs.

questions:
1. Can you clearly define "true logical reasoning"? What conditions need to be satisfied for a language model to demonstrate true logical reasoning?
2. Why was the performance of o1 not investigated in certain parts of the study, such as in Figure 3?
3. Given that some models like GPT4-o show robustness across numerical changes, do the authors believe that scaled training and model size can alleviate the identified limitations? If so, how does this impact the generalizability of the findings?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730486438533

thread_type:
official_review

<thread_object>
creator: reviewer_wg4b

<content>
summary:
This paper investigates the reasoning capabilities of LLMs using the GSM8K benchmark and introduces a new benchmark called GSM-Symbolic. The authors highlight the variability in model performance, particularly noting that LLMs struggle more when numerical values are altered compared to when only proper names are changed. They also explore the impact of question difficulty on performance, revealing that as complexity increases, both average performance and variance decline. The findings emphasize the need for more reliable evaluation methodologies and further research into the reasoning abilities of LLMs, suggesting that while these models show potential, their understanding of mathematical concepts remains limited

strengths:
+ GSM-Symbolic provides a strategy to enrich the math reasoning datasets. The symbolic template can be applied for other datasets.
+ The NoOp design is innovative, it shows undiscussed disadvantage of current LLM on reasoning tasks.
+ The experiment design is adequate and valid. This paper evaluates 25 models, the results provide a comprehensive overview of the current SOTA methods.

weaknesses:
- Figure 3 shows the chatgpt-o1 and GPT-4o has tiny accuracy drop. But even without GSM-Symbolic, running any LLM models for 50 times on the same dataset can cause this variance.
- For numerical variables, how did the authors choose their boundary? This is not explained in the paper
- To add clause in GSM-NoOp, we can see results drop a lot in Figure 8, but what is the standard to add this extra clause? Because adding clause can be subjective if authors already know some patterns will make the model fail.
- Part of the beginning paragraph of section 3 should belong to related work discussion.

questions:
1. For numerical variables, how did the authors choose their boundary? Why x belongs to (5, 100) in Figure 1? I think this range may also influence the performance, especially when some extreme values are selected.
2. To add clause in GSM-NoOp, we can see results drop a lot in Figure 8, but what is the standard to add this extra clause? Because adding clause can be subjective if authors already know some patterns will make the model fail.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730702294041

thread_type:
official_review

<thread_object>
creator: reviewer_7pzc

<content>
summary:
The paper "GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models" presents a thorough investigation into the mathematical reasoning capabilities of large language models (LLMs) using a newly developed benchmark, GSM-Symbolic. This benchmark, derived from the GSM8K dataset, includes modifications to test the robustness of LLMs against changes in numerical values, proper names, and question complexity.

**Key Contributions:**

1. **Introduction of GSM-Symbolic**: The paper introduces GSM-Symbolic, a benchmark that generates diverse and controllable mathematical question variants based on symbolic templates. This allows for more precise evaluations of LLMs by modifying elements like numbers and text within the questions to assess their impact on model performance.

2. **Empirical Analysis of LLM Performance**: Through extensive testing with GSM-Symbolic, the authors demonstrate that LLMs exhibit significant performance variability and sensitivity to minor changes in question structure. Their experiments reveal that even small perturbations, such as altering numerical values or adding irrelevant clauses, can significantly degrade the performance of state-of-the-art models.

3. **Exploration of Reasoning Fragility in LLMs**: The study details how the introduction of irrelevant information (via the GSM-NoOp dataset) into mathematical problems significantly impacts model accuracy, highlighting a critical flaw in the current models' ability to discern relevant from irrelevant data.

Overall, the paper provides a deeper understanding of the limitations inherent in current LLMs regarding their mathematical reasoning capabilities and offers a comprehensive framework for future research to explore and mitigate these weaknesses.

strengths:
**Strengths:**

1. **Comprehensive Experimental Design**: The authors explore various perturbations in mathematical questions such as changing names or numbers, adjusting the number of clauses, and introducing irrelevant information. These detailed experiments provide robust evidence of the fragility exhibited by prominent language models under slight modifications, thereby contributing significantly to our understanding of their limitations.

2. **Development of the GSM-Symbolic Dataset**: The introduction of the GSM-Symbolic dataset and its variants is a notable contribution. By coding 100 problems from the GSM8K dataset and creating corresponding modifications, the authors have developed a valuable resource that can facilitate future research into model robustness.

3. **Relevance and Depth of Analysis**: The topic is highly relevant to current discussions in the field, and the paper delivers extensive empirical data concerning critical questions about the robustness of current language models when subject to various perturbations.

weaknesses:
**Weaknesses:**

1. **Oversight of Computational Complexity**: The paper does not sufficiently address the computational challenges posed by the range of parameter values (5 to 100) used in their experiments. This range suggests that computing expressions like x + y + z, particularly with all parameters as two-digit numbers, could introduce significant computational errors unrelated to the models' reasoning capabilities. The potential impact of these computational difficulties on the results, particularly evident in Figure 4, suggests that the observed distribution shifts in Figure 2 might be attributed more to computational challenges than to data contamination.

2. **Ambiguity in Defining Reasoning**: The paper's definition of reasoning needs clarification. Asserting that models capable of only three-step inferences lack reasoning capabilities seems overly restrictive. Since the GSM8K focuses on the NLP aspect of reasoning rather than the length of reasoning required, it naturally tailors models to excel within these confines. Thus, introducing additional clauses or irrelevant details, as done in the experiments, effectively renders the questions out-of-distribution (OOD). This shift likely degrades performance due to the models not being trained to handle such modifications, which doesn't necessarily indicate a lack of reasoning ability but rather a limitation in the training paradigm.

questions:
**Q1: Computational Complexity Impact**
Could the authors provide a detailed analysis of how computational complexity affects the accuracy of the models tested? Specifically, it would be informative to understand the proportion of errors that can be attributed directly to the failure of numerical computations versus those caused by reasoning errors. This differentiation could help clarify the extent to which computational demands influence the performance of large language models on the GSM-Symbolic dataset.

**Q2: Impact of Fine-Tuning on Model Robustness**
Would the authors consider conducting experiments where some models are fine-tuned on the GSM-Symbolic dataset and its variants to assess whether this training approach enhances their robustness to the types of perturbations introduced in your study? This could provide valuable insights into whether the observed limitations are inherent to the model architectures or can be ameliorated through targeted training, potentially challenging the paper's conclusions about the fundamental nature of these limitations.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730755421772

thread_type:
official_review

<thread_object>
creator: reviewer_yt9o

<content>
summary:
This paper creates a novel dataset (GSM-Symbolic) with several different variants to test whether LLMs are capable of generalized reasoning. GSM-Symbolic has several variants which change the numbers involved in the problem and/or removes or adds clauses that change the difficulty level of the problem but not in a way that is too different from the original problem. They benchmark across a number of models and show that many LLMs show significant drops when changing the numbers only slightly or adding irrelevant information.

I have some questions that I raise in the sections below. In particular, I think some of the claims are not fully supported by the provided evidence and that it's not clear that you can make strong, broad claims about LLMs as opposed to claims about the particular models tested. Nevertheless, I believe this paper to be both high quality and important work and think that it should be accepted to the conference.

Nits:

l84, citation format error
L759: spacing error

strengths:
Paper tests a very good question. The state of LLMs is very strange right now. They can clearly solve very tricky problems. At the same time, they often fail in very elementary ways as well. This paper is an exceptional analysis of this, especially testing across several different variants of GSM-Symbolic. In general, I think this paper is a fantastic contribution to the field and as a result vote accept.

weaknesses:
Some weaknesses are referenced in the questions section. One additional question: would the authors be able to include statistical significance results in the Appendix or main results? I think this would significantly improve the paper.

Secondly, while the GSM-Noop experiments are very interesting, I think there is a large difference in the claim (also made by prior work) that LLMs are bad at handling irrelenvant context and them not performing reasoning.

Related Work: I think Srivastava+ 2024 should be cited and compared to, as it is a very similar method on the MATH dataset with similar-ish findings.

@misc{srivastava2024functionalbenchmarksrobustevaluation,
      title={Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gap}, 
      author={Saurabh Srivastava and Annarose M B and Anto P V au2 and Shashank Menon and Ajay Sukumar and Adwaith Samod T and Alan Philipose and Stevin Prince and Sooraj Thomas},
      year={2024},
      eprint={2402.19450},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2402.19450}, 
}

@misc{shi2023largelanguagemodelseasily,
      title={Large Language Models Can Be Easily Distracted by Irrelevant Context}, 
      author={Freda Shi and Xinyun Chen and Kanishka Misra and Nathan Scales and David Dohan and Ed Chi and Nathanael Schärli and Denny Zhou},
      year={2023},
      eprint={2302.00093},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2302.00093}, 
}

questions:
1. The abstract claims that “Specifically, the performance of all models declines when only the numerical values in the question are altered in the GSM-Symbolic benchmark.” However, if I’m reading this right, this is not what you report in Table 1 in A.2? For example, o1-mini and both Llama3 8B models seem to get a higher score on Symbolic compared to GSM8k (100).

2. One potential concern I have with the paper is that. Especially given (1), how do you differentiate between results that apply to LLMs more broadly compared  claims that apply broadly to specific LLMs? In particular, several models in your evaluations are well known to have interesting results regarding overfitting on GSM8k as per works in your current related works section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731541742023

thread_type:
public_comment

<thread_object>
creator: ~yifan_zhang16

<content>
title:
Public Comment on Similarity with TemplateGSM Dataset

comment:
Thank you for this detailed and thought-provoking submission on GSM-Symbolic and its potential to improve the evaluation of mathematical reasoning in large language models. While the proposed extensions and findings are compelling, I would like to raise a concern regarding potential overlap with an existing dataset, **TemplateGSM** (https://huggingface.co/datasets/math-ai/TemplateGSM), hosted on Hugging Face and documented in their dataset repository (Published at January 2024).

**Specific Concerns:**

1. **Similarity in Dataset Generation Approach**:  
   Both GSM-Symbolic and TemplateGSM seem to employ template-based question generation with systematic variations in numerical values and structures. TemplateGSM explicitly leverages templates to create a diverse set of mathematical problems derived from GSM8K. Could you clarify how GSM-Symbolic's symbolic templates differ in methodology and intent from TemplateGSM's approach Template-based Data Generation (https://templatemath.github.io/TemplateMath_Part_I.pdf)?

2. **Overlap with GSM8K and TemplateGSM**:  
   GSM-Symbolic claims to extend GSM8K with symbolic reasoning templates. However, TemplateGSM similarly builds upon GSM8K, introducing templates with symbolic components to test reasoning. Could you elaborate on how GSM-Symbolic ensures originality and distinguishes itself from TemplateGSM?

3. **Attribution and Citation**:  
   If GSM-Symbolic indirectly or directly draws inspiration from TemplateGSM, it would be helpful for the community to understand the relationship between the two datasets. Were any of the templates or ideas in GSM-Symbolic informed by TemplateGSM's work? If so, proper acknowledgment and citation would be warranted under academic and ethical standards.

I appreciate the effort and insight behind GSM-Symbolic, but addressing these questions will help clarify its novelty and ensure proper recognition of prior work. Thank you in advance for providing transparency on these matters.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732206870628

thread_type:
public_comment

<thread_object>
creator: ~desi_r._ivanova1

<content>
title:
Statistical analysis?

comment:
Firstly, I'd like to thank the authors for their interesting work. 

A few comments and questions:

1. The work lacks statistical evaluations. Given the focus of this work to look at and evaluate distributional properties of performance, the empirical analysis completely lacks rigorous statistical evaluations. The only attempt at recognising the existence of statistical noise is when a hand-wavy "one standard deviation" criterion is mentioned in section 4.1 of the paper.

2. I wonder why the variation of performance is unexpected ("It is noteworthy that this variation even occurs").

3. What checks have been performed to ensure that GSM8K and GSM-Symbolic come from the same distribution? There is evidence of distribution mismatch between the two -- the proposed template in Figure 1 cannot generate the original question. 

Finally, directly related to the discussion here:

> Additionally, modern state-of-the-art models handle operations involving 3-4 digit numbers quite effectively because of their memorization capabilities.

This is actually not the case, at least not for all models; particularly the smaller ones (which is what is evaluated in this work). You can see a plot the average success rate of correctly adding two $d$-digit numbers for Llama3-8b-instruct and Phi3.5-mini-instruct here https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc69bcead-4865-4632-b482-080f732c2d31_5000x3500.png. Even Llama3-8B cannot add 4 digit numbers perfectly!

In a blog post that I won't link here for the purpose of preserving anonymity, I show (among other things) that for Phi3.5-mini-instruct, using the proposed ranges in Figure 1 vs smaller ranges that would better represent the original question, can give raise to a performance gap of ~4.5% for that one question (table with results can be found here https://www.datawrapper.de/_/klEuv/). This again links back to potential distribution mismatch between GSM8K and GSM-Symbolic, and how much of the (alleged) performance decline is due to that.
</content>
</thread_object>

</thread>
<thread>
creation date:
1739860199720

thread_type:
public_comment

<thread_object>
creator: ~cheryl_lee1

<content>
title:
Could you please release the Noop dataset?

comment:
As stated in the title. Since the drop on the NoOP dataset is the most significant, it may be the most important dataset, but the current GitHub repo does not contain this dataset.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces GSM-Symbolic, a novel benchmark designed to evaluate the mathematical reasoning capabilities of large language models (LLMs) with a focus on controlled and nuanced assessments. Building on the widely used GSM8K benchmark, GSM-Symbolic employs symbolic templates to generate diverse question variants, allowing for more reliable and comprehensive evaluations of LLMs' reasoning abilities. The authors conduct extensive experiments across multiple state-of-the-art models, revealing significant performance variance when numerical values and question complexity are altered. Notably, the study shows that adding irrelevant clauses can lead to substantial performance drops, suggesting that current LLMs struggle with genuine logical reasoning and often rely on pattern-matching from training data. The paper also discusses the fragility of LLMs' mathematical reasoning, particularly as question complexity increases, and highlights potential areas for future research.

#### Contribution
1. **GSM-Symbolic Benchmark**: The introduction of GSM-Symbolic, with its symbolic templates and various question variants (e.g., GSM-names, GSM-vars, GSM-NoOp), provides a valuable tool for assessing LLMs' robustness and reasoning capabilities in mathematical contexts.
2. **Empirical Insights**: The paper offers empirical evidence of LLMs' limitations, demonstrating performance variance and fragility when faced with slight modifications or increased complexity in mathematical problems.
3. **Discussion on Reasoning**: The study contributes to the ongoing discourse about what constitutes "true logical reasoning" in LLMs, proposing a heuristic based on performance consistency across different instantiations of problems.
4. **Broad Evaluation**: The authors evaluate a large number of models, providing a comprehensive overview of current state-of-the-art methods and their limitations.

#### Weaknesses
1. **Statistical Rigor**: The paper lacks rigorous statistical evaluations, particularly in assessing the significance of performance variations across different models and question variants.
2. **Novelty Concerns**: Some reviewers argue that the methods used (e.g., altering numerical values, adding irrelevant information) are not novel and have been explored in prior works, raising questions about the paper's originality.
3. **Ambiguity in Reasoning Definition**: The paper uses terms like "true logical reasoning" and "genuine mathematical reasoning" without providing clear definitions, leading to potential confusion about the study's claims.
4. **Limited Exploration of Computational Complexity**: The impact of computational complexity on model performance is not thoroughly addressed, potentially confounding the interpretation of reasoning limitations.
5. **Potential Overlap with Existing Datasets**: Concerns were raised about the similarity between GSM-Symbolic and the TemplateGSM dataset, although the authors argue that their work was developed independently.