Exploring Group and Symmetry Principles in Large Language Models

## Abstract

Abstract Large Language Models (LLMs) have demonstrated impressive performance across a wide range of applications; however, assessing their reasoning capabilities remains a significant challenge. In this paper, we introduce a framework grounded in group and symmetry principles, which have played a crucial role in fields such as physics and mathematics, and offer another way to evaluate their capabilities. While the proposed framework is general, to showcase the benefits of employing these properties, we focus on arithmetic reasoning and investigate the performance of these models on four group properties: closure, identity, inverse, and associativity. Our findings reveal that LLMs studied in this work struggle to preserve group properties across different test regimes. In the closure test, we observe biases towards specific outputs and an abrupt degradation in their performance from 100 % percent 100 100\% 100 % to 0 % percent 0 0\% 0 % after a specific sequence length. They also perform poorly in the identity test, which represents adding irrelevant information in the context, and show sensitivity when subjected to inverse test, which examines the robustness of the model with respect to negation. In addition, we demonstrate that breaking down problems into smaller steps helps LLMs in the associativity test that we have conducted. To support these tests we have developed a synthetic dataset which will be released.

## 1 Introduction

Large Language Models (LLMs) have shown remarkable capabilities across various domains, serving as the foundation for numerous applications
(Anil et al., ; Bubeck et al., ; OpenAI, ; Brown et al., ; Kojima et al., ; Chowdhery et al., ; Saparov & He, ).

Additionally, several studies have explored various aspects of the reasoning capabilities of LLMs showing significant performance improvements compared to previous domain specific models (Kojima et al., ; Wang et al., ; Mukherjee et al., ; Mitra et al., ; Shen et al., ). These studies have highlighted LLMs being able to understand and solve complex problems. Despite comprehensive evaluation on various benchmarks, LLMs still might fail in unpredictable ways. In this work, we propose a simple yet helpful framework to study LLMs behavior from a different perspective grounded in group and symmetry principles.

Group and symmetry principles which made significant contributions to numerous fields, such as physics, mathematics, and chemistry (Sarlet & Cantrijn, ; Jaffé & Orchin, ; Hargittai & Hargittai, ; Fujita, ; Gazizov & Ibragimov, ; Weyl, ; Polak & Trivers, ). These principles are foundation for understanding the fundamental structure of laws and the behavior of complex systems. For instance, in physics, group and symmetry principles have played a pivotal role in shaping modern physics by offering profound insights into the fundamental structure of physical laws and the behavior of physical systems. Noether’s theorem serves as a prime example, stating that every differentiable symmetry of the action of a physical system with conservative forces has a corresponding conservation law (Kosmann-Schwarzbach et al., ; Brading & Brown, ). As another example, temporal translation symmetry states that the laws of physics remain unchanged over time. This invariance of physical laws under time translation leads to the conservation of energy. This principle is essential to our understanding of various physical phenomena and is widely applied across numerous disciplines, including mechanics, thermodynamics, and electromagnetism (Gildener, ; Aharonov et al., ; Sasa & Yokokura, ; Lewis, ). Other group and symmetry principles in physics, including translation, rotation, and gauge symmetries, have contributed to the discovery of conservation laws and the formulation of fundamental theories like the Standard Model of particle physics (Weinberg, ; Feynman & Weinberg, ). These principles have also been instrumental in understanding the behavior of complex systems, such as condensed matter systems and cosmological models (Zee, ; Anderson, ).

Exploring the role of group and symmetry principles in LLMs can lead to:

- •
Alternative Perspective to Measure LLMs’ Performance: Investigating the role of symmetry in LLMs can provide new strategies for evaluating their performance, potentially leading to better learning methods.
The significance of symmetry principles and group properties in these domains can be exemplified by the Winoground task and dataset (Thrush et al., ), which evaluates the ability of vision and language models to conduct visio-linguistic compositional reasoning by matching images with captions containing identical sets of words in different orders. For example, given an image and two captions, ‘an old person helping a young person’ and ‘a young person helping an old person’ a model that understands the non-symmetric nature of these sentences and leverages group properties can provide the correct answer (Thrush et al., ; Lin et al., ). By connecting the role of symmetry and group properties in LLMs to the reasoning capabilities required for tasks like Winoground, we can develop a more comprehensive understanding of model performance and devise improved learning methods.
- •
Insights for Interpretability: Studying group properties and symmetry principles in LLMs can lead to more interpretable models, similar to the insights gained from saliency maps. This method is based on the idea that if a minor perturbation in the input does not alter the meaning but results in a different output, the perturbed portion of the input is crucial for the model’s prediction (Simonyan et al., ; Sundararajan et al., ; Zeiler & Fergus, ). These principles help uncover hidden relationships and structures within the data, thereby facilitating improved decision-making and trust in the model’s predictions. For example, in an identity test, introducing various symmetries, such as inverse symmetry, to perturb the input query causes the output result to change. This is similar to obtaining a saliency map by perturbing the prompt query and feeding it to the LLM.
While saliency maps and symmetry principles share similarities in their focus on understanding the importance of different input elements, their approaches differ. Saliency maps rely on perturbations to identify crucial input components, whereas symmetry principles and group properties delve into the inherent structure and relationships within the data. Group properties investigate the algebraic structures governing the interactions between data elements, such as the presence of an identity element, the existence of inverses, and the associative property. In addition, symmetry principles examine how certain aspects of the data remain unchanged or exhibit similar patterns under specific transformations, including rotations, reflections, and translations. By combining these approaches, a more comprehensive understanding of the data’s underlying organization can be achieved.
- •
Robustness, Reliability: Analyzing the influence of group properties on LLMs can aid in designing ensemble strategies that combine the predictions of multiple LLM’s outputs, to enhance overall robustness and generalization. Additionally, it can help to better examine the confidence of LLM’s output. For instance, using the closure, inverse, identity, and associativity tests, we observe that LLMs studied in this work fail for sequences longer than $15$ elements, and the confidence level decreases as we increase the length of the sequence in the simple addition task.
- •
Generalization across Domains: Incorporating symmetry and group properties into LLMs can improve their generalization capabilities across various domains such as chemistry. For example, for a language model that is trained to predict the products of a reaction $A+B\rightarrow C+D$, by understanding that the reaction $B+A$ should have a similar prediction as $A+B$ due to the symmetric nature of the reactants, the model can leverage this knowledge to improve its performance on reaction prediction tasks. The understanding of symmetry and group properties in this case allows the language model to recognize the invariance of reaction outcomes under different reactant orders, leading to more consistent predictions across a wide range of chemical reactions.

Our primary focus in this study is to assess performance of LLMs in arithmetic reasoning tasks using these principles. Our goal is NOT to show that they have good or bad performance in arithmetic operations as this has already been studied in previous studies (Kojima et al., ; Mitra et al., ; Wang et al., ), but to use them as scenarios to show the added value of using group properties for analysis. We investigate the performance of GPT-4 and GPT-3.5 on four group properties: closure, identity, inverse, and associativity. Our main findings are:

- •
Language models studied in this work exhibit significant sensitivity to the identity test. For instance, GPT-4 achieves a $100\%$ accuracy rate in the closure test for sequences involving the addition of ones, with lengths varying from $5$ to $35$. However, when we assess the identity test by incorporating zeros into the input, even at a smallest proportion of $25\%$ of the sequence length, GPT-4’s accuracy experiences a considerable decline, approaching zero.
A similar pattern is observed in GPT-3.5. A good analogy for the identity test is introducing irrelevance to the input of natural language. Consider the following example from the GSM-IC dataset (Shi et al., ):
{mdframed}
[backgroundcolor=lightgray]
Original Problem:
Q: Elsa has 5 apples. Anna has 2 more apples than Elsa. How many apples do they have together?
Problem with Irrelevant Context:
Q: Elsa has 5 apples. Anna has 2 more apples than Elsa. Liz has 4 peaches. How many apples do they have together?
The authors in (Shi et al., ) demonstrate that even with a variety of prompting techniques on the GSM-IC dataset the models are all sensitive to irrelevant information in the input. By leveraging group properties, specifically identity test, we can assess the robustness of the model in a more controlled and cost effective manner.
- •
In the identity test, we observed that the accuracy of models such as GPT-4 and GPT-3.5 decreases when zeros are inserted into the summation of ones. The performance decline follows this order: adding irrelevant information randomly, insertion in the middle, adding irrelevant information at the beginning, and lastly, at the end. Our results align with (Liu et al., ), which demonstrated that changing the position of relevant information within the input context of language models leads to variations in performance. This finding suggests that introducing irrelevant elements at various positions within natural language sequences can pose a significant challenge for models like GPT-4 or GPT-3.5. However, employing group properties and simple tests, such as addition, can aid in validating and identifying these issues early on with less computation costs.
- •
In our analysis of the inverse properties within group properties, we found that the LLMs investigated in this work display sensitivity when subjected to inverse tests. Our inverse test consists of adding negative ones to the sum of ones, yielding a final result of zero. Inverse properties bear resemblance to the introduction of negation in natural language. Similarly, the study by (Truong et al., ) highlights that LLMs exhibit multiple limitations in handling negation, including an inability to reason effectively under negation. By implementing the test framework proposed in this study, these issues can be identified in advance in a given LLM.
- •
In the associativity test, we decompose each summation into smaller steps for the model to process. Our findings indicate that GPT-4 demonstrates a significant improvement in performing addition tasks compared to GPT-3.5 when problems are broken down into smaller components. Our results highlight the importance of decomposing problems into smaller, more manageable pieces that can be effectively solved by the model which is reported using more complex tasks in (Kojima et al., ). By employing group analysis, it is possible to identify and address these issues through the implementation of straightforward tests proposed in this work.

## 2 Symmetry in LLMs

Symmetry and invariance are fundamental concepts in understanding the behavior of systems (Goodman et al., ). An object or quantity is said to be invariant if it remains unchanged under transformations (Kosmann-Schwarzbach et al., ). Consider an arbitrary quantity $F=F(A,B,C,...)$ that depends on different quantities. If we transform $A,B,C,...$ to their respective primed variables $A^{\prime},B^{\prime},C^{\prime},...$, and we have

$$ $F(A^{\prime},B^{\prime},C^{\prime},...)=F(A,B,C,...)$ (1) $$

then $F$ is said to be invariant under the given transformation.

Invariance can also be described through the concept of symmetry. A system is symmetric if it remains the same after a transformation or class of transformations. For instance, a physical system is symmetric under rotations if it can be rotated in any direction and remains unchanged. The set of all transformations that leave a given object invariant is called a symmetry group (Olver, ; Goodman et al., ).

Throughout this paper, we will use the following symmetries to investigate the capabilities of LLMs:

Translation Symmetry: This type of symmetry refers to the invariance of a system or pattern under a spatial transformation, such as shifting or sliding. In the context of LLMs, translation symmetry can help us understand how the models respond to changes in the position of elements within a sequence (Weyl, ).

Random Swapping Symmetry: Also known as permutation symmetry, this concept involves the invariance of a system or pattern under the exchange of its elements. In our study, we will use random swapping symmetry to analyze how LLMs handle rearrangements of input elements, providing insights into their robustness and generalization capabilities (French & Rickles, ).

Inverse Symmetry: This type of symmetry is characterized by the invariance of a system or pattern under an operation that reverses its elements or their order. In the context of LLMs, we will explore how the models perform when presented with inputs that have been transformed using inverse symmetry, shedding light on their ability to recognize and process different representations of the same information (Morandi et al., ).

To better understand the implications of these symmetries, we can examine them within the context of group theory.
A group is a set $G$ and binary operator ‘$\circ$’ that satisfies closure, identity, inverses, and associativity (Aschbacher, ; de La Harpe, ).

A group is a set $G$ and operator $\circ$ such that:

- •
Closure: $G$ is closed under $\circ$; i.e., if $a,b\in G$, then $a\circ b\in G$.
- •
Identity: There exists an identity element $e\in G$; i.e., for all $a\in G$ we have $a\circ e=e\circ a=a$.
- •
Inverses: Every element $a\in G$ has an inverse in $G$; i.e., for all $a\in G$, there exists an element $a^{\prime}\in G$ such that $a\circ a^{\prime}=a^{\prime}\circ a=e$.
- •
Associativity: The operator $\circ$ acts associatively; i.e., for all $a,b,c\in G$, $a\circ(b\circ c)=(a\circ b)\circ c$.

Group principles are essential in the laws of nature, as they encapsulate regularities that remain consistent regardless of specific dynamics. These principles impart structure and coherence to natural laws, enabling a better understanding of physical events and the discovery of the laws themselves.
(Schwichtenberg, ; Lax, ; Tung, ; Bishop, ; Golubitsky & Stewart, ).

Our goal is to conduct a comprehensive analysis for a given LLM to determine whether they maintain group properties and symmetry principles. In this investigation, we will cover a broad spectrum of symmetries, including but not limited to rotational, translation, scaling, swapping, and inverse symmetries.
Incorporating group properties and symmetry principles can help LLMs better understand and process structure and relationships. The following are some examples of how these concepts can enhance LLMs’ performance in various tasks:

- •
Paraphrasing: Understanding the symmetries in linguistic structures can help LLMs generate more accurate and diverse paraphrases. For example, given the sentence ‘Cats are great pets.’, an LLM aware of subject-object symmetry might generate a paraphrase like ‘Great pets are cats.’
- •
Text Classification: By identifying symmetries in text features, LLMs can better classify documents according to their topics, sentiment, or authorship. For instance, recognizing symmetry in word patterns and distributions could help LLMs differentiate between news articles and opinion pieces.
- •
Coreference Resolution: Understanding symmetries in pronoun-antecedent relationships can improve LLMs’ ability to resolve coreferences. For example, in the sentence ‘Sally talked to her friend, and then she went home,’ an LLM that recognizes the symmetry between ‘Sally’ and ‘she’ can infer that both refer to the same person.
- •
Syntax Parsing: Group theory concepts can be applied to analyze and parse the hierarchical structure of sentences. By recognizing the symmetry and relationships among different syntactic structures, LLMs can better understand and process complex sentences.

Examining group properties can help us understand the fundamental characteristics of the LLMs and potentially improve their performance in reasoning tasks due to the following factors:

- •
Identifying patterns: By leveraging group properties, LLMs can effectively identify and analyze underlying patterns and relationships within a problem, enhancing their problem-solving capabilities. For instance, the set of even numbers can be considered as a group, which enables an LLM to unveil underlying connections and patterns among them. For instance, it can determine that the sum of two even numbers is always even. By recognizing these patterns, the LLM can make accurate predictions and solve problems related to number sets and arithmetic operations more efficiently.
- •
Comprehending problem structure: Investigating group properties can potentially contribute to address complex reasoning tasks more efficiently. For instance, recognizing the group properties of integers under addition operator allows LLMs to devise strategic approaches to solve arithmetic tasks. When solving for the sum of a series of integers, the LLM can employ associativity to rearrange terms, thereby simplifying the problem. Additionally, recognizing the role of identity elements and inverses can help the LLM quickly identify shortcuts or eliminate unnecessary calculations, leading to more efficient problem-solving and accurate results.
- •
Reliable solutions: The utilization of group properties in reasoning tasks contributes to the generation of solutions that are both accurate and reliable, ensuring high-quality outcomes. For instance, if an LLM preserves the group properties of addition on a list of integers, it suggests that the LLM has a good understanding of the concept of addition operation.

Our contributions in this work are:

- •
An alternative evaluation perspective: Symmetry and group principles as tools for evaluating and understanding LLMs offer a different angle for evaluation, and can provide new insights about the inner workings of LLMs, helping to identify their strengths and weaknesses.
- •
Dataset: We have developed a synthetic dataset specifically designed to evaluate the performance of LLMs in terms of group principles. This dataset allows us to systematically test LLMs’ abilities to in terms of group principles.

## 3 Experiments

### 3.1 Group of Addition on a List of Integers

We choose the addition operator as the running example throughout the paper to show how the proposed framework operates. To test if a given LLM preserves the properties of the group of addition on a list of integers, the addition of integers in a list should ideally preserve certain symmetries, such as rotational, mirror, and swapping symmetries. To systematically assess the model’s ability to preserve these group properties, we propose the following experimental approach.

These tests ensure to check if a given LLM not only understands the concept of addition and the associated group properties but also the symmetries inherent in the operation.

### 3.2 Results

We present the results of our experiments on the summation group. The summation of integer values forms a group because the sum of any two integers is also an integer, thereby satisfying the closure property. The identity element in this group is zero, and the inverse of a value is its negative counterpart. Furthermore, the summation of integers preserves the associativity property. To assess the group properties using the summation operator, we work with a set of simple arithmetic expressions containing only the elements one, zero (serving as the identity element), and negative one (acting as the inverse element). We then proceed to investigate all the group properties within these expressions. Initially, we compared our findings with other open-source language models, such as LLAMA-2. However, the performance of these models was significantly lower than that of GPT-3.5 and GPT-4-32k, leading us to concentrate on the latter for our analysis. Our experiments involving these models can be found in Appendix Appendix: Experiments on SLM. We have conducted each experiment $10$ times and have reported the average accuracy.

### 3.3 Closure Test

For the closure test, we have examined a set of elements consisting solely of ones, combined using the summation operation. We have created sets of varying lengths and test expressions ranging from a summation of five ones ($1+1+1+1+1$) to a sum of $150$ ones.

The results are presented in Figure 1. The x-axis displays varying lengths of expressions consisting of repeated ones in summation, while the y-axis represents the accuracy of the two LLMs, GPT-4-32k and GPT-3.5. The color indicates the average accuracy over $10$ runs for each test. It becomes evident that GPT-3.5 and GPT-4-32k provide accurate results for sums of ones up to $35$ elements; however, their performance declines beyond this point, failing to maintain closure properties. This observation suggests that LLMs’ capabilities for performing summation are significantly limited beyond a certain sequence length. Another insight from Figure 1 is that these LLMs can accurately compute sums for $50$ and $100$ ones. However, further analysis reveals that they predominantly return values of $100$ and $50$ when the actual values are around and not exactly $100$ and $50$, respectively, indicating a significant bias towards these values. Figure 2 showcases the frequency of GPT-4-32k outputting $100$ (blue) and $50$ (red) when the ground truth ranges from $5$ to $150$ for closure test expressions which represents a bias towards these values. Additionally, this experiment demonstrates that, due to these biases, repeating the experiment will not improve the results.

Figure: Figure 1: Closure test: Average accuracy of GPT-4-32k and GPT-3.5 for sums of ones. The x-axis illustrates the varying lengths of expressions composed of summations of repeated ones. The y-axis denotes the accuracy of the two LLMs, GPT-4-32k and GPT-3.5. The color represents the average accuracy obtained from $10$ runs for each test.

Figure: Figure 2: Number of times GPT-4-32k outputs $100$ (blue) and $50$ (red) compared to ground truth for closure expressions. This visualization emphasizes the biases in the LLMs’ responses and offers a deeper insight into their limitations when handling summation tasks.
Refer to caption: x2.png

### 3.4 Identity Test

In our identity test, zero is the identity element for the summation of integers of ones. For this test, we will first add different proportions of zeros to our summation expressions. We will choose the ratios of $[0.25,0.5,0.75,1]$ for adding zeros with respect to the expression length. For example, for the ratio of $0.5$ and the expression $1+1+1+1+1$, we will modify it to $1+1+1+1+1+0+0$. By passing these expressions, we can investigate if and to what extent LLMs studied in this work can preserve the identity test.

In our experiment, we can also apply different symmetries to each expression. We apply inverse, random swapping, and translation symmetry. For the inverse symmetry, we will place zeros from the end of the sequence to the beginning, and for the translation symmetry, we will shift all the zeros to the middle of the expressions. For random swapping, we will swap the zeros randomly. Then, we provide these expressions as input to the LLMs. For example, for the expression $1+1+1+1+1$, we pass the following expressions to LLMs to examine their identity preservation test:

- •
$1+1+1+1+1+0+0$ (adding identity elements)
- •
$0+0+1+1+1+1+1$ (inverse symmetry)
- •
$1+0+1+1+1+1+0$ (random swapping symmetry)
- •
$1+1+0+0+1+1+1$ (translation symmetry)

Figure 3 presents the results of these expressions with different lengths ranging from $5$ to $150$ and displays the accuracy of each LLM on each of these tests with different ratios of adding zeros.
Altering the expressions using different symmetries changes the results and deteriorates the accuracy in many cases. The accuracy of these models degrades for various symmetric expression variations beyond a sequence length of $5$.

Figure: Figure 3: Identity Test. The average accuracy of GPT-4-32k and GPT-3.5 when evaluating sums of ones with varying expression lengths and applying different symmetries. The x-axis represents the expression lengths, while the y-axis indicates the accuracy for GPT-4-32k and GPT-3.5 under various symmetry conditions. The color intensity signifies the average accuracy obtained from $10$ runs for each test.
Refer to caption: x3.png

### 3.5 Inverse Test

To test the inverse properties of a group, we add the negative values of each expression to the expression itself and examine different variations of the expression. We then test different modifications of the expression using symmetry principles, such as inverse, random swapping, and translation symmetry. For inverse symmetry, we add the inverse element to the beginning of the expression. For translation symmetry, we add the inverse element to the middle of the expression. For random swapping, we add the inverse elements to random positions within the expression. For example, to test the inverse properties, we modify the expression 1+1+1+1+1 to the following variations, with the expectation that the ground truth for each expression should be zero:

- •
$1+1+1+1+1-1-1-1-1$ (adding inverse elements)
- •
$-1-1-1-1-1+1+1+1+1+1$ (inverse symmetry)
- •
$1+1-1+1-1-1+1-1-1+1$ (random swapping symmetry)
- •
$1+1+1-1-1-1-1-1+1+1$ (translation symmetry)

There is a caveat here: if LLMs are biased towards zeros, this might affect the results. However, this is not a significant concern, as we are not relying solely on this experiment to understand how inverse properties impact these models. We could test these expressions by changing them in a way that the summation is non-zero, for example to 1+1+1+1+1-1-1 or 1+1+1+1+1-1. However, for simplicity, we will only consider adding the same number of negative ones to each expression in this experiment. Figure 4 shows the result of our experiment for the inverse properties. We can observe from the figure that the LLM’s outputs for some of these expressions fail early on, while for others, it returns accurate results. This suggests that the model’s performance may be influenced by the complexity or structure of the expressions.

Figure: Figure 4: Inverse Test. The average accuracy of GPT-4-32k and GPT-3.5 when evaluating sums of ones and their inverses for various lengths. The x-axis represents the expression lengths, while the y-axis indicates the accuracy for GPT-4-32k and GPT-3.5 under various inverse symmetry conditions. The color intensity signifies the average accuracy obtained from $10$ runs for each test.
Refer to caption: x4.png

### 3.6 Associativity Test

In this section we investigate the group’s associativity properties. This test is crucial because if LLMs can understand associativity properties, it implies that they can simplify problems by decomposing them down into smaller components and solving them. Another benefit of analyzing this property is we can mitigate possible biases for example in a case like the summation of values near $100$ where LLMs produce an incorrect output of $100$, the associativity properties can be used to break down the problem into smaller components and mitigate the effect of biases toward these values.

For the associativity test, the expressions from the closure test have been broken down into smaller components. We have decomposed the original expression into smaller segments with ratios of $3/8$ and $5/8$, referred to as test 1, and into segments with ratios of $1/4$ and $3/4$, referred to as test 2. For each ratio, the query is divided into two segments and each segment has been passed to the LLM to obtain the result. Then, the outputs of the two segments are provided and the LLM is queried to obtain the final result. For example, for the query $1+1+1+1+1+1$ with test 2, we break it down into two segments: $1$ and $1+1+1+1$, and pass each one to the model. Suppose the LLMs’ output for each query is $1$ and $4$; then, we will pass $1+4$ to the LLMs one last time to obtain the final result. We will repeat each experiment $10$ times and report the average accuracy. Figure 5 shows the results of the associativity test. We can derive several insights from this experiment:

- •
Breaking down problems into smaller sub sequences improves the accuracy of LLMs.
- •
Segmenting and breaking down problems using test 1 yields better results for LLMs. This is reasonable since the other test creates shorter and longer segments, with the longer ones being more difficult for LLMs to solve, leading to a higher number of incorrect answers overall.
- •
LLMs fail to preserve associativity beyond a certain point, as they also fail to maintain the closure test.

Figure: Figure 5: Associativity Test. The average accuracy of GPT-4-32k and GPT-3.5 for the associativity test for test 1 (top) and test 2 (bottom). The x-axis represents the expression lengths, while the y-axis indicates the accuracy for GPT-4-32k and GPT-3.5. The color intensity signifies the average accuracy obtained from $10$ runs for each test.
Refer to caption: x5.png

## 4 Conclusion and Future Work

In this study, we introduced a framework for testing the behavior of large language models based on group and symmetry principles. Our experiments involved GPT-4 and GPT-3.5, examining their performance on four group properties: closure, identity, inverse, and associativity. The results indicated that they face challenges in maintaining group properties under various circumstances. We also showed that similar behavior is observed for small language models like Llama2, Mistral, Vicuna and Phi2.
The models showed significant performance drop across all tests, which might be attributed to the insufficient memory retention with respect to context. For example, after a sequence length of $15$ for identity test and sequence length of $50$ for closure, performance becomes $0\%$. We conducted tests with both open-weight and closed-weight models, finding that although different tokenizers may contribute to the problem, the root cause could extend beyond better tokenization strategies. A good direction for future work is exploring whether these tests can provide insights and be used as a predictor for model’s performance on real-world language understanding and generation tasks.

## Appendix: Experiments on SLM

In the appendix we show some of our experiments with smaller language models. Our initial tests 3 presented challenges for smaller language models like Mistral-7B-Instruct. For example, in the identity test with a sequence length of five, the model had trouble generating the accurate ground truth as shown in Table 1.

**Table 1: Mistral-7B-Instruct output for the identity test with a ground truth of $5$, as described in Experiment 3.**
| Test | Ground Truth | Mistral-7B-Instruct |
| --- | --- | --- |
| Identity Test | 5 | [6, 6, 7, 5, 6] |
| Translation Symmetry | 5 | [4, 5, 4, 4, 4] |
| Inverse Symmetry | 5 | [6, 6, 6, 6, 6] |
| Random Swapping | 5 | [6, 6, 6, 6, 6] |

As a result, we designed simpler tests specifically tailored for these models. In this study, we develop and release a dataset containing arithmetic questions embedded within natural language texts set in real-life scenarios. We focus on testing both smaller and larger models with this dataset, as the required reasoning involves only a few steps. The objective of the experiment is to apply group and symmetry principles to evaluate the language models’ ability to solve arithmetic problems accurately. Furthermore, this experiment emphasizes the practical application of group and symmetry principles, aiming to uncover biases and limitations within these models.

The gray boxes showcase a selection of examples from this dataset.

- •
After creating the questions, we use the random swapping symmetry technique to shuffle the middle sentences of each question in the dataset, while keeping the first and last sentences in their original positions. By utilizing the random swapping symmetry technique, we aim to investigate its effect on the language model’s performance. For each question, we generate $5$ different random permutations of the sentences in the text and report the average accuracy as the performance metric. This approach allows us to assess the model’s ability to adapt to varying sentence structures and maintain its problem-solving capabilities.
- •
In the second part of the experiment, we introduce adding irrelevant information to the questions to further evaluate the model’s capabilities. For each question, we generate additional sentences, , with one example showcased in the purple box titled ’Irrelevant information’ (see Appendix: Experiments on SLM). We then shuffle the middle sentences of each question in the dataset, along with the irrelevant sentences. We provide the modified input to the model and perform five iterations to calculate the average accuracy as a performance metric. The results can be found in Table 2.

By applying the random swapping symmetry technique, we observed that the Mistral-7B-Instruct language model’s output, indicated in blue, consistently exhibited missing or skipped parts shown in red in the same location across all shuffled versions for some of the questions in the dataset.

[backgroundcolor=lightgray, frametitle=Mistral-7B-Instruct, Version 1]
Samantha has 2 stuffed animals. Then, she received one more stuffed animal from her friend. Next, her aunt gave her one more stuffed animal. Then, she won a stuffed animal at a school event. She bought a stuffed animal from her favorite store. Later, she found one more stuffed animal in her toy box. She also bought one stuffed animal from the mall. Her brother gave her another stuffed animal. After that, her dad bought her another stuffed animal. Then she borrowed another stuffed animal from her sister. Finally, she got one more stuffed animal as a gift from her neighbor. How many stuffed animals does Samantha have now?

Mistral-7B-Instruct output:
Samantha started with 2 stuffed animals. After receiving one from her friend, her aunt gave her one, she won one, bought one, found one, bought one, received one from her brother, received one from his dad, and received one from her neighbor, Samantha now has a total of 11 stuffed animals

[backgroundcolor=lightgray, frametitle=Mistral-7B-Instruct, Version 2]
Samantha has 2 stuffed animals. Then, she received one more stuffed animal from her friend. Her brother gave her another stuffed animal. Then, she won a stuffed animal at a school event. She also bought one stuffed animal from the mall. Then she borrowed another stuffed animal from her sister. Next, her aunt gave her one more stuffed animal. She bought a stuffed animal from her favorite store. Later, she found one more stuffed animal in her toy box. After that, her dad bought her another stuffed animal. Finally, she got one more stuffed animal as a gift from her neighbor. How many stuffed animals does Samantha have now?

Mistral-7B-Instruct output:
Samantha started with 2 stuffed animals, then she received 1, her brother gave her 1, she won 1, bought 1, borrowed 1, received 1, bought 1, found 1, and received 1 from neighbor, making a total of 11 stuffed animals.

[backgroundcolor=lightgray, frametitle=Mistral-7B-Instruct, Version 1]
Emily has two puzzles. Then, she received one more puzzle from her friend. Next, her aunt gave her one more puzzle. Then, she won a puzzle at a school event. She bought a puzzle from her favorite store. Then she borrowed another puzzle from her sister. Later, she found one more puzzle in her toy box. She also bought one puzzle from the mall. Her brother gave her another puzzle. After that, her mom bought her another puzzle. Finally, she got one more puzzle as a gift from her neighbor. How many puzzles does Emily have now?

Mistral-7B-Instruct output:
Emily started with two puzzles. She received one puzzle from her friend, one from her aunt, won one at a school event, bought one, borrowed one from her sister, found one, bought one at the mall, received one from her brother, and received one as a gift from her neighbor. In total, Emily now has 13 puzzles.

[backgroundcolor=lightgray, frametitle=Mistral-7B-Instruct, Version 2]
Emily has two puzzles. She bought a puzzle from her favorite store. Then, she received one more puzzle from her friend. Her brother gave her another puzzle. After that, her mom bought her another puzzle. Later, she found one more puzzle in her toy box. Next, her aunt gave her one more puzzle. Then, she won a puzzle at a school event. She also bought one puzzle from the mall. Then she borrowed another puzzle from her sister. Finally, she got one more puzzle as a gift from her neighbor. How many puzzles does Emily have now?

Mistral-7B-Instruct output:
Emily has a total of 13 puzzles now. She got one from her favorite store, one from her friend, one from her brother, one from her mom, one from her toy box, one from her aunt, one from the school event, one from the mall, and one from her neighbor.

We observed that the Llama-2-7B-chat model exhibits similar behavior, where the language model skips some of the context of input query, leading to incorrect answers.
{mdframed}[backgroundcolor=lightgray, frametitle=Llama-2-7B-chat]
Samantha has one stuffed animal. Next, her aunt gave her one more stuffed animal. She got one more from her closest friend. After that, her dad bought her another stuffed animal. Later, she found one more stuffed animal in her toy box. Then, she won a stuffed animal at a school event. She also bought one stuffed animal from the mall. Her brother gave her another stuffed animal. Then she borrowed another stuffed animal from her sister. Then, she received one more stuffed animal from her friend. She bought a stuffed animal from her favorite store. Finally, she got one more stuffed animal as a gift from her neighbor. How many stuffed animals does Samantha have now?

Llama-2-7B-chat output: Answer: Samantha has 10 stuffed animals. Explanation: 1. She had one stuffed animal to start with. 2. Her aunt gave her one more, so she has 2 stuffed animals now. 3. Her closest friend gave her one more, so she has 3 stuffed animals now. 4. Her dad bought her another, so she has 4 stuffed animals now. 5. She found one more in her toy box, so she has 5 stuffed animals now. 6. She won one at a school event, so she has 6 stuffed animals now. 7. She bought one from the mall, so she has 7 stuffed animals now. 8. Her brother gave her another, so she has 8 stuffed animals now. 9. She borrowed one from her sister, so she has 9 stuffed animals now. 10. She received one more from her friend, so she has 10 stuffed animals now. Therefore, Samantha has 10 stuffed animals in total.

One implication of group and symmetry principles in this scenario is that by utilizing the output of the language model, we can identify the missing parts and potentially address inconsistencies in the output of language model which could lead to further improvements in the model’s performance and a deeper understanding of the underlying patterns.

Table 2 shows the accuracy of different models on the for this dataset. Additionally, we introduce irrelevant information to each question using GPT4 model and calculate the accuracy of different models. For instance, in the case of question 1 within the box, we incorporate the following irrelevant information:

**Table 2: Comparison of language model accuracy, highlighting the impact of question permutation and the introduction of irrelevant information on model performance.**
| Model | Permutation | Irrelevancy |
| --- | --- | --- |
| Mistral-7B-Instruct | 26.6% | 18% |
| Mistral-7B-Instruct + COT | 33.3% | 24% |
| Llama-2-7B-chat | 6.89% | 1.96% |
| Llama-2-7B-chat + COT | 47.45% | 5% |
| GPT4 | 16.6% | 42% |
| GPT4 + COT | 100% | 80% |

Another observation we made was that employing a chain of thought and step-by-step thinking improves the performance of the model, as demonstrated in (Kojima et al., ). However, introducing irrelevant information to the question still negatively impacts the performance of language models like GPT-4-COT, resulting in a 20% decrease.

The accuracy of GPT-4 increased when irrelevant information was added to the question. Upon further investigation, we discovered that GPT-4, when presented with irrelevant information, generated answers that were closer to step-by-step thinking.

## Appendix: Application of Symmetry Principles in GPT-4v Model Use Cases

In another experiment, we utilized GPT-4v to find the behaviour of GPT4-v under permutation symmetry. In this experiment, given a query figure we ask GPT4-v to compare it with another two figures using the following prompt.

We conducted two tests to explore GPT-4v’s limitations in using group and symmetry principles. In both tests, we compared the query figure to Figure 1 and Figure 2, with one of the figures being identical to the query figure and the other being different. In the first test, Figure 1 was identical to the query figure, while in the second test, Figure 2 was identical.

We found that in first setting that Figure 1 is identical to query figure the GPT4-v response was:

However, for the second test, where Figure 2 was identical to the query figure, the response was:

In another case, all three figures were exactly the same, and this was the response provided by GPT-4v:

Upon examining figure permutations, we found that the GPT-4v model’s responses can vary significantly, highlighting its sensitivity to figure arrangement. By employing symmetry and group principles, we can better understand these inconsistencies and improve the model’s performance.

## Appendix: Ablation Study

To investigate whether there is a bias towards specific points like $100$ and $50$, and to ensure that this is not solely related to the number 1s used in the study, we also experimented with substituting 1s with words such as ‘apples’, ‘oranges’, and ‘bananas’. We asked the language models to count the number of these items within the ranges $[40-60]$ and $[90-110]$, and The predictions obtained when substituting ones with ‘apples’ are illustrated in Figure 6. Similar behavior was observed for other cases as well. Red dots represent incorrect predictions, while blue dots indicate correct predictions. As can be seen, the language model frequently predicts incorrectly for values around $100$ and $50$, exhibiting a bias towards these values.

Figure: Figure 6: Scatter plot of GPT-4-32k predictions for closure test expressions using various items, illustrating the bias towards values $50$ and $100$. Red dots represent incorrect predictions, while blue dots indicate correct predictions. The plot demonstrates the model’s tendency to predict $50$ and $100$ more frequently, even when the actual values are slightly different.
Refer to caption: x6.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 1
questions: - In the abstract, the authors mention “assessing [language models’] reasoning capabilities remain a significant challenge. I am unaware of such challenges and the authors do not elaborate on this point in the main text further? Can you give some examples from the literature where people have shown this to be the case?
- Why do the authors call the closure test the closure test? It seems that the closure property ($a, b \in G \Rightarrow a\circ b \in G$) is not being tested, but rather the abelian property of the group (lines 231-237).
- The closure test described in lines 231-237 seems different from what the authors describe in lines 281-295. And, after reading this latter section, I no longer understand what the goal of this test is. Can the authors elaborate on this?
- Lines 441-442, for the associativity test, the authors choose only ratios (3/8, 5/8) and (1/4,3/4). Why were these specific numbers chosen and are there ablations over the “pivot” point. Did the authors test instead of splitting the problem into two distinct parts, splitting it into multiple (i.e. greater than 2)?
- Lines 455, the authors claim to that LLMs fail to preserve associativity beyond a certain point. It seems that there are multiple confounding variables here. For example, if you have length 120 then a (3/8,5/8) split corresponds to lengths (45, 75). Of these two splits, the longer one is essentially the closure test. So, why is the closure test not a subset of the associativity test?
- Based on the previous question, I guess I don’t understand what the associativity test is testing. Because you are breaking the problem down for the language model, rather than the language model recognizing that there are recursive subproblems here. So even if you demonstrate that an LLM perfectly solves this associativity test, what would you be able to say about how its generalizing?
- The authors extend their results from addition problems to the GSM8K dataset. They add irrelevant information to the prompts using GPT. How did you verify that the information being added was indeed irrelevant? I do not see any information on this in the appendix.
soundness: 1
strengths: The symmetries and structure inherently contained in a problem can often give us a lot of information about it. I agree with the authors that this is a fundamental idea which has already permeated many areas of science. Recently, we have already seen them being applied to machine learning through many facets (e.g. geometric deep learning, neurosymbolic AI, and more recently, interpretability). Thus, the topic of the paper is timely and relevant to the field.
summary: In this paper, the authors develop a framework for language model evaluation based on group properties and demonstrate its application to addition. Their evaluation framework centers on four key properties: closure, identity, association, and inverse. The authors reveal that GPT-3.5 and GPT-4 fail to maintain consistent performance across closure, identity, and association tests. Moreover, they show that leveraging the associative property of addition—breaking down large addition problems into smaller, more manageable pieces—improves the performance of these language models.
weaknesses: - The authors strongly emphasize that the contribution of the paper is *not* their benchmarks on arithmetic tasks but rather a general benchmarking framework. In this case, the contribution of the paper is unclear to me. Based on my understanding of the literature, it seems that the field is already aware of using  group properties/structures to improve/evaluate model performance (see [1,2,3,4,5] and much more…). To this end, many benchmarks like the authors mention themselves already incorporate these elements into their evaluation process [6].
- Given this example of arithmetic, it is unclear to me how this framework would generalize to other more complicated tasks that the community is interested in: one where the symmetries exist, but it is not at all obvious how to design benchmarks based on these symmetries. Consider examples like text-to-image, image-to-text, hallucination detection, alignment, entity resolution, etc.
- As is, the paper is very poorly written. It is hard to follow what the main claims, the experimental settings, and what even the main results are.
- The related literature section is also nonexistent/weak. I feel that the related literature should answer at least the following two questions:
    - How is this framework similar/different to other evaluation frameworks that already exist in the literature?
    - Are the experiments/results that we get from this framework consistent with what we see in the related literature (on this arithmetic task). Or even better, do the experiments/results generalize or explain some phenomena that the community does not understand?

I strongly believe this work is still in progress and would greatly benefit from substantial revisions and additional rounds of review beyond the scope of the ICLR review cycle. 

[1] Godfrey, Charles, et al. "On the symmetries of deep learning models and their internal representations." *Advances in Neural Information Processing Systems* 35 (2022): 11893-11905.

[2] Liu, Ziming, and Max Tegmark. "Machine learning hidden symmetries." *Physical Review Letters* 128.18 (2022): 180201.

[3] Winter, Robin, et al. "Unsupervised learning of group invariant and equivariant representations." *Advances in Neural Information Processing Systems* 35 (2022): 31942-31956.

[4] Crabbé, Jonathan, and Mihaela van der Schaar. "Evaluating the robustness of interpretability methods through explanation invariance and equivariance." *Advances in Neural Information Processing Systems* 36 (2023): 71393-71429.

[5] Wang, Yipei, and Xiaoqian Wang. "Self-interpretable model with transformation equivariant interpretation." *Advances in Neural Information Processing Systems* 34 (2021): 2359-2372.

[6] Srivastava, Aarohi, et al. "Beyond the imitation game: Quantifying and extrapolating the capabilities of language models." *arXiv preprint arXiv:2206.04615* (2022).

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: - Sections 3.1 and "Experiments on SLM" section of the appendix demonstrate that this framework can be utilized with any set of whole numbers. Why are the main experiments limited to elements from the set {0,1}?
- Similarly, in order to more accurately model real world problems, which contain both affirmative and negative information, it may be useful to conduct these addition-based group property evaluations with elements from the group {-1, 0, 1}.
- The translation and random swapping tests, specifically in the context of identity preservation, may result in the same input pattern. Is there any significance to or observable insight from the differing results of these tests?
- What is the intuition behind the ratios selected for Test 1 and Test 2 of the associativity test (Sec 3.6)? Providing results over a range of ratios may better identify the optimal split.
- This work proposes a variant of GSM-8K with irrelevant information in Section 3.7. How does this variant differ from existing adversarial datasets such as GSM-8K (M3) [1], GSM-8K-Adv [2], and GSM-NoOp [3], which are also perturbed variants of GSM-8K?



[1] Xie, R., Huang, C., Wang, J., & Dhingra, B. (2024). Adversarial Math Word Problem Generation.

[2] Anantheswaran, U., Gupta, H., Scaria, K., Verma, S., Baral, C., & Mishra, S. (2024). Cutting Through the Noise: Boosting LLM Performance on Math Word Problems.

[3] Mirzadeh, I., Alizadeh-Vahid, K., Shahrokhi, H., Tuzel, O., Bengio, S., & Farajtabar, M. (2024). GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models.
soundness: 2
strengths: - There is definite value to examining the types of reasoning errors by testing independent properties to identify these patterns of behaviour in other tasks.
- This work builds on previous literature which demonstrate the susceptibility of models to simple perturbations in input data.
- There is some (qualitative) analysis comparing model outcomes on actual mathematical word problems (MWPs).
- The theoretical basis is well-researched and results in a useful framework for identifying error patterns beyond simple accuracy measures.
- There are some interesting insights re: model bias for certain numbers.
summary: This work examines how well LLMs adhere to principle of symmetry and group properties such as associativity and transitivity in their reasoning. These capabilities are tested by way of conducting experiments over four properties: closure, identity, inverse, and translation. These experiments reveal persistent reasoning errors ranging from bias towards certain numbers to a sensitivity to perturbations. This work also correlates these error patterns over the simple arithmetic setting to similar behaviours over more complex tasks. Finally, a synthetic dataset is proposed that contains these adversarial data samples curated to identify model reasoning pitfalls.
weaknesses: - The implication of the results and analyses in this work is that if LLMs can pass these property-based tests, they exhibit certain skills associated with an understanding of these properties. eg. 
if this framework shows that a model has an understanding of associativity, the model can decompose and solve problems. However, there is no evidence provided for this hypothesis, for example by comparing model accuracy on the associativity test and in parallel on another pre-existing task that requires decomposition.
- Similarly, there is a lack of more comprehensive quantitative results based on accuracy metrics that confirm the hypothesis that patterns observed over the simple {1,0} addition task are also observed over more complex tasks such as solving natural language MWPs. This makes it difficult to definitively prove the claim that these tests can offer insights into the type of reasoning errors made by models.
- The {1,0} based addition test is arguably too simplistic to gauge overarching error patterns in LLMs. Furthermore, the tests proposed in this work (identity, closure, etc) could be performed on problems of far greater complexity.
- Given the insight that models are biased towards numbers such as 0, 50, 100: the inverse test is likely to yield optimistic results, as the correct answer is always 0. It may be more illustrative to perform this test with negations that amount to non-zero results, such that the sum of results over all tests in the experimental setting is zero.
- Table 3: there is no baseline performance provided over questions in their native, unperturbed state.
- Page 15: The prompt to rate similarity does not definitively define the rubric, and there is potential for ambiguity in interpreting the descriptions associated with the scores 2, 3, and 4.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 3
questions: What insights beyond those in [1,2,3,4] cited above does your benchmark provide? What is the cost difference between using your benchmark and a set of benchmarks that surface the same weaknesses?
soundness: 2
strengths: **Originality**: The first-principles approach of this paper is refreshing. Grounding a language model robustness evaluation dataset in group theoretical invariances is both clever and has the potential to provide insights beyond what is currently known about language models' behaviors.

**Quality**: The quality of the writing is overall good. It situates the work with ample and appropriate citations of related literature. 

**Clarity**: The authors clearly state the purpose and contribution of the paper. The method is described clearly and illustrated nicely in the figure on p. 5.

**Significance**: Analyses of this sort have the potential to contribute valuable insights into the workings of language models and how to improve them.
summary: This paper takes a first-principles approach to the evaluation of language models. The authors create a succinct dataset of prompts comprising addition operations under various perturbations. The purpose of the perturbations is to evaluate whether, and the extent to which, a language model is invariant to such perturbations. The perturbations test closure, identity, inverse, and associativity properties. 
 With this dataset, they report results on GPT 3.5 and 4.0 in the main body of this paper, and results on other models in the appendix. The authors also introduce irrelevant text into the prompts of GSM8K and report results on GPT-4o and Mistral-7B Instruct.
weaknesses: **Originality**:  In general, papers that focus on specific operations -- like this paper, or like [1] -- demonstrate a weakness, provide valuable insights based on that weakness, and possibly explore experimentally ways to overcome the weakness. This paper demonstrates weaknesses and derives little new insights. While the concept of the dataset described in this paper is novel in ways, I'm afraid that the originality of the *insights* provided by the dataset are quite limited. The limitations of language models under conditions that require variable amounts of computation are pretty well known by now. For instance, the insight that language model accuracy on arithmetic tasks is constrained by the length of the (implied) computational graph was precisely defined and thoroughly evaluated in [1]. Their robustness under variation in the position of inputs in the context window have been explored in [2]. And the effectiveness of improving language model accuracy on such tasks by encouraging them to traverse high-probability regions of their hidden states is also well known [3]. I would encourage the authors to consider how group theoretical principles could be explored more thoroughly or systematically to provide insights beyond what has been shown already. For example, some recent work [4] has shown that recurrent networks consistently out-perform transformers as detectors of regular languages. Do similar inequalities exist between network architectures with respect to group theoretical principles?

**Quality**: The scale of the experiments reported here are small (10 runs) number of models evaluated is modest. The results of the dataset created by the authors are reported on GPT 3.5 and 4.0. The ancillary experiment described in the paper was performed on GPT-4o and Mistral-7B-Instruct. While Open AI only released their o1 model on September 12th of this year, the dataset in this paper is so small that running it against o1 could provide insight into a variable-computation model that has not yet been evaluated much (although see [5]). While simplicity is a virtue, the dataset evaluated here could be created in, I estimate, about 4 hours, so the authors could have spent much more time on other things, like evaluating group theoretical principles on more than just addition. 

**Clarity**: I think the clarity of the paper is not a weakness. I would encourage the authors in a future version of the paper to put less emphasis on the potential insights that group theoretical principles can bring to language model evaluation and much more on the actual insights.

**Significance**: See my comments above under **Originality**. One point made by the authors is that the sheer smallness of their dataset implies that problems identified by other, larger datasets can be done at lower cost using their dataset. If this is indeed the case, the paper would be improved by showing at a minimum how much less expensive it is. A potentially interesting future line of research is robustness benchmarks with provably minimal cost and provable equivalence (of what is tested and surfaced) with more expensive benchmarks.

* [1] Dziri et al, Faith and Fate: Limits of Transformers on Compositionality, NeurIPS 2023
* [2] Liu et al, Lost in the Middle: How Language Models Use Long Contexts, TACL 2024
* [3] Wei et al, Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, NeurIPS 2022
* [4] van der Poel et al, MLRegTest: A Benchmark for the Machine Learning of Regular Languages, JMLR 2024
* [5] Valmeekam et al, LLMs Still Can't Plan; Can LRMs? A Preliminary Evaluation of OpenAI's o1 on PlanBench, arXiv:2409.13373

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 1
questions: The methods are mostly clear. It is mostly a presentation and experimental scope issue. See weaknesses.
soundness: 3
strengths: The authors use the known approach of analyzing symmetry within a system to probe LLMs and discover their shortcomings.
summary: The authors present different principles of symmetry and explain how those principles can be used to analyze the shortcomings of Large Language Models (LLM). The authors then provide experiments across the different principles to show how and where these LLMs fail, specifically looking at how these models perform under larger contexts over varying symmetries. All experiments are conducted on varying chains of addition between -1, 0, 1.
weaknesses: Main concerns:
1. The presentation of the results is extremely poor. Do not use heat maps to present accuracy. Typically you want to use line graphs for experiments that have are varying one parameter many times. Try using line graphs or bar graphs for all of the figures that are heat maps.
2. In the experimental sections, each of the results are shortly discussed in regard to how they show that the LLMs are failing. It would be useful to also discuss how these insights can lead us to build better LLMs.
3. The experimental setting of using only -1, 0, 1 makes these experiments feel like toy experiments. Using sentences and paragraphs, natural language, would be much more interesting and significant.
4. Excessive use of bullet points. Write out full paragraphs.
5. From this paper, it is still unclear how closure and associativity are different.
6. The introduction and background span 4 full pages. For a paper such as this, you could probably cut down 1 or 2 pages of content from those 4 pages so that there is more room to provide experiments and thoroughly discuss results.

Writing Comments:
Page 1, Line 41: Group and symmetry principles which made --> Group and symmetry principles have made
Page 2, Line 60 - 102: These do not need to be bullet points. The paper would be more aesthetically pleasing if the bullet points were removed and the topic markers were instead bolded. Also, Page 2, Line 58 should probably be changed to mention each of the topics instead of ending on colon.
Page 3,  Line 113 - 126: The break from bullet points and figure insertion looks odd. Keep the text with the previous bullet point and move the figure to the top of the page.
Page 4 - 5, Line 198 -222: Again, the topic markers for each bullet point can just be bolded and the bullet points can be removed.
Page 3 & 5: Should the figures presented on each page not have a caption and figure number?
Page 5, Line 269: one, zero, and negative one --> 1, 0, -1
Page 8, Line 407 - 408: Having 1.5 lines between two almost full pages figures is poor presentation.

Meta-review: This paper aims to develop a general framework to evaluate the reasoning capabilities of LLMs using group properties. While the topic is timely and important, all four reviewers give low scores due to the following concerns:  1) the contribution is not clear as group properties are frequently used for evaluation in the literature; 2) the presentation is poor that the literature review is insufficient and the main conclusions are not clear; 3) the experiment is limited to very simple scenarios. No rebuttal is provided from the authors. AC agrees this work is still in progress and requires substantial improvement.

Decision: Reject