Alice in Wonderland: Simple Tasks Reveal Severe Generalization and Basic Reasoning Deficits in State-Of-the-Art Large Language Models

Authors: Marianna Nezhurina, Lucia Cipolina-Kun, Mehdi Cherti, Jenia Jitsev
Primary area: foundation or frontier models, including LLMs
Keywords: large language models, foundation models, generalization, reasoning, function testing, evaluation, benchmarks, robustness, function breakdown

Abstract:
Large Language Models (LLMs) are often described as being instances of foundation models - that is, models that possess strong generalization and therefore transfer robustly across various tasks and conditions in few-show or zero-shot manner, while exhibiting scaling laws that predict generalization improvement when increasing the pre-training scale. These claims of strong generalization and advanced reasoning function enabling it rely on measurements by various standardized benchmarks where state-of-the-art (SOTA) models score high. We demonstrate here a dramatic breakdown of generalization and basic reasoning of all SOTA models which claim strong function, including advanced models like GPT-4 or Claude 3 Opus trained at the largest scales, using a simple, short common sense problem formulated in concise natural language, easily solvable by humans (AIW problem). The breakdown is dramatic as it manifests in both low average performance and strong performance fluctuations on natural problem variations that change neither problem structure nor its difficulty, while also often expressing strong overconfidence in the wrong solutions, backed up by plausible sounding explanation-like confabulations. Various standard interventions in an attempt to get the right solution, like chain-of-thought prompting, or urging the models to reconsider the wrong solutions again by multi step re-evaluation, fail. We take these observations to the scientific and technological community to stimulate re-assessment of the capabilities of current generation of LLMs as claimed by standardized benchmarks. Such re-assessment also requires common action to create standardized benchmarks that would allow proper detection of such deficits in generalization and reasoning that obviously remain undiscovered by current state-of-the-art evaluation procedures, where SOTA LLMs obtain high scores. Code for reproducing experiments in the paper and raw experiments data can be found at https://anonymous.4open.science/r/AITW_anonymous-69A6/

Decision: reject
Reviewer scores: 6, 5, 3, 6, 6

Reviews and discussion:
<thread>
creation date:
1730163009925

thread_type:
official_review

<thread_object>
creator: reviewer_hx2d

<content>
summary:
Authors study reasoning capabilities of various SOTA LLMs in a controlled environment, by synthesizing a very simple yet efficient task, Alice in Wonderland (AIW). This task is composed of multiple variations of the following template: "Alice has N brothers and she also has M sisters. How many sisters does Alice’s brother have?". Authors systematically study more than 20 models by varying M/N, changing family relations, introducing redundant information, and varying between prompt templates. Authors showed that models not only fail on this simple task showing high variation between prompt templates, but also that their failure can not be attributed to arithmetic or commonsense knowledge errors, but occur due to the lack of generalization and basic reasoning abilities.

strengths:
- Authors discuss an important problem of LLM reasoning abilities
- Paper is clearly written
- Proposed evaluation framework is novel and simple to implement and verify
- Authors perform extensive experiments across various models and prompt templates
- Detailed ablation studies on AIW variations support main claims of the paper

weaknesses:
- Even though authors prove that SOTA LLMs fail on AIW task, I don't think we can claim that they are not capable of robust reasoning. On the contrary, paper shows that LLMs are capable of some types reasoning (like arithmetic, or basic family relations), but fail on the others (logical reasoning).
- There are multiple formatting issues in the paper, probably caused by moving text between templates, that hurt overall presentation of the paper (see Questions section for example).

questions:
1. Why do you think Llama-3 performs so much worse then Llama-2 model? What framework did you use to run evaluations for models with  open weights? I wonder, if there might be any issues with prompts, as Llama-3 models require different special symbols in chat template than Llama-2.

2. There are some issues with citations across the paper where brackets are missing in most of the citations, for ex. in lines 41-42: "...visual recognition Radford et al. (2021) or language understanding Devlin et al.(2018); Raffel et al. (2020); Brown et al. (2020), l..." should be  "...visual recognition (Radford et al., 2021) or language understanding (Devlin et al., 2018; Raffel et al., 2020 ...".  Multiple periods are missing: lines 111, 117, 192, 309, 458, 460, 463.  Chapter 3.1.1 "original. like" -> "original, like" in multiple places.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730537910771

thread_type:
official_review

<thread_object>
creator: reviewer_5s6m

<content>
summary:
This paper demonstrate a dramatic breakdown of generalization and basic reasoning of all SOTA models (ncluding advanced models like GPT-4 or Claude 3 Opus) which claim strong function, using a simple, short, conventional common sense problem formulated in concise natural language (AIW problem). The authors observe that large language models (LLMs) exhibit significant performance fluctuations on simple problems across minor variations that should not impact problem-solving ability at all. Additionally, various standard interventions, such as chain-of-thought prompting, failed to yield correct solutions in the AIW problem. These observations highlight the need to re-evaluate the claimed capabilities of the current generation of LLMs.

strengths:
* This paper is well-written and presents clear ideas.

* The authors conduct extensive experiments on over 36 LLMs to demonstrate the breakdown of SOTA LLMs on the somple AIW problem.

* Fully Open-sourced code and data to reproduce the result.

weaknesses:
* The problem setting of AIW has certain interfering factors. After some attempts, I found that the main reason LLMs perform poorly on AIW origin is due to easy thinking. For example, "Alice has 3 brothers, and each of these brothers has the same sisters, who are Alice's sisters. Alice has 6 sisters, so each of her brothers has 6 sisters." The issue here is that the LLM overlooks counting Alice herself, rather than lacking reasoning ability. I believe that testing similar problems in mathematical reasoning tasks (like GSM8K or MATH) would be more convincing.

* Quite a few typo errors. line 016, few-show -> few-shot; format of most of the citations is wrong.

questions:
See weaknesses.

Recently, I came across another paper that is similar in content to this study. I know this paper was published online prior to [1],  I'm just curious about what advantages the authors believe the AIW dataset presented in this paper has compared to [1], which focuses on the mathematical reasoning task.

[1] GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models
</content>
</thread_object>

</thread>
<thread>
creation date:
1730566528355

thread_type:
official_review

<thread_object>
creator: reviewer_nzax

<content>
summary:
This paper tests SOTA LLMs’ reasoning abilities by testing them with a bunch of variants of the AIW problem.

strengths:
1. The tests are across different SOTA LLMs.

weaknesses:
1. The whole paper is about one type of questions:  “Alice has N brothers and she also has M sisters. How many sisters does Alice’s brother have”. I personally feel like it is hard to judge model’s capabilities based on one type of question alone. Model’s generalization and reasoning abilities are maybe on a spectrum and with only one question, it is hard to tell where the model falls on this spectrum.

2. GPT-4o has superior performance, possible suggesting this might have to do with model size or training?  

3. I don’t consider “female boost” as totally redundant information. For one thing, if you are testing the model’s reasoning abilities, it should disentangle model’s syntactic understanding as a separate thing. “She” as a sole indicator of Alice being a female is more a syntactic problem, which shouldn’t part of model’s burden if one’s goal is simply to test reasoning abilities. 

4. Personally, I feel like this paper adds nothing significantly interesting to the existing discussion on whether LLM can reason or generalize. For one thing, a pure test on reasoning should not rely much on extra knowledge. The test in the paper (AIW) needs model’s understanding of “she” as Alice (syntactic) and basic family structure (external knowledge). The actual reasoning, on the other hand, is in my opinion, perhaps not the main bottleneck. This is also supported in the paper where “female boost” variants can improve performance.

questions:
1. For figure 1, what do the numbers like 55, 56, 63, 69 mean?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730658811241

thread_type:
official_review

<thread_object>
creator: reviewer_3myn

<content>
summary:
The authors discovered a surprisingly simple and concise problem that makes most LLMs, including state-of-the-art models, fail. The problem, namely AIW, belongs to the class of basic reasoning problems where humans excel. The authors show that LLMs fail on the vanilla version of the AIW problem and semantically equivalent variations; techniques such as chain-of-thoughts or more advanced prompting methods fail at mitigating such issues.

strengths:
I am really surprised that such a simple example makes LLMs fail, so I consider this discovery valuable. I tried to prompt a few models, and I agree with the authors that this problem is indeed hard even for models like GPT-4 (though I tried a few times with GPT-o1-preview, and it correctly solves the task, but it surprisingly fails to reply with nonsense when we input negative numbers!).

The article is well written and easy to follow, and the AIW results are robust enough to support the claim that most LLMs fail on such problems.

weaknesses:
The authors did not try to add illustrations (e.g., k-shot) to mitigate the issue. I tried to add an illustration, but some models still failed. That analysis would add value to the work. While more expensive, fine-tuning a small model would also add value to the consistency of the case study they present.

Beyond that, my biggest concern is that the authors tried and reported only one example of failure across multiple models. 
To make an analogy with the adversarial robustness literature, this is equivalent to finding a single adversarial example in computer vision that makes most models misclassify an input (an example of a ‘universal trigger’).
The paper lacks a consistent analysis of other examples, and, in this form, it reduces the contribution to an exciting yet anecdotal showcase of failure. Plenty of articles show failure cases of LLMs on *many* examples and variations; one very popular is [1].

Furthermore, the authors do not provide a solution or a tentative plan to mitigate the problem (but that is not necessarily a limitation).
The authors do not give a reasonable rationale behind why LLMs fail on the AIW problem. Interestingly, it points out that LLMs fail on the AIW Light Arithmetic Total Girls, but that is another anecdotal showcase of failure that does not tell us much about why LLMs fail on such simple problems.
For example, a model that solves the vanilla AIW would possibly “create” a graphical representation of Alice and her brothers and sisters; then, the LLM can count the number of edges from one of the brothers connected to all the sisters. That means a sufficient (but not necessary) condition to solve the problem is being able to perform 2-hops reasoning and counting (from one of the brothers to all the sisters). LLMs seem to lack such capability.

[1] Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks, Tomer Ullam.

questions:
1) What happens when N and M grow larger than 7, and why do they decide to set that as the upper bound on such variables? 

2) Have the authors tried with floating and/or negative values for N and M? If a model still replies with the consistent (yet wrong) reasoning, that is a strong hint a model does not understand the task under consideration (i.e., it cannot connect numerical and graphical reasoning with family relationships).

3) Have the authors tried to ask the model to generate a graphical representation of Alice’s family and then solve the task?

4) Why do the authors focus on a single example and not on a consistent range of variations of similar problems?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730677692186

thread_type:
official_review

<thread_object>
creator: reviewer_7c6d

<content>
summary:
The paper demonstrates that LLMs struggle with generalisation on a simple problem. Specifically the authors construct a question and various variations thereof of a `simple' family relationship question. (i.e. in short the question is: Alice has N brother and M sisters. How many sisters does Alice's brother have?) The paper demonstrates that small variations of this question break state-of-the-art LLMs, even across different prompting techniques.

strengths:
The strengths of the paper:
1. Good number of experiments specific to the question demonstrated by the authors
2. Careful analysis across a wide variety of models.
3. Interesting finding that breaks models.

weaknesses:
The weakness of the paper:
1. The study while interesting and definitely highlights a problems with modern LLMs is quite limited by the actual test set (basically being based on a single question and variations thereof).
2. The study is quite limited into the actual limitations of the model. 

Concretely, although many models were run on this small dataset (and it is understandable that so many models can only be run on smaller datasets [reasonably]), the contribution is quite limited regardless. **Finding specific phrasings that break a model is very common** to most problems and to most people that have done prompt engineering.

Furthermore, the actual analysis while removes high-level doubts in the approach (such as the female boost, or the control question) do not provide deeper insights into what might be going on. Very interesting work in this regard would be "Physics of Language Models", which provides excellent analysis of how model perform and why they generalise poorly. https://physics.allen-zhu.com/

Generally, your work is very interesting and should be pursued further. Well done, however, in terms of research contribution it requires more interesting datasets (than single examples that work poorly and then others that work well, as mentioned earlier this is very common for most tasks). Also, your analysis should be much more detailed in terms of how and why models perform poorly or well on these tasks. (Again, the Physics of Language Models is an amazing work (not ours, unfortunately ;)).

questions:
Some questions that could help you with you research:
1. What specifically do you think can discovered about LLMs using your research (going beyond that LLMs perform poorly on specific examples, but perform better on others?)
2. How could you construct a dataset that measures that specific quality?
3. How could you then propose methods to overcome a fundamental problem that you have identified?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732545963862

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal summary

comment:
We would like to thank all reviewers for the feedback. We provide a summary of the rebuttal, containing points we think are helpful to emphasize for discussion.

# Testing generalization and falsifying SOTA LLMs strong function hypothesis

While reviewers stressed the clarity of the idea and extensive experiments in our work, we would like to clarify a potential misconception wrt. to the work that we think is repetitively echoing through the feedback. This misconception concerns the purpose of the AIW problem constructed in our work for testing generalization, relation of this problem to previous anecdotal attempts to induce model breakdowns of various kinds and the role of systematically controlled variations in properly measuring model sensitivity to problem structure preserving perturbations that reveals generalization deficits. 

Main goal of our study was to test the claims of strong function attributed to current SOTA LLMs, specifically regarding strong zero-shot generalization and robust basic reasoning. Such claims are mainly based on standardized benchmarks (MMLU, HellaSwag, GSM8k, MATH, etc) where current SOTA LLMs obtain high scores. We were curious whether these claims can be put to test using much simpler, more minimalistic problems, as problems used in standardized benchmarks have often formulations overloaded with various additional context and knowledge. We aimed for a problem as simple as possible though still useful for measuring generalization of SOTA LLMs. 

AIW problem, which has a short, unambiguous natural language formulation, was the outcome of this effort (Fig. A https://hackmd.io/_uploads/HJULH2IG1l.png;  Suppl. Tab. 2). Despite its simplicity, we observed surprisingly low correct response rates averaged across various conditions for all SOTA LLMs, including the most advanced ones like GPT-4/4o or Claude 3 Opus (Fig. 1,  Suppl. Fig. 8). The scores were substantially lower than the ones SOTA LLMs obtain on benchmarks with seemingly more complex problems, already posing an intriguing discrepancy to standardized benchmark measurements.  

More importantly however, introducing controlled natural variations into AIW problem template by instantiating numbers N,M involved in its formulation, we created a procedure that measures model sensitivity to perturbations that preserve problem structure and difficulty. This allowed us to probe models’ generalization - should it be intact, model would be capable of handling the problem equally well across variations. This is in contrast to standardized benchmarks, where problems are static and there is no default way to measure model sensitivity to problem variations. Using this procedure, we observed strong fluctuations across problem variations (Overview: Fig. K https://hackmd.io/_uploads/rkPA9H-myg.png; Fig. 2, 6). This is surprising not only because the problem is simple, but also because AIW variations do not change problem structure or its difficulty, being just instantiations of different numbers natural to the problem. Puzzling is why there should be any difference in performance across AIW variations at all. Given the variations should be actually irrelevant for problem handling, expected would be either struggling equally to solve any of the variations or handling them equally successful - assuming generalization is intact. Thus, observing such strong fluctuations in all models that manage to get significant non-zero average correct response rates, we can conclude that models are not able to properly infer the same underlying problem structure which is behind each AIW variation, which, given simplicity of the posed problem, points to severe generalization deficits. 

We think it can be helpful to imagine how humans would perform on such AIW variations, to stress what the observed breakdown of model robustness and overall low average correct response rates mean. Having a test with average human probands confronted with the AIW variations over many trials (we do > 30 trials per variation to compute correct response rates)

> Variation 1: Alice has 3 brothers and she also has 6 sisters. [Correct answer: 7 ] \
> Variation 2: Alice has 2 sisters and she also has 4 brothers. [Correct answer: 3 ] \
> Variation 3: Alice has 4 sisters and she also has 1 brother. [Correct answer: 5 ] \
> Variation 4: Alice has 4 brothers and she also has 1 sister. [Correct answer: 2 ] \
> \
> How many sisters does Alice’s brother have?

we think it would be highly shocking if obtained statistics would reveal performance across variations varying wildly between very high and very low rates, the way we observe it for SOTA LLMs. We expect humans to handle any variation equally well, as we expect generalization and basic reasoning to be intact. It therefore should be equally shocking to observe SOTA LLMs breakdown in such a simple scenario, given the claims of robust graduate level problem solving put forward for models of GPT-4 and Claude 3 Opus class.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733095706469

thread_type:
official_comment

<thread_object>
creator: reviewer_7c6d

<content>
title:
Comment after Reviews

comment:
Overall, this work is interesting and has good suggestions. 

Our last comment as reviewer summarises our proposal how to improve this work. Specifically, extending the dataset to make it more robust to future changes (and we do not mean massive expansion, however, meaningful enough for this work to be valid next year).
</content>
</thread_object>

</thread>
<thread>
creation date:
1733229613647

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear reviewers,

we would like to thank you all again for time and involvement in the discussions.

As the discussion time nears its end, we would like to encourage you to go through discussions, and should those be insightful and resolve some of the concerns, to consider reflecting this in the scores.

We would like to highlight again additional experiments performed in the rebuttal period upon reviewers' request:

- Experiments with modified problem templates that differ from AIW Original. For instance, we either introduce Alice and Bob as two entities in the problem structure to deal with, or we replace brothers and sisters entities with male/female friends, abandoning family specific frame. Using same experimental procedure to create variations of these problem versions, we observe the same pattern as for the AIW original, especially the strong fluctuations across variations, confirming the existence of the same generalization deficits for further problem examples (Fig. I https://hackmd.io/_uploads/BJ1nqj3MJx.png)

- An experiment concerning an AIW version where numbers for brothers and sisters are instantiated to be in an exaggerated range not realistic for a typical family scenario. In this AIW version, offset 60 is added to numbers in AIW original problem. We observe the same pattern as in AIW original (Fig. H https://hackmd.io/_uploads/H1sLdj2Mye.png) - low correct response rates and strong fluctuations across variations. We also see slightly lower correct response rates on average compared to AIW original. This might point to further generalization deficits becoming apparent when dealing with numbers outside the expected problem specific range, despite problem structure left unchanged.

- An illustrative example of a debunking procedure using the recent case of NuminaMath-7B that was ranked 1st at the recent AIMO competition, solving 29/50 private set problems of olympiad math level (https://huggingface.co/AI-MO/NuminaMath-7B-TIR). Based on that evals, the claim was widely put forward that the model is capable of solving high school olympiad math problems (https://www.aimodels.fyi/models/huggingFace/numinamath-7b-tir-ai-mo). AIW problem has average elementary school level and does not require any advanced math knowledge. We tested NuminaMath-7B on AIW and observed a strong collapse of this model on AIW problem, with correct response rates close to 0 across AIW variations 1-4. Using AIW Light control problems, we can also see that NuminaMath-7B can handle all the low level operations and knowledge required to deal with family structure, ruling out that those are the issues. Using the AIW problem setting, we thus can contradict the strong claim of being capable to robustly deal with math problems (Fig F, https://hackmd.io/_uploads/SybG2hqz1x.png). Especially, breakdown in such a simple setting rules out that model will be able to deal robustly with olympiad level math tasks, debunking the claim and raising further questions about AIMO benchmark procedure used to measure model performance.

For the collected data for this debunking experiment, see anonymous repo https://anonymous.4open.science/r/AITW_anonymous-69A6/collected_responses/raw_data_inspection/NuminaMath-7B_AIW_versions_problem_set_checked.json

We hope this together amounts to convincing evidence that experimental setup we describe in our work is useful for community as reproducible measurement method for systematically probing models' generalization using structure  preserving variations of simple problem setting. The method is also useful for falsification of the strong function hypothesis, to debunk overblown claims that rely on benchmarks which overlook such clear deficits as revealed by the AIW problem and its variations. We also hope that study provides impulses and clear roadmap to create novel benchmarks that use problem structure \& difficulty preserving perturbations to stress test model generalization systematically, so that such deficits do not remain hidden, also allowing for measurable progress in improving model generalization capability.
</content>
</thread_object>

</thread>


Metareview:
metareview:
Summary: The paper investigates the generalization and reasoning deficits in state-of-the-art Large Language Models (LLMs) by employing a simple, common-sense reasoning problem called the "Alice in Wonderland" (AIW) problem. This problem reveals severe shortcomings in generalization and reasoning among advanced models like GPT-4 and Claude 3 Opus, even under variations of the problem that do not alter its difficulty or structure. The authors highlight the discrepancy between the claims of robust reasoning capabilities made by standardized benchmarks and the observed failures on simple tasks. Additionally, the paper introduces systematic variations and control tests to isolate the issue and provide evidence of fundamental deficits in LLMs’ ability to generalize across minimal perturbations.

Strengths:
- Important problem being studied and key shortcomings of LLMs being exposed
- Code and raw experimental data are made available for validation.

Weakness:
- Lukewarm response from all but one reviewer and the positive reviewer didn't champion the paper
- Limited Scope of Problem: The analysis is focused on a single problem type and its variations, restricting the generalizability of the findings.
- Lack of Deeper Diagnostic Insights: While generalization failures are observed, the paper does not delve into the underlying architectural or training-related causes of these deficits.
- Comparative Benchmarking: Insufficient discussion on why standardized benchmarks fail to detect these limitations and how alternative designs could address this.
- Broader Dataset: Reviewers suggested expanding the dataset to ensure robustness and broader applicability.
- Some formatting and citation issues in the manuscript

Decision: Given the lack of enthusiasm from the reviewers and limited scope, unfortunately, the paper can't be accepted in its current form and addressing all the concerns would warrant another round of reviewing.