Detecting generalization deficits in large language and reasoning models by using natural variations in simple problems
Authors: Marianna Nezhurina
Lucia Cipolina-Kun
Mehdi Cherti
Jenia Jitsev
Venue: Accepted by TMLR ()

Abstract: Large language and reasoning models (LLMs, LRMs) are instances of foundation models exhibiting scaling laws that predict generalization improvement when increasing the pre-training scale. As such, they are supposed to possess strong generalization and therefore transfer robustly across various tasks and conditions in few-show or zero-shot manner. Such claims rely on various standardized benchmarks that should measure core functions like generalization and reasoning, where state-of-the-art (SOTA) models score high. We demonstrate remarkable zero-shot generalization deficit in most SOTA models which claim strong function, including reasoning models like DeepSeek R1 or o1-mini, trained at the largest scales, using a simple, short common sense math problem formulated in concise natural language, easily solvable by humans, which we term Alice in Wonderland, AIW, problem. The deficit manifests in strong performance fluctuations on natural variations in the simple problem template that do not change either problem structure or its difficulty at all. By testing models on further control problems with similar form, we rule out that deficit might be rooted in minor low-level issues like natural language or numbers parsing. In conventional LLMs, we observe strong overconfidence in the wrong solutions, expressed in form of plausible sounding explanation-like confabulations. Many models showing the deficit also collapse close to 0 accuracy on AIW problems, while still exhibiting high scores on various standardized benchmarks. We show how this illusion of strong function might be caused by leakage of test sets into training. For reasoning models, while observing clearly improved performance compared to LLMs, we still see strong fluctuations on problem variations that keep structure and difficulty unchanged. Our observations suggest that current LLMs and LRMs possess generalization deficits that can be detected by controlled structure and difficulty preserving variations in simple problems, in contrast to standardized benchmarks which contain problems of higher difficulty but fail to detect such clear deficits. Code for reproducing experiments in the paper and raw experiments data can be found at https://github.com/LAION-AI/AIW

Published in Transactions on Machine Learning Research (01/2026)
Detecting generalization deficits in large language and rea- soning models by using natural variations in simple problems
Marianna Nezhurinal:244 Lucia Cipolina-Kun Mehdi Cherti Jenia Jitsey
"LAION Juelich Supercomputing Center (JSC), Research Center Juelich (FZJ)
3 School of Electrical and Electronic Engineering, University of Bristol Open-y (Open-Sci) Collective "Corresponding authors: fmm.eezuurina.jjjileee)/ @fz-juelichh.ee jurhh// de, contact&Qlaion.at
Reviewed on OpenReview: https: openreview. net/ forum? id= frA7u7n22
Abstract
Large language and reasoning models (LLMs, LRMs) are instances of foundation models exhibiting scaling laws that predict generalization improvement when increasing the pre- training scale. As such, they are supposed to possess strong generalization and therefore transfer robustly across various tasks and conditions in few-show or zero-shot manner. Such claims rely various standardized benchmarks that should measure core functions like generalization and reasoning, where state-of-the-art (SOTA) models score high. We demonstrate remarkable zero-shot generalization deficit in most SOTA models which claim strong function, including reasoning models like DeepSeek RI or ol-mini, trained at the largest scales, using a simple, short common sense math problem formulated in concise natural language, easily solvable by humans, which we term Alice in Wonderland, AIW, problem. The deficit manifests in strong performance fluctuations on natural variations in the simple problem template that do not change either problem structure its difficulty at all. By testing models on further control problems with similar form, we rule out that deficit might be rooted in minor low-level issues like natural language or numbers parsing, In conventional LLMs, we observe strong overconfidence in the wrong solutions, expressed in form of plausible sounding explanation-like confabulations. Many models showing the deficit also collapse close to 0 accuracy on AIW problems, while still exhibiting high scores on various standardized benchmarks. We show how this illusion of strong function might be caused by leakage of test sets into training. For reasoning models, while observing clearly improved performance compared to LLMs, we still see strong fluctuations problem variations that keep structure and difficulty unchanged. Our observations suggest. that current LLMs and LRMs possess generalization deficits that can be detected by controlled structure and difficulty preserving variations in simple problems, in contrast to standardized benchmarks which contain problems of higher difficulty but fail to detect such clear failure!
1 Introduction
State-of-the-art (SOTA) large language and reasoning models (LLMs, LRMs) are important instances of foundation models. A central property attributed to foundation models is strong, robust zero-shot general- ization, enabling them to solve problems not present during the training. Evidence of strong generalization rely mainly on high scores obtained in various standardized benchmarks. Such benchmarks contain sets of problems that reach up to advanced high school (MMLU [Hendrycks et al. (2020). GSM8k Cobbe et al.
(2021), MATH and MATH-500 Hendrycks al. (2021)), olympiad (AIME24/2) AIME 2024 or graduate (GPQA-Diamonl Rein et. al. (2224) difficulty levels.
!Code reproducing experiments paper and experiments can found ATW repo

---

Published in Transactions on Machine Learning Research (01/2026)
AIW Variatins 1-4
1.0
1.0
Veriation Aike has brothers and she also has sisters (Correst answer: 7]
55
57
#wyt, 53
Variation Alice has 2 sisters and she also has 4 brothers (Correct answer: 3]
22: S4
ain 15, 50
WW st
Variation Alice has sisters and she also has 1 brother. [Correct answer: $]
Variation 4; Alkc has brothers and she also has sister. [Correct answer: 2]
0.8
0.8
0.8
#wv3 03
sww. 21
How mony sisters does Alice's brather have?
Correct resporse refe
Correct resporse rota
Correct response
0.6
0.6
frompt types doellwing after main problem description and question abovel:
STANDARD: Solve this problem and provide the final answer in tollowing farm: "aike Answer:
0.4
0.4
0.4
THINKING (V2) Before providing answer to problem, think carefully liep step) and double mistakes, Provide then the final answer in following form: "thtt Answer:"
path the correct solution any
0.2
0.2
0.2
RESTRICTED: To answer the question, DO NOT OUTPUT ANY TEXT EXCEPT following format that contains final "MMM Answer:
0.0
0.0 GPT-4
0.0
GPT-4
GPT-4
Figure 1: AIW problem is simple common sense math problem with short, concise formulation, serving as a minimalist sctting to detect deficits in 0-shot generalization. Problem template is used to create AIW variations 1-4 (left) by instantiating different numbers N, M of brothers and sisters, keeping problem structure and difficulty unchanged. We measure sensitivity of models to those variations, observing strong performance fluctuations. Here on example of GPT-4 (gpt-4-0613) tested with various prompt types STANDARD, THINKING, RESTRICTED, a color per each variation 1-4, executing 60 trials per each variation. Correct response rate varies strongly depending on the variation. F.g., it drops close to 0 for variation 3, while going up to 1 for variation 4. This observation is consistent for different prompt types STANDARD, THINKING and RESTRICTED, showing that observed fluctuations are not due to prompt variation. Strong performance fluctuations natural, structure preserving variations of such simple problem points to severe lack of robustness and generalization deficits. Numbers in the legend are prompt IDs (Suppl. Tab. 3
Recent works however provided evidence that high scores obtained on standardized benchmarks drop significantly if better controlling for contamination of training data by test sets Golchin & Surdeanu (2023); Li& Flanigan (2024). Further works questioned whether standardized benchmarks properfly reflect model function and generalization by pointing out various function failures that were seemingly incompatible with claimed strong capabilities MMtchell] (2023): : Wu et al. (2023); Frieder et al.] (2024). It thus still remains unclear whether scores on standardized benchmarks can be used as generalization error estimates. Instead of signaling strong generalization, high scores on benchmarks might simply reflect the degree of test set leakage into training. Should high scores on standardized benchmarks mislead about actual model generalization capability, it becomes further unclear how to reliably detect generalization deficits and debunk strong function claims in cases where those are based on memorization of test samples instead of proper generalization.
To shed light on this situation, we propose a distinct approach to detecting generalization deficits. We start from definition of a simple problem template that supports natural variations which keep problem structure and difficulty unchanged. We generate then set of problem instances originating from same problem template using such variations. As each problem instance has same structure and difficulty, models that generalize well should exhibit, none or only slight performance differences for each instance. In contrast, performance fluctuations on instances of same problem should reveal models with generalization deficits.
To demonstrate the approach, we create a short, common sense natural language math problem template. The problem (in following Alice Wonderland, AIW problem) template has following form: "Alice has N brothers and she also has M sisters. How many sisters does Alice's 's brother have?". Instantiating natural numbers N,M M < allows natural variations that do not change problem structure and difficulty and thus should not affect ability to solve resulting problem instances. (Fig. D.
When tested on instances of simple AIW problem, we obscrve SOTA LLMs exhibiting strong performance Iluctuations, showing accuracies close to 100% on some, while dropping close to 0 on others, despite all instances being structure and difficulty preserving variations of the same simple problem (Sec. 3.1). The strong fluctuations remain across various prompt types, including chain of thoughts. To rule out that observed sensitivity to variations is rooted in inability to execute robustly various basic operations necessary to handle the problem, we test models on control problems containing same operations as required to solve the original
2

---

Published in Transactions on Machine Learning Research (01/2026)
(Sec. 3.1.1p.
We observe models are able to solve all variations of control problems without performance Iluctuations. This provides evidence that models can handle all the basic operations well, and observed sensitivity is due to inability to correctly infer same problem structure across instances, which is hallmark of failure to generalize.
Confronted with evidence of zero shot generalization breakdown all tested SOTA LLMs exhibit on such a simple problem, we investigate further AIW problem versions to see whether the same phenomenon of model sensitivity to problem structure preserving variations appears consistently. We use same technique of varying numbers in problem templates and observe same strong fluctuations of model performance also other problem versions, obtaining further evidence that the generalization deficit is generic and not unique to original AIW formulation. We also observe SOTA LRMs like 03-mini OpenAl (2025a), ol-mini OpenAl (2024e), DeepSeek-Ri Guo et al. 2025 or various distilled reasoning models exhibiting strong fluctuations on ATW problem versions (Sec. 3.2). Already in models that use simple supervised finetuning (SFT) on small amounts of reasoning traces data unrelated to AIW (c.g, S1.1 32B Muennighoff et al. (2025)), we observe elevated average correct. response rates and thus clearly improved performance compared to conventional LLMs. However, persisting strong fluctuations on problem structure and difficulty preserving variations reveal again lack of robustness and generalization deficit also for this model class.
We observe that many models score high on standardized benchmarks containing problems of higher difficulty, while exhibiting strong performance fluctuations on variations of simple AIW problems that are far below such levels. We hypothesize that this illusion of strong function conveyed by high benchmark scores is partly rooted in test set leakage and poisoning of training data, which distorts generalization measurement. We provide evidence for potential test set leakage by contrasting performance of models like Claude 3.5 Sonnet older (before its appearance) and more recent (after its appearance) AIW problem versions. We accompany the evidence with experiment showing how Llama 3.1 8B goes from zero to strong performance across AIW variations if using AIW problem data for fine-tuning (Sec. 3.1.2p.
In summary, our study makes following contributions: (i) presenting a novel approach to detect generalization deficits by measuring performance fluctuations across natural, difficulty and structure preserving variations of same simple problems (ii) showing SOTA LLMs and LRMs exhibit strong fluctuations on simple AIW problems (iit) designing control problems to ensure the observed issues are due to generalization deficit (iv) showing how discrepancy between high scores standardized benchmarks and observed generalization deficits on AIW problems can be explained by test set leakage, warning against the illusion of strong function mediated by the benchmark scores.
Unlike evaluations that vary problem difficulty and execution length (see eg Shojaee el al. (0255 for
Tower-of-Hanoi problem based setup), introducing additional confounds, our approach preserves problem difficulty, allowing to attribute failures on problem instances clear to generalization deficits. We also study far simpler problems- most AIW versions are elementary-school level, so failures are especially diagnostic.
Our study highlights that novel evaluation procedures and benchmarks are required to properly reflect models' generalization. Strong performance fluctuations exhibited by LLMs and LRMs on variations of simple AIW problems reveal generalization deficits that are incompatible with claims of robust problem solving (ranging up to olympiad or graduate difficulty level) as conveyed by high scores on standardized benchmarks. Using structure and difficulty preserving variations of simple problem templates offers thus an effective way to test for model sensitivity and generalization failures that are not detected by standardized benchmarks.
2 Methods & Experiment Setup
2.1 Simple common sense reasoning problems with difficulty & structure preserving variations
AIW Problem. To measure models' sensitivity to problem irrelevant variations and thus probe the zero-shot generalization, we use following problem template: "Alice has N brothers and she also has M sisters. How many sisters does Alice's 's brother have?" The problem has a simple common sense solution which assumes all sisters and brothers in the problem setting share the same parents. The correct response C number of sisters is easily obtained by calculating C M+1 (Alice and her sisters), which gives the number
3

---

Published in Transactions on Machine Learning Research (01/2026)
of sisters Alice's brother has. To create problem variations, we choose to vary natural numbers N, N.M = 7, restricting number variations to a simple common sense setting. This way we obtain ATW variations 1-4 (see Suppl. Tab. 3] that all pose same problem using variations irrelevant for problem solving, leaving its difficulty and structure unchanged. See Suppl. Sec. E for the resulting variations. We use prompp types, RESTRICTED, STANDARD and THINKING, to ensure we measure models across various prompt formulations and check the observations hold independent of employed prompt type (sce also Sec. 2.2 and
Suppl. Sec. B
2.1.1 Control AIW Light problems
To control for models failures not coming from inability to execute basic problem specific operations, we make various versions of AIW problem AIW Light Family, AIW Light Arithmetic Siblings and AIW Light Arithmetic Total Girls that are designed to lest various operations. Staying similar to AIW original, control problems test basic family relations structure handling, identifying relevant entities, executing arithmetic operations and in general, natural language and numbers parsing required for handling the original AIW problem.
The AIW Light problems keep problem template close to the original, changing only the final question part such that the posed modificd question tests particular operations. The shared common template part taken from the original AIW problem: "Alice has N brothers and she also has M sisters. The variations 1-4 are again created in the same way like in AIW original by varying natural numbers of brothers and sisters, while ensuring that the natural numbers for final correct answers in AIW original and AIW Light are matched across variations 1-4 (see also Suppl. Sec. B.I.
AIW Light. Family. ATW Light Family continues common template with following question: "How many brothers does Alice's 's sister have?". To solve the problem, reporting already given number of brothers is sufficient the correct answer is C N. This requires only basic grasping of relational family structure (understanding entity 'Alice's sister', binding female attribute to Alice and realizing Alice and her sisters share same brothers). It does not require execution of any arithmetic or set operations, in contrast to AIW original. Should the issues with solving AIW original be rooted in handling basic family structure, we should see models also failing here. For variations 1-4, see Suppl. Tab. 5
AIW Light Arithmetic Siblings. AIW Light Arithmetic Siblings continues common template with following question: "How many siblings does Alice have?" To solve the problem, summming up already given numbers of brothers and sisters is sufficient the correct answer is C'= N + M. This requires basic grasping of relational family structure (realizing Alice's siblings are her sisters and brothers) and selection and exccution of clementary arithmetic sum operation. In contrast to AIW original, it does not require execution of set operations nor binding sex attribute to Alice to properly assign her to correct sets. Should the issues with solving AIW original be rooted in selection and execution of elementary arithmetic operations in family frame, we should see models also failing here. For variations 1-4, see Suppl. Tab.
AIW Light Arithmetic Total Girls. AIW Light Arithmetic Total Girls continues common template with following question: "How many girls are there in total?". To solve the problem, it is necessary to bind female attribute to Alice via the pronoun 'she', to assign correct female attributes to the sisters and to execute the correct arithmetic sum operation adding all the obtained girls the correct answer is C = M + 1 (also coinciding with correct answer for AIW problem). This requires basic grasping of family structure (realizing who are the girls in the family) and selection and execution of elementary arithmetic sum operation. In contrast to AIW original, it does not require execution of set operations to properly assign Alice to sisters set. Should the issues with solving AIW original be rooted in binding correct sex attributes or counting total members of particular sex in family frame given its structure, we should see models also failing here. For variations 1-4, see Suppl. Tab. 6
2.1.2 AIW problem versions with similar structure
To see whether observations are consistent for other problems of related kind, we construct further two problem versions, AIW Extended (AIW Ext) and AIW Friends that have similar problem structure and difficulty as AIW original. For both problems, the correct answer still has the form C = M 1, where M
4

---

Published in Transactions on Machine Learning Research (01/2026)
depends on natural number variation within the problem template. We employ both problem versions further test. for generalization, as it allows us to have both variations within a problem version (natural numbers variations) and also check model sensitivity to changes across problem templates, while still keeping highly similar problem structure and similar difficulty across problems (in contrast to AIW Light control problems which are designed to test. low level operations involved, such that their templates pose problems of lower difficulty than AIW original).
AIW extended (AIW Ext.) In this problem version, two entities, Alice and Bob are employed as sister and brother, otherwise keeping the problem template and structure close to the AIW original. See Suppl. Sec. B.2 for the full problem formulation with variations 1-6.
AIW Friends In this problem version, the family frame setting is abandoned. Instead, we use male and female friends in problem formulation. This way, while we vary the problem formulation, the problem structure and difficulty are still highly similar AIW original (brothers and sisters are male and female siblings). We use an additional statement to ensure there is no common sense ambiguity in this problem version. See Suppl. Sec. B.2 for the full problem formulation with variations 1-6.
2.1.3 Harder AIW problem versions
To test. what happens if the simple AIW problems are further extended lowards a substantially harder difficulty level, providing further challenge for the tested models, we construct problem versions of higher difficulty increasing mumber of relations and entities, while still koeping same rather simple relational logic structure. We formulate the problems such that despite their increased difficulty, the correct answer still has the form C²= M + I, with M depending on given natural number instantiation within the problem template. While the solution to +IW+ and AIW Colleague Circles problem is harder to obtain than the solution to AIW original with its much simpler structure, the problem solving capabilities required to cope with these problems are far below from olympiad or graduate difficulty level, on which many recent LLMss and LRMs are claimed to perform successfully.
+IW+ problem. We constructed an +IW+ problem that features additional hierarchy and distractors when describing family structure that includes various entities like brothers, sisters, cousins, uncles and aunts. It has also a substantially longer template than AIW original. See Suppl. Sec B.2.1 for full formulation and its variations.
AIW Colleague Circles. This problem deals with work colleagues setting, and the difficulty is increased by introducing circles of malc and female collcagues with various relations that has to be handled to properly answer the question. The template is also longer than AIW original. See Suppl. See B.2.1 for full formulation and its variations.
2.2 Prompt types and response parsing
It is well known that so-called prompt engineering can heavily influence the model behavior and model response quality Arora et al. (2022); Wei et al. (2022); ; White et al. (023)). To check that our observations reflect model sensitivity to controlled problem structure preserving variations in same manner independent of particular prompt type, we employed 3 various prompt types to provide model's input: STANDARD (prompt with instruction to format final answer output as a natural number), THINKING (prompt that in addition encourages thinking in spirit of CоT) and RESTRICTED (prompt with instruction to output nothing else but final answer as a natural number). THINKING v2 prompt type is a minor variation of THINKING type that just. adds "step by step" after already existing 'think carefully' phrasing (control experiments show that THINKING and THINKING v2 are equivalent in terms of observed performance, so we use both interchangeably). STANDARD, THINKING and THINKING v2 prompt types allow models to generate any output length and thus not restricting compute before delivering the final answer. RESTRICTED is used as control with restricted output to measure baseline of model performance when compute for producing final answer is limited (Suppl. Tab.
Parsing model responses. To perform evaluations of model performance, is necessary to parse and extract the final answer from the responses provided by the models. Each input to the model is combination
5

---

Published in Transactions on Machine Learning Research (01/2026)
of AIW problem variation, followed by one of prompt types as described before. To keep the parsing procedure simple, we add to each problem prompt following output format instruction: 'provide the final answer in following form: #### Answer: We observed that all models we have chosen to test were able to follow such an instruction, providing a response that could be casily parsod. We also ran control experiments without such formatting rnstruction in the problem formulation, ensuring that behavior does not. depend on it.
2.3 Selecting models for evaluation and conducting experiments
We are interested in testing current statc-of-the-art models that claim strong function, especially in gen- eralization and reasoning, backed up by high scores shown on standardized benchmarks that are assumed to measure generalization and reasoning capabilities to solve problems. We therefore select models widely known and used in the ML community that also appear in the top rankings of the popular LLM leaderboards, like openLLM leaderboard by HuggingFace or ELO leaderboard by LMsys. We provide the overview of the selected models in Suppl. Tab. P and list in Suppl. Tab. 13 the corresponding standardized benchmarks where they obtain strong scores.
We expose selected SOTA LLMs and LRMs, inchuding advanced models at largest scales (see Suppl. Tab. 2
and further AIW versions (Suppl. Sec. B.2), using different prompt types as described above. For each combination of model, AIW problem variation and prompt lype, at least 30 trials are collected to compute correct respouse
to AIW problem variations 1-4 (Suppl. Tab. 3), AIW Light control problems (Suppl. Tab. 4 5
rates, Suppl. Fig. 27 For details on correct response rates estimation procedure, see Suppl. Sec. A.2
We use hosting platforms that offer API access local deployment via vLLM Kwon et al. 2024 for testing the models, and aulomatize the procedure by scripting the roulines necessary to prompt models with our prompts set. The routines are simple and can be used by anybody with access to the APIs (we used LiteLLM and Al for our experiments) or to locally hosted models to reproduce and verify our results. We protocol all the data from interactions with the models to enable community checking. We release all the collected raw response data, correct response rates estimates and routines used to conduct experiments as open-source for reproducibility and further usage.
3 Results
3.1
Breakdown of SOTA LLMs on the simple AIW problem
AIW reveals severe generalization and reasoning deficits in SOTA LLMs. Following the procedures deseribed in See we expose the selected modols that claim strong function and reasoning capabilitics (Suppl. Tab. 2) and measure their correct response rale performance across and for each AIW variations 1-4 using various prompt types, executing > 30 trials for each combination (see also Suppl. Tab. 3 and Suppl. Fig. 27 The results suggest that confronted with the AIW problem, models suffer a severe function breakdown. This breakdown has two main manifestations:
1. Low correct response rates. Despite evident problem's 's simplicity, the majority of models stay well below correct response rate of = 0.4. We sumnarize the main results in the Fig. 2 Many models are not able to deliver a single correct response (Suppl. Tab. 13 Suppl. Fig. G.. The only major exceptions from the observation of very low correct response rates are the largest scale closed models GPT-40/4 and Claude Opus. These models obtain correct response rates well above p 0.4, lcaving the remaining large and smaller scales open-weights (e.g., Hiistral/MMxtrn Llama 2/3, Qwen 2.5, Command R+, and Dbrx Instruct) and closed- weights models (e.g., Gemini Pro, Mistral Large) far behind. Many of the models that claim high scores on standardized benchmarks, show very low correct response rates close to 0, eg. Llama-3-8B, Mixtral-8x22BB, DBRX instruct, or exhibit even complete breakdown on AIW with correct response rate of zero across all variations, eg Command R+ or Qwen1.5-722 (Suppl. Tab. 13
The results presented the Fig. [ show estimates for correct response rates averaged across STANDARD, THINKING and THINKING v2 prompt types (Suppl. Tab. 3 prompt IDs provided for reproducibilit; Suppl.
Fig. 6 with more models scoring 0). RESTRICTED prompt type was used further control, providing
6

---

Published in Transactions on Machine Learning Research (01/2026)
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
gpt-4o-2024-05-13
Srudde--opus-20240022
gpt-4-0613
(wen2.5-7b-instrrn
qwen2.5-72b-instrun
gpt-fo-mini2222-00718
tiwen2.5-14biinsrruce
deepseek-v3
Vertetion -Ake han brothers and she has sameen [Forect amower??
meta-llamaa3.-770-instruutt
Voriation Alli hos p50ers and the also has 1 brotteer [Sorect answer Variationn Alice foo brothers and she also hao clster Coonect anower
llama-2-7bb-caat
Ilamm-2-7b-chat
Hove suters doss Alecl brosker nove?
qwenn2.5-32-innstruct
sermtcess I-t. STAADNED prowat
werttios S-L. THESS30 prcept
ABW variation 1-t, ThBecII vc proegt
llama-2-13b-chat
1.0
1.0
1.0
claude-3sommee-222440222
20
mistral-lare-22402
0.8
0.8
0.8
dbrx-instruct
open-mixtral-8x22b
twwnn255-bbinnttuuct
0.5
0.6
meta-llama-3.--8b-intttt
claude-3-haile
2hudee33hailu-22240307
0.4
gpt-3.5-turbo-0125
open-mistral-7b
0.2
0.2
open-mixtraa--x7b
qwen2.5-11. 5-1..5bbinsstruct
command-r-plus
0.0 GP1-
GPT-4
0.0
Figure 2: Severe deficits of conventional LLMs on simple AIW problem. (main) Models correct response rale, average over STANDARD, THINKING, THINKING v2 prompt types and AIW variations 1-4. (inlay) Strong fluctuations on AIW variations 1-4, despite problem structure and difficulty remaining entirely unchanged across variations. Overall correct response rate averaged across variations does not reveal these fluctuations (shown on example of GPT-4). Numbers the legend are prompt IDs (Suppl. Tab. 3
baseline for performance with restricted compute by forcing models into short outputs (see Suppl. Sec. C and Suppl. Fig. 7. The 3 models crossing p = 0.4 as better performing group are the GPT-4o (p = 0.589), Claude 3 Opus (p- = 0.507) and GPT-4 (p- 0.4803). Other open-weights and closed models are lagging far behind. For better performers, when inspecting the responses with correct final answers, we see also correct reasoning backing up the final answers. In the poor performers, among the few responses with a correct final answer we encounter responses where final answer turns out to be an accident of exccuting entirely wrong reasoning. In such cases, various mistakes lead coincidentally to the final output number corresponding to the right answer. Such responses are encountered in models with low correct performance rates (p 0.4) (sce Suppl. Sec. D for response examples, and for raw data), and we correct via manual inspection the status of correct response for such cases.
2. Strong performance fluctuations across irrelevant AIW problem variations. Importantly, we observe strong fluctuation of correct response rates across AIW variations 1-4 as introduced in Sec. 2 Such fluctuations strongly affect better performers with higher average correct response rates like GPT-4/40 and Claude 3 Opus (as poor performers have often correct response rates across all variations close to 0, that room for fluctuations exist that very low performance level). As shown in the Fig. 2 (inlay) for AIW original formulation and Fig. 3 for various AIW versions (A) original, (B) AIW Ext and (C) AIW Friends, the correct response rates can fluctuate between being close to 1 to being close to O, depending on AIW variation. Remarkable is that such fluctuations appear despite AIW variations being all instances of the very same simple problem, as changes in numbers used across AIW variations do not change either the problem structure or its difficulty at all. This lack of robustness in simple problem setting hints on severe deficits generalization. The strong fluctuations across variations appear independent of employed prompt types (Fig. 2 (inlay), Fig. [., while correct response rate averaged across all variations also varies across prompt types, showing in addition expected prompt type dependency (Suppl. Fig.
3.1.1 Control experiments using AIW Light problems
we execute control experiments using AIW Light problems (Fig. 3 E F) to rule out that observed deficits are caused by low level issues like natural language/
As outlined in Sec. 2.1
trnguage/numbers numbers parsing related to
7

---

Published in Transactions on Machine Learning Research (01/2026)
AIW Var iniloms Original and AEW Light Contral
lemplkte Wiarodhers and has Malsters
0.8
UetrenssAae salpe un ceatee and the sas bac arocher Loome niee
resprtess
0.6
/fors betther
sEN Stiginal OLU breakdownd
How many samtn alres Alicc ovetter tovertorred C-M 11 (A)
0.4
New muayy beneterr dieer Alieet siner tower/omme:: C 01
ADN Light Contral B.L succeno)
How rovey o sillegg (hrs skite iiceet
(Correat C- MT (P)
0.23
How monyy gerb are shice rosal?
[corread C MM11 11 (0)
Prompp tyre
0..0
09:44
Dustea Gee175 Me Usve 1.1 166 Esno 3D GAN775
this probium, think carclully atsp and double shock the the cornect for maytakk. Provice then the roud arswer in Follow ng fore Heger Ammwwer:
Before
3.0
AMV and dnrter Oer wraseee
Wendon dit thi fe lerth are Corrct (ower1)
E
dovid e Mot ster and hon leee on Coree rover 33
coup sushwe
0:0
59143
4F14
LM6620
USTA A1 180
0.2
697.36
6954
Codd-30 Owress 100 Lans 111 300 100 50 75
Caldedas
c35
3.0g
fommsi Frence Vatteder
0.3
644
G
044
0:2
625
6PH-
Coste30 Owird5 72t 23 U1 Gmo723 TO 02at033
Figure 3: Strong fluctuations on problem variations (a color per each variation 1-4) observed on AIW original (A) and two further versions, AIW Extended (AIW Ext (B) and AIW Friends (C), using THINKING v2 prompt type. On all AIW versions, better performers (GPT-4/4o, Claude 3 Opus) show strong fluctuations, getting higher correct response rates (p > 0.3) on some variations, while dropping strongly close to zero on others, despite variations 1-4 leaving problem structure and difficulty unchanged, only instantiating different. numbers N, M into problem templates. AIW Ext (B) and AIW Friends (C) show overall altered correct response rales (eg, for GPT-4/4o, Claude 3 Opus) compared are common phenomenon across the problem versions, hinting on generic generalization deficits that are independent of specific problem formulation. Differences in performance between the problem versions provide addit ional evidence for lack of model robustness, as problem structure and difficulty is highly similar between parsing) and lack
AIW original (A)). Strong fluctuations
the versions. Control experiments. We rule out low-level issues of specific knowledge by AIW Light control experiments (E) Family (F) Total Siblings and (G) Total Girls. Across all AIW Light control problems, strong fluctuations disappear. Models collapsing on AIW variations close to 0 obtain high correct response rates control. This proves that handling family relations, binding sex attributes, parsing numbers, handling elementary arithmetics, etc, are all intact and not the cause for breakdown on AIW variations.
tokenization or by inability to deal with specific problem setting, eg handling family relations. In all following experiments, for each ATW Light variation, 60 trials were executed to estimate correct response rate and its variance. We observe that models that undergo collapse close to 0 on AIW original variations are able to perform well on variations of all AIW Light problems, such that strong fluctuations observed on AIW original mostly disappear. Strong performance on control problems show that models can handle family structure and bind attributes in AIW Light Family (Fig. 3 E), and select and execute necessary arithmetic operation to compute quantities following correct inference of family structure in AIW Total Siblings (Fig. 3 F) and AIW Total Girls (Fig. 3 G). This demonstrates that handling various basic operations necessary to solve AIW original problem is not the issue.
3.1.2 Debunking strong function illusion
After public release of AIW original problem (May 2024), number of models claiming strong function were released, including Claude 3.5 Sonnet (June 2024), also confirmed by standardized benchmarks. We show
8

---

Published in Transactions on Machine Learning Research (01/2026)
A
a/v
B
C
206
D
1te11 213
shrol
oo vL. 264
350 10. 157
alw 158
240
wall 270
200
06n V2. 204
on 187
2M.12 236
12
243
1.0
1.0
1.3
L4
0.8
0.8
0.8
0.0
0.3
0.1
0.
0.6
0.8
0.6
0.4
0.4
0.4
0.4
0.1
0.4
0.2
02
0.0
Claude-3.5 S
0.0
Claude-3.5$ S
0.0
Caude-3333 5
0.0
Lamo 3.1.b
Ulme
(Sara AL PA
Figure 4: Debunking strong function illusion. Claude 3.5 Sonnet Risc (A), (B) and Fall (C), and its possible explanation via Lest set contamination using Llama 3.1 8B fine tuning experiment. While correct response rates go up close to 1 on variations 1-4 for (A) AIW original and also for (B) AIW Bob version, strong breakdown is observed on AIW Ext (C). Strong fluctuation occurs despite ATW Ext being highly similar to AIW original, revealing severe generalization deficit. Possible explanation lies in data contamination that can create illusion of strong function, as shown by the experiment with fine-tuning Llama 3.1 8B. In its original state, the performance samples containing AIW original variations 1-4, mixed with Alpaca instruction data, correct response rates shoot high up for ATW original (E) and ATW Bob version (F). On ATW Ext (G) however performance collapses, again revealing generalization deficit and resembling observations for Claude 3.5 Sonnet.
AIW original variations 1-4 is close to 0 (D). After fine-tuning on generated
here how such claims can be misleading by testing the model on AIW original and slightly modified AIW versions (Sec. 2.1.2p. Tn Fig. 4 we see that while the model handles well both ATW original (Fig. 4 A) and version where Alice is replaced with Bob (Fig. B), the performance drops strongly AIW Ext featuring both Alice and Bob (Fig. 4 C). AIW Ext and original AIW have highly similar problem structure, however ATW Ext was not revealed before Claude 3.5 Sonnet release. We hypothesize thus that strikingly different behavior on AIW Ext roots in test set leakage of AIW original problems into pro- or posttraining data.
We provide supporting evidence that test set leakage might indeed cause observed illusion of strong function by fine-tuning open weights Llama 3.1 8B on the data containing various problem instances generated from observe strong performance improvement of the fine-tuned Llama 3.1 8B on AIW original. It scores close to 1 on most of AIW variations, while still poorly performing on AIW Ext (Fig. 4 G). Models like Claude 3.5 Sonnet report high scores on standardized benchmarks, however strong function claims cannot be dcrived from those, as we see from strong collapse and performance fluctuations on variations of rather simple problems as AIW Ext.
AIW original problem template. As evident in Fig. E F similar picture emerges
3.1.3 Further relevant observations.
1. Standardized benchmarks failure. We observe failure of various standardized benchmarks (GSM8K, MATH, MMLU, ARC, etc) to properly reflect generalization and basic reasoning skills of SOTA LLMs by noting significant disparity between the low models' performance on the AIW problem and the high benchmark scores (sce Suppl. Sec. C.5p 2. Dominance of wrong responses We mcasure distribution of natural numbers responses on output, showing that for AIW variations with low correct response rate, peaks are on wrong answers, excluding majority voting methods as a fix (Suppl. Sec. C.4). 3. Confubulations and overconfident tone We observe that wrong responses are often accompanied by persuasive explanation-like confabulations and overconfident tone about correctness of the wrong solutions provided by the models, which can further mislead model users (Suppl. Sec. E
9

---

Published in Transactions on Machine Learning Research (01/2020)
mendican 2 Minddy maae Sioed Eoreed aesee
Aer on hur wat Hondy (hrclla female blmnts Daeeci mone 4:
0.8
Now whwreryyyiitttyyy pionh wee voau touud Afe and
0.-
4/ vl, 560
DeesSenckl 6728
UMD
4011-00
QwV--223
OF-Qwer-222
Ric-amed
Al A1-Qwem-322
Deepdeec 625a Lanaal 1606
62 35-min--225-01
opectt www.rtaiienn owen
Supprree
CRAS wwr.seaaelecdeee. B4 Bcamade
Figure 5: Reasoning models show severe deficits in generalization, while outperforming conventional LLMs. (A) Strong fluctuations of correct response rates on variations of ATW Friends problem (a color per each variation 1-6) exhibited by reasoning models. The fluctuations affect to same extent SFT only (eg S1.1 32B, LIMO-32B, Open Thinker-32BB and SFT+RL (eg QwQ 32B, DeepSock-R1 671B) reasoning models. Conventional LLMs (DeepSeek-v3 671B, Llama 3.1 405B) are strongly outperformed by reasoning models. For reference, ol o1-preview is shown, which has much smaller fluctuations and higher overall correct response rates, where data contamination cannot be excluded as cause for the observed performance. Numbers in the legend are prompt IDs, see AIW repo (B) Albeit suffering from strong fluctuations, larger-scale reasoning models set themselves apart from the conventional language models. Shown are correct response rates averaged across all variations 1-6 of AIW Friends, AIW Plus, and AIW Circles Collcagues problems. Larger reasoning models on a 32B scale populate the mid level correct response rale range 0.3 0.6 (the only exception being R1-Qwen-32B), while conventional LLMs largest scale stay well below 0.2. As reference, we show closed rcasoning models 03-mini and ol-preview that show only weak fluctuations and settle in the upper performance region above 0.7. Test data contamination cannot be excluded for models with closed data.
3.2 Severe generalization deficit in advanced reasoning models revealed by AIW variants
Recent reasoning models have shown strong increase compared to conventional LLMs in scores on standardized benchmarks related to problem solving at olympiad or graduate difficulty levels (AIME24 ALME (2024),
MATH500 Hendrycks et al. (0021), GPQA Diamond Rein et al. 20024)). We take various types of reasoning models created from strong base using either real or synthetic reasoning traces data: those using both conventional supervised fine-tuning (SFT) and reinforcement learning (RI.) (DeepSeek R1 Guo et al. 2025 created from DeepSeek v3; QwQ 32B and Light-R1-32B created from Qwen 2.5 32B Instruct) and those employing SFT only, distilled using reasoning traces, either synthetically generated from DeepSeek R1 (S1.1 32B Muennighoff et al. (0025), Thinker 32B Team (2025a b). based on Qwen 2.5 32B Instruct), from DeepSock R1 Zero (RI-Distilled-Qwen--- mix of real and synthetic reasoning data (LIMO-32B Ye et al. (2025)).
and Di-Distilled-LLammaa
Guo et al. 2025 ) or curated from
We measure the performance of these models using AIW problems, to test the strong claims behind the high scores achieved on reasoning benchmarks (scc also Sec. 3.1.2) and also check how these models compare to conventional LLMs. To reduce confounds of test data leakage as described in Sec. 3.1.2 Fig. and Suppl. Sec. C.2, Suppl. Fig. [4] we exclude AIW original and AIW Ext from the experiments, as reasoning models released very recently might have been exposed to data from those or very similar problems due to their public availability. We take thus AIW Friends, AIW Plus and AIW Circles Colleagues as problem test set (see Suppl. Sec. B.2 for AIW variants overview)
10

---

Published in Transactions on Machine Learning Research (01/2026)
As evident from Fig. reasoning models also exhibit strong fluctuations across AIW variations (shown for ATW Friends variations 1-6, THINKING v2 prompt, Fig. (A), ol-preview being frontier closed model exception where data contamination can't be entirely ruled out). While average correct response rates for reasoning models increase across variations with larger model scale, strong fluctuations clearly persist. Despite reasoning models clearly outperforming conventional LLMs in average correct response rales (Fig. 5 (B)) we thus still observe severe zero-shot generalization deficits on variations of simple AIW problems, also for larger scale reasoning models (see also Suppl. Sec. C.1). Standardized reasoning benchmarks cannot detect the observed generalization deficit: models showing strong performance fluctuations show high benchmark scores. Models scoring high on MATH-500, ATME24 and GPQA-D can also have very low ATW Friends scores, Tab. For similar observations on further AIW versions, AIW Plus and AIW Circles Colleagues, see Suppl. Sec.
Suppl. Fig. 12
13
Model
AIME24
MATH-550
CPQA-D
AIW Friends
s1.1-32B
0.647
0.890
0.601
0.698
DeepSeek-R1
0.798
0.973
0.715
0.588
ol-mini
0.636
0.900
0.600
0.529
DS-RR--Distilldd-Laae Llama-70B
0.700
0.945
0.652
0.537
DS--L--Distiledd-Llama--
0.417
0.891
0.490
0.150
DS-RU-Diisilledd-wea---7
0.555
0.928
0.491
0.107
TpenThoougtt337BB
0.690
0.900
0.537
0.070
0.726
0.943
0.621
0.003
Table 1: High scores on reasoning benchmarks containing advanced high school, olympiad or graduate level problems (MATH-500, AIME24, GPQA-D) do not guarantee high scores on much simpler AIW Friends problem. Lack of model robustncss to AIW Friends variations are also not reflected in benchmark scores. Standardised reasoning benchmarks are thus not suitable for detecting problem solving and generalization deficits.
4 Related work & limitations
Mcasuring LLMs' capabilities. Since the seminal breakthroughs in language modelling Devlin et al. (2018); Raffel et al. (2020); ; Brown et. al. (2020), measuring LLM capabilities became indispensable for evaluations and model comparison. To measure how well a language model performs on reasoning, there cxists a plethora of different standardized reasoning benchmarks such as ARC Clark et al. (218).. PIQA Bisk
et al. 22200), GSM8K Cobbe et al. (202T), HellaSwag Zellers et al. (2019), MMLU Hendrycks al. (2020),
WinoGrande Sakaguchi et al. (2019) and more recent. (ATME24 ATME) (2024) and GPQA Diamond Rein et al.
(2024) ). Multiple works aim on improving reasoning performance of LLMs measured by those standardized
benchmarks in various ways Wei et al. (2022) Yao et al. 2024 Zhou et al. (2022): Wang et al. 2022 Pfau
et al. 2024g: Guo et al. 2025).
Stress-testing LLMs' weaknesses. Paralleling impressive progress shown by LLM research, cautious voices have been raising concern about discrepancy between claimed capabilities as measured by standardized benchmarks and true LLM reasoning skills by presenting carefully selected evidence for model failures Mitchell] (2023). In response, the rescarch community has been undertaking attempts to create more challenging benchmarks like HELM Liang et al. (2023) or BIG-bench Srivastava al. (2023) These benchmarks also aimed at properly testing generalization capabilities beyond memorization, in line with recent works that pointed out high test dataset contamination due to large-scale pre-training on web-scale data Golchin &
Surdeanu 2023 Li & Flanigan
2024).
Multiple studies Wu et al. (023) Dziri et al. (2024); ; Lewis & Mitchell 20044 Berglund et al. (023
Moskvichev et al. (2023); Huang et (2023) have shown breakdowns language models reasoning capa bilities
different scenarios and also lack of robustness to variation of problem formulation Zong et al. (2024) Zheng
et al. (0224). Other works were looking into particular reasoning failures like deficits in causality inference
11

---

Published in Transactions on Machine Learning Research (01/2026)
Jin et al. (2023b a). These works operate often with formalized, rather complex problems that does not have simple common sense character expressible in natural language. Similar in spirit to our work, simple math word problems were used in Patel et al. (2021) p to show model breakdowns, but models on current frontiers that claim strong generalization and advanced capabilities were not tested. Recently, Mirzadeh et al. (2024) made use of similar approach to create variat from templates of GSM8K problems. This work however does not provide for strong models any conclusive evidence of function breakdown on simple problems, fact showing evidence that models like GPT-4o or Llama 3 SB can handle those well (see Suppl. Sec. C Suppl. Fig. 29). In contrast, we are able to measure lack of model robustness and strong fluctuations also for frontiers LLMs and reasoning models pretrained at largest scales. We control for problem difficulty to keep it unchanged, also ensuring problem execution length not becoming a confound for failures, so that we can clearly attribute those to generalization deficits, in contrast to Shojaee el al. (2025). A key limitation of our current approach is the lack of sufficient diversity in AIW problem variations. This can be addressed in future work by systematic procedural instance generation for broader evaluation across more diverse problem types.
5 Discussion & Conclusion
In our work, using a simple AIW problem (Sec. [] that can be easily solved by adults and arguably even children, we observe striking breakdown of SOTA LLMs performance when confronted with variations of the AIW problem template that do not change problem structure or its difficulty (Suppl. Tab. 3) The breakdown is manifested in (i) low average correct response rates (Fig. 2 and (ii) strong performance fluctuation across structure and difficulty preserving natural variations of the same problem, which reveals severe zero-shot generalization deficit of the models in this case (Fig. 3 While clearly outperforming conventional LLMs, recent reasoning models like DeepSock-R1 also show strong fluctuations when facing variations of different ATW versions (Fig. 5). By executing control experiments, we show that models do not lack capability to perform basic operations necessary to solve AIW problem (Sec. 3.1.1 Fig. 3 E-F). This provides further support for the hypothesis that observed performance fluctuations origin from failure to properly infer common structure from instances of the same problem and thus signal generalization deficit.
Our study also points to failure of standardized benchmarks to properly measure model generalization and detect generalization deficits as revealed by testing on AIW problem variations (Tab. [1 Suppl. Sec. C.5 Fig. 22 Tab. 13). Standardized benchmarks assigning high scores to SOTA LLMs and LRMs fail to signal clear generalization deficits made evident by breakdown on natural variations of simple AIW problem. We note that despite observed brcakdown with low average correct response rates and strong fluctuations, the reasoning can be functional and also conventional LLMs, especially al larger scale (eg. GPT-4, Claude 3 Opus), show instances of fully correct reasoning (see Suppl. Fig. [2 33 As our results show, this reasoning capability is however fragile and cannot be accessed robustly, as reflected by strong performance fluctuations on natural variations of problems as simple as AIW.
Detecting generalization deficits via AIW problem testing helps to avoid illusion of robust strong function. It reveals discrepancy between high scores on problems of graduate olympiad difficulty contained in standardized benchmarks and inability to handle robustly simple AIW problems. As hint on origin of this apparent contradiction we show that for highly similar, but different AIW versions, models can exhibit strongly different performance (Fig, 4 (A-C)). Fine-tuning experiment reveals that exposing model to AIW problem examples during training can strongly push performance and abolish fluctuations across variations of the given problem. However, strong fluctuations on a different, similar ATW problem version that is not shown during training remain (Fig. 4 (D-G)). This again provides evidence for generalization deficits as root. cause and lest set leakage being one possible explanation for the strong function illusion.
The evidence from our study serves thus as a warning against attributing to LLMs and LRMs strong generalization and robust reasoning on problems of higher difficulty by relying solely on high scores the models achieve on standardized problem solving benchmarks. While we see in the recent reasoning models a promising direction to enable stronger model function than conventional LLMs offer, clear generalization deficits still persist in LRMs also on problems as simple as AIW Friends, as reflectod strong performance fluctuations (Fig. 5). While already simple SFT on reasoning traces data can thus strongly boost LLMs, the generalization deficit however remains, also when using SFT+RL. To properly measure generalization
12

---

Published in Transactions on Machine Learning Research (01/2026)
and guide further progress, more sensitive benchmarks required. AIW problem and its variations offer measurement technique that can reveal lack of robustness and generalization deficits that current benchmarks do not detect. Variations built into problem templates can serve as method for creating new benchmarks that are; in contrast to current common benchmarks, no longer static and can probe robustness to variations and also better counter test set leakage (Fig. +. providing more reliable measurement. and lesting of model generalization and reasoning. New benchmarks should follow Karl Popper's principle of falsifiability Popper (1944), aiming to falsify strong function hy hypothesis by spotting deficits, and thus showing possible directions for model improvement, while also offering protection from overblown claims about model's capabilities.
Acknowledgments
We would like to express gratitude to all the people who are working on making code, models and data publicly available, advancing community based rescarch and making rescarch more reproducible. Specifically, we would like to thank all the members of the LAION Discord server community and Open- (Open-Sci) Collective for providing fruitful ground for scientific exchange and open-source development.
LCK acknowledges the Helmholtz Information & Data Science Academy (HIDA) for providing financial support enabling a short-term research stay at Juelich Supercomputing Center (JSC), Research Center Juelich (FZJ) to conduct research on foundation models and their evaluation.
MN, MC and JJ acknowledge co-funding by EU from Digital Europe Programme under grant no. 101195233 (openEuroLLLM), co-funding from EuroHPC Joint Undertaking programme under grant
101182737
(MINERVA), funding by EU Horizon under grant no. 101214398 (ELLIOT), as well as funding by the Federal Ministry of Research, Technology and Space of Germany (BMFTR) under grant no. 011S24085C (OPENHAFM), under the grant 16HPC117K (MINERVA) and under the grant no. 01TS22094B (WestAT A1 Service Center West). We also acknowledge compute resources provided via WestAl compute grant 'Mcasuring and enhancing advancod reasoning capabilitics of foundation models via local model deployment' (westai0007, westai0066) at Juelich Supercomputing Center (JSC).
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint UrXiv:2303.08774 2023.
AIME. Awwrimammmnnnnaeeeee invitational mathematics examination (AIME), 2024. URL /ttLLttt::///artffpoolllaaaaasa Problems, and Solutions
com
Alibaba. Introducing qwen1.5, 2024a. URL http:://paallnggttttt./llgggg//e
Alibaba. Introducing qwen2.5, 2024b. URL Alibaba. Hello qwen2, 2024c. URL [ttpe://geeabbaggttbbb.bb//gggg
//
Anthropic. Introducing the next generation of claude, 2024a. URL httpp://ww..attirgicccco////ee claude-3-familly
Anthropic. Claude 3.5 sonnet, 2024b. URL
Anthropic. The claude 3 model family: Opus, sonnet, haiku, 2024c. URL Slaude3-model-coe
Simran Arora, Avanika Narayan, Mayee F Chen, Laurel Orr, Neel Guha, Kush Bhatia, Ines Chami, and Christopher Re. Ask me anything: simple strategy for prompting language models, In The Eleventh International Conference on Learning Representations, 2022.
httpe:///dssccod.ggg//ggreeee https://discordd www.w/oor.....geeoeteeeeeeeeeeee
13

---

Published in Transactions on Machine Learning Research (01/2026)
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arviv preprint or arx:::209. 1609, 2023.
Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The reversal curse: Lims trained on" a is bt fail to learn' is a'. orXiv preprint arXiv:2309.1220 2023.
Aerri.Al. Litellm, 2024. httpp://ggghab..cc/Brrrr///////
Yonatan Bisk, Rowan Zellors, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Peler Clark, Isaac Cowhey Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge. arXiv preprint 15f3:880005455 2018.
Karl Cobbe, Vincet Kosaraju, Mohammad Bavarian, Mark Chen, Hcewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 14168, 2021.
Cohere. Introducing command r+: A scalable 11m built for business. URL http::///cchhree.cmm/bbbggg Conmand-r-plus m-o.sppp-summm..oooooo-------
Cohere, Command r+ documentation, 2024a. URL http:p////osssscceeec.cco//o///p
Cohere. Model card for c4ai command rt, 2024b. URL httg:://hugiigggae.ccc//oooeeeer
coai-coomand----puus
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810. 180.0880, 2018.
Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Sean Welleck, Peter West, Chandra Bhagavatula, Ronan Le Bras, et al. Faith and fate: Limits of transformers on compositionality. Advances in Newral Information Processing Systems, 36, 2024.
Simon Frieder, Luca Pinchetti, Ryan-Rhye Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Petersen, and Julius Berner. Mathematical capabilities of chatgpt. Advances in Newral Information Processing Systems, 36, 2024.
Shahriar Golchin and Mihai Surdeanu. Time travel in Ilms: Tracing data contamination in large language models. arXiv preprint arXiv:230.0084 08493, 2023.
Google. Gemma: Introducing new statc-of-thc-art open models, 2024a. URL https://blog.got taccholoogy/ddvvlloprrrs////ss httpa/pprrrreeeeemmmmee-eee-eoro
Google. Gemma model card, 2024b. URL
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Xr preprint arXiv:2501.129/8 2025.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Stein- hardt. Mcasuring massive multitask language understanding. In International Conference on Learning Representalions, 2020.
14

---

Published in Transactions on Machine Learning Research (01/2026)
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. or Xiw preprinl arXiv:2103.03877 2021.
Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulmier, et al. Mistral 7b. arxiv preprint arXim:2310.0682) 2023.
Zhijing Jin, Yuen Chen, Felix Leeb, Luigi Gresele, Ojasv Kamal, Zhiheng LYU, Kevin Blin, Fernando Gonzalez Adauto, Max Heimann-Weiner Mrinmaya Sachan, and Bernhard Schölkopf. Cladder: A benchmark to assoss causal reasoning capabilities of language models. In Thirty- Conference on Neural Information Processing Systems, 2023a. URL https::/openrrevee: wtt.://wwww...ee.eooorrro//////a
Zhijing Jin, Jiarui Liu, LYU Zhiheng, Spencer Poff, Mrinmaya Sachan, Rada Mihalcea, Mona T Diab, and Bernhard Schölkopf. Can large language models infer causation from correlation? In The Twelfth International Conference on Learning Representations, 2023b.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Twasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199 22213, 2022.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. vllm, 2024. URL attps://githiu com/vllm htt//vllllppoor/ wtw.lmmmpooeet/vill.
Martha Lewis and Melanie Mitchell. Using counterfactual tasks to evaluate the generality of analogical reasoning in large language models. arXiu preprint arXiv:2402.0800 2024.
Changmao Li and Jeffrey Flanigan. Task contamination: Language models may not be few-shot anymore. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 18471 18480, 2024.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Alexander Cosgrove, Christopher D Manning, Christopher Re, Diana Acosta-Navas, Drew Arad Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxin Yao, Jue WANG, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri S. Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Andrew Chi, Sang Michael Xic, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yilan Mai, Yuhui Zhang, and Yuta Koreeda. Holistic evaluation of language models. Transactions on Machine Learning Research, 2023. TSSN 2835-8856. URL Featured Certification, Expert Certification.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chenggi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 2024.
Meta. Meta and microsoft introduce the next generation of Ilama meta, 2023. URL comm/nera//223//77/11mmmm
meta
Meta. Introducing meta llama 3: The most capable openly available 11m to date, 2024a. URL https:
//ai //ai.meta.co http:bbloo/meta--laaaaa
Meta. Introducing llama 3.1: Our most capable models to date, 2024b. https::/ai..ett..com
meta-1lama-3-1
Meta. Meta laaaa 3 model card., 2024c. URL http::////gaaaaooooooaaaaaaaaaaa MODEL CARD md
15

---

Published in Transactions on Machine Learning Research (01/2026)
Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models. arXiv preprint XXx::2410005229, 2024.
Mistral-AI-Team. Announcmmett mistral ai 7b, 2023.
URL https:/misstraa.aa/neegg
amnouncing-mistral-70
Mistral-Al-Team.
Announcement mixtral ai 7x22b, 2024a.
URL http:///mistral..a/nness
mixtral-8x22b/
Mistral-AI-Team.
Announcement mixtral ai 7x8b, 2024b.
URL
https://mistrall.ll//er
mixtral-of-experts/
Mistral-AI-Team. Mistral ai models api versioning, 2024c.
URL https://dbcs.mistral
Mistral-AI-Team. Au large mistral ai frontier ai in your hands, 2024d. URL http:://nlstzal.al//een
mistral-large/
Melanie Mitchell. How do we know how smart ai systems are? Science, 381(6654)) 2023.
Mosaic. Introducing dbrx: A new state-of-the-art open 1lm. URL http:://wwwdataarrcckk..... /
Arsenii Kirillovich Moskvichev, Victor Vikram Odouard, and Melanie Mitchell. The conceptare benchmark: Evaluating understanding and generalization in the are domain. Transactions on machine learning research, 2023.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. ar preprint arXiv:2501.19395 2025.
OpenAL. Gpt-4 turbo and gpt-4 model docs., a. URL gpt app-4-tuuroo-ad--pt--
OpenAl. Models openai gpt 3.5 turbo docs, b. URL http:://aaatooo..ppppaa.o/oo/ooo
t-t-bob-turo
OpenAl. Models openai gpt 3.5 turbo update, aadeels-and-aee and-apd updates/
URL http:://openaii.com/nnee
new embedding models modelss
OpenAL. Introducing chatgpt, 11 2022. URL http:s////pppaaacc/o///gggaaaaag
OpenAl. Gpt-4o model docs., 2024a. URL http:p////eeetoo..epaaa...ooooo/
OpenAl. Announcmentt: Hello gpt-4o, 2024b. URL http:///s///oeaaaaaa/oo///oaa/oa
OpenAL ol model docs., 2024c. URL http:p/s//plaattm......oo.ooo/os
OpenAI. ol announcement and model
card, 2024d.
URL
https://openai.com
card/
OpenAl. ol-mini model docs., 2024c. URL
OpenAl.
Announcement: ol-mini, 2024f.
URL
https://openai.commnneee
openai-on www.-mmiiiadvvaiinn--oomm cost cost- -wt.-oooooociiiitooooooooiiiooio
OpenAL ol preview model docs., 2024g. URL http:///ppppppttoooo.oo.o...ooop
01 1h-previex-222-09-11
OpenAL. Announcement: ol
preview,
2024h.
URL
https://openai..om//i
introduccing-oppnnii------pe
16

---

Published in Transactions on Machine Learning Research (01/2026)
OpenAL. 03-mini model docs., 2025a. URL http://///.bbbtt.........../////
OpenAL. Announcement: 03-mini, 2025b. URL httpp:///pppaa..ccooommm/p/aappp
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are nlp models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Compulational Linguisties: Human Language Technologies, pp. 2080 2094, 2021.
Jacob Pfau, William Merrill, and Samuel R Bowman. Let's think dot by dot: Hidden computation in transformer language models. arXiv preprinl 2rXi:22444.155 15758, 2024.
Sundar Pichai and Demis Hassabis. Introducing gemini: Google's most capable ai model yet, 2023. URL https://blog htogg////aaacooooo////////////ga
Sundar Pichai and Demis Hassabis. Our next-gemeratiion model: Gemini 1.5, 2024. URL https: httpss//blog
Karl Raimund Popper. The logic of scientific discovery. 1934.
Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint 2rXi:2412.15111 2025. URL attps://arxias org/abgg attpaab//./22222 2412.
Colin Raffel, Noam Shazoer, Adam Roberts, Katherine Lec, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified lext-to-texi (ransformer. Journal of machine learning research, 21(14)):1-67, 2020.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrpp, Jean-baptiste Alayrac Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint SSS::2403.05533 2024.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level moogle-proff q&aa benchmark. In First Conference on Language Modeling, 2024.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WINOGRANDE: an adversarial winograd schema challenge at scale, 2019.
Parshin Shojaee, Iman Mirzadeh, Keivan Alizadeh, Maxwell Horton, Sany Bengio, and Mehrdad Farajtabar. The illusion of thinking: Understanding the strengths and limitations of reasoning models via the lens of problem complexity. Xiv preprint arxww:2506.0 06941, 2025.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonse Agnieszka Kluska, Aitor Lewkowyez, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nic, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane, Anantharaman S. Iyer, Anders Johan Andreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhlmüller, Andrew M. Dai, Andrew La, Andrew Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Eirat, Aykut Erdem, Ayla Karakaş, B. Ryan Roberts, Bao Sheng Loe, Barrel Zoph, Bartiomiej Bojanowski, Batuhan Ozyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ekmekci, Bill Yuchen Lin, Blake Howald, Bryan Orinion, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Argueta, Cesar Ferri, Chandan Singh, Charles Rathkopf, Chenlin Meng, Chitta Baral, Chiyu WU, Chris Callison- Burch, Christopher Waites, Christian Voigt, Christopher D Manning, Christopher
17

---

Published in Transactions on Machine Learning Research (01/2026)
Potts, Cindy Ramirez, Clara E. Rivera, Clemencia Siro, Colin Raffel, Courtney Ashcraft, Cristina Garbacca, Damien Sileo, Dan Garrette, Dan Hendryeks, Dan Kilman, Dan Roth, C. Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Mosegui González, Danielle Perszyk, Danny Hernandez, Danqi Chen, Daphne Ippolito, Dar Gilboa, David Dohan, David Drakard, David Jurgens, Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam; Dieuwke Hupkes, Diganta Misra, Dilyar Buzan, Dimitri Coelho Mollo, Diyi Yang. Dong-Ho Lee, Dylan Schrader, Ekaterina Shutova, Ekin Dogus Cubuk, Elad Segal, Eleanor Hagerman, Elizabeth Barnes, Elizabeth Donoway, Ellie Pavlick, Emanuele Rodolå, Emma Lam, Eric Chu, Eric Tang, Erkut Erdem, Ernie Chang, Ethan A Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Funice Engefu Manyasi, Evgenii Zheltonozhskii, Fanyue Xia, Fatemeh Siar, Fernando Martínez- Plumed, Francesca Happé, Francois Chollet, Frieda Rong, Gauray Mishra, Genta Indra Winata, Gerard de Melo, Germán Kruszewski, Giambattista Parascandolo, Giorgio Mariani, Gloria Xinyue Wang, Gonzalo Jaimovitch-Lopez Gregor Betz, Guy Gur-Ari, Hana Galijasevic, Hannah Kim, Hannah Rashkin, Hannaneh Hajishirzi, Harsh Mchta, Hayden Bogar, Henry Francis Anthony Shevlin, Hinrich Schuctze, Hiromu Yakura, Hongming Zhang, Hugh Mee Wong, Ian Ng, Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kernion, Jacob Hilton, Jaehoon Lee, Jaime Fernández Fisac, James B Simon, James Koppel, James Zheng, James Zou, Jan Kocon, Jana Thompson, Janelle Wingfield, Jared Kaplan, Jarema Radom, Jascha Sohl-Dickstein, Jason Phang, Jason Wei, Jason Yosinski, Jekaterina Novikova, Jelle Bosscher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel, Jesujoba Alabi, Jiacheng Xu, Jiaming Song, Jillian Tang, Joan Waweru, John Burden, John Miller, John U. Balis, Jonathan Batchelder, Jonathan Berant, Jörg Frohberg, Jos Rozen, Jose Hernandez-Orallo, Joseph Boudeman, Joseph Guerr, Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan, Katerina Ignatyeva, Katja Markert, Kaustubh Dhole, Kevin Gimpel, Kevin Omondi, Kory Wallace Mathewson, Kristen Chiafullo, Ksenia Shkaruta, Kumar Shridhar, Kyle McDonell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang, Liam Dugan, Lianhui Qin, Lidia Suntreras-Ochando Louis-Philippe Morency, Luca Moschella, Lucas Lam, Lucy Noble, Ludwig Schmidt, Luheng He, Luis Oliveros- Luke Metz, Lütfi Kerem Sencl, Maarten Bosma, Maarten Sap, Maartje Ter Hoove, Maheen Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, Maria Jose Mannirez-uintana, Marie Tolkiehn, Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew L Leavitt, Matthias Hagen, Mátyás Schubert, Modina Orduna Baitemirova, Melody Arnaud, Melvin McElrath, Michael Andrew Yee, Michael Cohen, Michael Gu, Michael Ivanitskiy, Michael Starritt, Michael Strube, Michal Swędrowski, Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Mitch Walker, Mo Tiwari, Mohit, Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, Mukund Varma T, Nanyun Peng, Nathan Andrew Chi, Naycon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nicole Martinez, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar, Niveditha S. Tyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Autonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Pu Liang, Paul Vicol, Pegah Alipoormolabashi, Peiyuan Liao, Percy Liang, Peter W Chang, Peter Eckersley, Phu Mon Htut, Pinyu Hwang. Piotr Milkowski, Piyush Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, Qing Lyu, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Racfer Gabriel, Rahel Habacker, Ramon Risco, Raphaël Millière, Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan Le Bras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Russ Salakhutdinov, Ryan Andrew Chi, Seungjac Ryan Lee, Ryan Stovall, Ryan Techan, Rylan Yang. Sahib Singh, Saif M. Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel R. Bowman, Samuel Stern Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebastian Gehrmann, Sebastian Schuster, Sepideh Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixiang Shane Gu, Shubh Pachchigar, Shubham Toshniwal, Shyam Upadhyay, Shyamolima Shammie Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo- Hwan Lee, Spencer Torene, Sriharsha Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Stella Biderman, Stephanie Lin, Stephen Prasad, Steven Piantadosi, Stuart Shieber, Summer Misherghi, Svetlana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq Ali, Tatsunori Hashimoto, Te-Lin Wu, Théo Desbordes, Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili, Timo Schick, Timofei Kornev, Titus Tunduny, Tobias Gerstenberg, Trenton Chang, Trishala Neeraj, Tushar Khot, Tyler Shultz, Uri Shaham, Vedant Misra, Vera Demberg, Victoria Nyamai, Vikas Raunak, Vinay Venkatesh
18

---

Published in Transactions on Machine Learning Research (01/2026)
Ramasesh, vinay uday prabhu, Vishakh Padmakumar, Vivek Srikumar, William Fedus, William Saunders, William Zhang, Wout Vossen, Xiang Ren, Xiaoyu Tong, Xinran Zhao, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yangqiu Song, Yasaman Bahri, Yejin Choi, Yichi Yang. Yiding Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yufang Hou, Yuntao Bai, Zachary Seid, Zhuoye Zhao, Zijian Wang, Zijie Wang, Zirui Wang, and Ziyi Wu. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. 11805, 2023.
OpenTThoughts Team. OpenTThinkrr 32B. http:///ungginniiccccoo/o/oo/oos Thw.neeerehhiie-eeeeere- -2BB, January 2025a.
Thoughsss Team. Open Thoughts. https://open-ttoonghttaaa January 2025b.
TogetherAL. Togetherai api, 2024. URL http::://ddoc..oogeetto aii/
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Ammr Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Sounya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswamii, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Audrew Poulton, Jeremy Reizenstein, Rashi Rungla, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Nlyan Zarov, Yuchen Zhang, Angela Fan, Melanic Kambadur, Sharan Narang, Aurolien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. arXiv, 2023b. URL https://arxttii htt////xxxvvvrrrer wttp:vv..rrgbb///3010/ ht////////2///// 292880
Xuezhi Wang, Jason Wei, Dale Schaurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2022.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. thain-of-thouuhtt prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 248377, 2022.
Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C Schmidt. A prompt pattern catalog to enhance prompt engincering with chatgpt. arXiv preprint ar Xiv:22302..1 112..11100 11382, 2023.
Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim. Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks, arXiv e-prints, pp. arXiv 2207, 2023.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024.
Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less more for reasoning. Xixiv preprint ar arXi:2202200338 033887 2025.
19

---

Published in Transactions on Machine Learning Research (01/2026)
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Associalion for Compulational Linguistics, 2019. URL
07830
Wenqi Zhang, Yongliang Shen, Linjuan Wu, Qiuying Peng, Jun Wang, Yueting Zhuang, and Weiming Lu. Self-contrast: Better reflection through inconsistent solving perspectives. arXiv preprint ox1002400.00000, 2024.
Chujic Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlic Huang. Large language models are not robust multiple choice selectors. In The Twelfth International Conference on Learning Represenlations, 2024. URL htt.:ppaaaaaaaaaoooooooooooaaaaa
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Lc, et al. Least-to-most prompting enables complex reasoning in large language models. Xiv preprint ar arXiv:2205. 10625, 2022.
Yongshuo Zong, Tingyang Yu, Ruchika Chavhan, Bingchen Zhao, and Timothy Hospedales. Fool your (vision and) language model with embarrassingly simple permutations. In Forty-first International Conference on Machine Learning, 2024.
20

---

Published in Transactions on Machine Learning Research (01/2026)
Supplementary.
A Additional details on performed experiments
Here we give further details on the procedures around the executed experiments.
A.1 Models selected for experiments
To provide overview over origin core tested models used for the AIW experiments, we list those Suppl Tab. 2 All tested models use same default inference hyperparameters, T 0.1, top-p = 1.0 (we executed control experiments to check that various settings do not change the main pattern in the observed behavior). The output of standard L.LMs was limited to 2048 tokens, and evident from Suppl Fig. 28 most observed
responses stayed well below this limit. When testing recent reasoning models, the limit was raised to 32k 43k, depending on model type.
A.2 Evaluating model responses
The formatting instruction makes it possible to extract. for each prompting trial whether model has provided correct answer to the AIW problem posed in the input. We can interpret then any number n of collected responses executing trials given a particular prompt for a given model (n number of Bernoulli trials), observing in each i-th trial a Bernoulli variable X; = f0, We interpret the number of correct responses E, X2 as random variable following a Beta-Binomial distribution with unknown probability p of correct. response that we also treat as random variable that comes from a Beta distribution, i.e. p~ Beta B), where a and are parameters of the Beta distribution. To obtain plots showing correct response ratios, we would like estimate Beta distribution underlying p, and for that, we first estimate the mean of p and its variance from the collected observations. To estimate we use the formula for estimating the mcan of for binomial distribution: p X/n (i.e. as proportion of successes). We can report the estimate as the estimate of the correct response rate of a given model and also, compare the correct response rates of various tested models. Moreover, we can estimate the variance of the probability of a correct response by using the following formula:
1
n
var(X.) nvar(XX)) n2 n n1
p(1 p)
var -
X1
var (X;)
(1)
71
n
The estimates of the variance and the standard deviation of p can be thus obtained by using á ((1-)) and ((1-A) respectively. Using the estimated variance and mean of p, we can use the following relations for the variance: (o"= maš(a+8tm) and the mean (w= x+3 in order to obtain a and 3 parameters for the [a+))(aat++]]
Beta distribution. To simulate data for the plots, we draw N random samples corresponding to correct and incorrect responses using the estimated distribution of p and obtain the plots showing performance the task for various models of interest as full distribution of the respective p.
B Problem versions, prompt types and variations
AIW Problem. To measure models' sensitivity to problem irrelevant variations and thus probe the zero-shot generalization, we use problem templates, as described in Sec. 2.1 We create controlled problem variations by varying natural numbers N, M < 7, obtaining AIW variations 1-4 (see Suppl. Tab. 3 where problem structure and difficulty remains unchanged, which should ensure variations being irrelevant for problem handling. The resulting variations are as following:
Variation Alice has 3 brothers and she also has sisters. [Corectt answer: 7
Variation 2. Alice has 2 sisters and she also has 4 brothers. [Correct answer: 3)
21

---

Published in Transactions on Machine Learning Research (01/2026)
Table 2: Names, origin and versioning of core test models used in the experiments.
Name
Origin
Released
Open Weights
Sources
03-mini
OpenAl
31.01.2025
No
OpenAl
2025a
01-2024-12-17
OpenAl
17.12.2024
No
OpenA
2024c
ol-preview
OpenAI
12.09.2024
No
OpenAl
2024g
ol-mini
OpenAl
12.09.2024
No
OpenAl
2024c
GPT-40-2024-05-13
OpenAT
13.05.2024
No
Achiam et al.
2023
Ope-
nAI (2024a b)
GPT-44turbo-2024-000000
OpenAT
09.04.2024
No
Achiam et al.
2023
Ope-
(a)
GPP--4012-ppreview
OpenAl
25.01.2024
No
Achiam al.
2023
Ope-
[A][(a)
GPT-4-0613
OpenAI
13.06.2023
No
Achiam al.
2023
Ope-
[A]
GPT-3.5-turbo-0125
OpenAT
24.01.20 2024
No
OpenA
2022
Authropic
21.06. 2024
No
Anthropic
202 4b
Claude-3-opus-2224000
Authropic
04.04
Anthropic
2024a
Anthropic
04.03.202 2024
No
Anthropic
2024a
-laude33hhaikn-2022000
Anthropic
04.03.2021
Anthropic
2021a
Gemini 1.0 Pro
Google
06.12.2023
No
Pichai &
Hassabis
2023
Team et al.
(2023
Gemini 1.5 Pro
Google
16.02.2024
No
Pichai & Hassabis
2024
Reid et al.
2024)
gemma-2b/7b-it Mistral-large-2402
Google
05.04.2024 (v1.1)
Yes
Google 2024a
Mistral AT
26.02.2024
No
Mistral AT- Team
2024cc db
Mistral-meddhum23332
Mistral AT
23.12.2023
No
Mistral-A Team
202 1c
Mistral-smalll221022
Mistral AI
26.02.2021
No
Mistral-AV Team
202 lc
d
open-mixtrral-x222binsssss
Mistral AI
17.04.2022
Yes
Mistral Al Team
2024c
v0.1
Opeu--miixtrall&&2--inntttuu v0.1
Mistral AI
11.12.2023
Yes
Mistral AI- Team
2024c
open-miitrral----instrcce
Mistral AT
11.12.2023
Yes
Jiang et al. 2023) Mistral
AT-Teem 2024e
2023
Command R+ Dbrx Instruct Llama 2 7B, 13B, 70B Chat
Cohere
14.04.2024
Yes
Cohere
2024a
Mosaic
27.03.2022
Yes
Mosaic
Meta
18.07.2023
Yes
Meta 2023
Touvron et al.
2023b
Llama 3 8B, 70B Chat
Mela
18.04.2024
Yes
Meta 2024a
c
Llama 3.1 8B 405B Instruct
Meta
23.07.2024
Yes
Meta
2024b
Qwen 1.5 1.8B 72B Chat
Alibaba
04.02.2024
Yes
Bai et al.
2023
Alibaba
20244a
Qwen 2 05B 72B Instruct Qwen 2.5 0.5B 72B Instruct
Alibaba Alibaba
07. 06. 2024
Yes
Alibaba
2024c
29.09.2024
Yes
Alibaba
2024b
Qwen
2025
DeepSeek R1 671B DeepSeek v3 671B
DeepSeek
20.01.2025
Yes
Guo
2025
DeepSeek
26.12.2024
Yes
Liu et al.
2024
Variation 3. Alice has 4 sisters and she also has brother. [Correct answer:
Variation 4. Alice has 4 brothers and she also has sister. [Correct answer: 2
Question: How many sisters does Alice's brother have?
22

---

Published in Transactions on Machine Learning Research (01/2026)
Prompt types. For testing the model dependence input prompt type when solving AIW and AIW Light problems to see whether our observations hold independent of prompting, we used three main prompt types STANDARD (original prompt with answer formatting instructions), THINKING (prompt that encourages thinking with answer formatting instructions) and RESTRICTED (prompt that instructs model to output only Tormatted answer and nothing else). THINKING v2 prompt lype a minor variation of THINKING type that just adds "step by step" after already existing 'think carcfully' phrasing (control experiments show that THINKING and THINKING v2 are equivalent in terms of observed performance, so we use both interchangeably, Suppl. Fig. Sb}.
For cach trial, models receive thus an input that has a form <instantiated-errnan template> <prompt-type>> where < fmnstantiaaeedd-empllee is template with substituted numbers instantiating one of problem variations 1-4 (1-6 for some of experiments in Appendix) containing the question, followed by <prompt-type> with task and output instructions corresponding to one of prompt types as described above. See Suppl. Fig. ?? for an illustratory example showing GPT-4 evaluation.
All AIW problem versions employed in this work use the same construction described above, differing either in the main part or in the question of their template. We make the correct answers match across variations of different problem versions to ensure the source of variation is confined to the problem input only and no further source of variation enters through different numbers in the outcomes.
B.1 Control AIW Light problems
To control for models struggling either with basic family relations structure handling or with executing arithmetic operations in frame of the posed AIW problem, we make various versions of AIW problem AIW Light Family, AIW Light Arithmetic Siblings and AIW Light Arithmetic Total Girls, relying on main structure of ATW original template while adapting the question accordingly to operations be tested, as described in Sec. 2.
AIW Light Arithmetic Siblings. The problem version has following variations 1-4:
Variation 1. Alice has 3 brothers and she also has sisters. [Correct answer: 7]
Variation Alice has 2 sisters and she also has brother. [Correct answer:
Variation Alice has 4 sisters and she also has brother. [Correct answer:
Variation 4. Alice has 1 brother and she also has sister. [Correct answer: 2)
Question: How many siblings does Alice have?
AIW Light. Family. The problem version has following variations 1-4:
Variation 1. Alice has 7 brothers and she also has sisters.
[Correct answer: 7]
Variation 2. Alice has 4 sisters and she also has 3 brothers.
[Correct answer: 3
Variation 3. Alice has 2 sisters and she also has 5 brothers.
[Correct answer: 5)
Variation 4. Alice has 2 brothers and she also has sisters.
[Correct answer: 2]
Question: How many brothers does Alice's 's sister have?
AIW Light Arithmetic Total Girls. The problem version has following variations 1-4:
Variation 1. Alice has 6 sisters and she also has brothers. [Correct answer: 7]
Variation Alice has 2 sisters and she also has brothers. Correct answer: 3)
Variation Alice has 4 sisters and she also has brother. [Correct answer:
Variation 4. Alice has 1 sister and she also has 4 brothers. [Corectt answer:
Question: How many girls are there in total?
23

---

Published in Transactions on Machine Learning Research (01/2026)
See Suppl. Tab. 4
5 6
for examples with full prompt versions for each problem and its variations
B.2 Further AIW problem versions
AIW Extended. As described Sec. 2.1.2 ATW Extended (AIW Exi) employs Alice and Bob as two entities, keeping the template structure close the AIW original The resulting variations are following:
Alico and Bob sister and brother.
Variation Alice has 3 sisters and Bob has 6 brothers. [Correct answer:
Variation 2. Alice has 2 sisters and Bob has 2 brothers. CCorrcct answer: 3
Variation 3. Alice has 1 sister and Bob has 4 brothers. [Correct answer: 5]
Variation 4. Alice has 3 sisters and Bob has 1 brother. [Correct answer: 2]
Question: How many brothers does Alice have?
In further experiments, also use additional variations 5-6 as following:
Variation 5. Alice has 2 sisters and Bob has 3 brothers. [Correct answer: 4]
Variation 6. Alice has 3 sisters and Bob has 5 brothers. [Correct answer: 6 Question: How many sisters does Alice's brother have?
AIW Friends. As mentioned in Sec. 2.1.2 in this problem version, we abandon the family frame setting. Instead, we use male and female friends in problem formulation. Note the problem structure is still related to AIW original (as brothers and sisters are male and female siblings). We use an additional condition to ensure there is no common sense ambiguity in this problem version. The resulting variations are as following:
Variation 1. Alice has 3 male friends and she also has 6 female friends. [Correct answer:
Variation 2. Alice has 2 female friends and she also has 4 male friends. [Correct answer: 3
Variation 3. Alice has 4 female friends and she also has male friend. [Correct answer: 5)
Variation 4. Alice has 4 male friends and she also has 1 female friend. [Correct answer: 2]
All mentioned persons friends with each other and have other friends aside.
Question: How many female friends does male friend of Alice have?
In further experiments, we also use additional variations 5-6 as following:
Variation 5. Alice has 2 male friends and she also has 3 female friends. [Correct answer: 4 Variation Alice has 5 female friends and she also has 3 male friends. [Correct answer: 6
Question: How many female friends does male friend of Alice have?
The two problem versions AIW Ext and AIW Friends have similar problem structure and difficulty to AIW original. They thus can be also seen as control to ensure our observations are consistent among various problem formulations that have similar problem difficulty (in contrast to AIW Light control problems which are designed to lest. low level operations involved, such that their templates pose problems of lower difficulty than AIW original). See Suppl. Tab. g 10 for examples with full prompt versions for each problem and its variations.
B.2.1 Harder problem versions
To test what happens if the simple AIW problem is further extended towards substantially harder difficulty level, providing further challenge for the tested models, we construct further problem versions of higher
AAll prompts and their IDs available http:///thhacc..ooeaaaaaaaaaaaaa
24

---

Published in Transactions on Machine Learning Research (01/2026)
difficulty while keeping same relational logic structure appeal and conduct experiments described in following.
+IW+ problem. We constructed an AIW+ problem that features additional hierarchy and distractors when describing relational family structure (see Suppl. Sec ?? for full formulation). AIW+ problem template has following form: 'Alice has M sisters and N brothers in total. Her mother has 2 brothers. She also has sister who does not have children and who has X nephews and nieces in total. Alice's father has 2 sisters. He also has brother who has Y nephews and nieces in total, and who also hos K |sons/ daughters]. How mony cousins does Alice's 's sister have?'.
The solution to +IW+ problem is harder to obtain than the solution to AIW original with its much simpler structure. Solving +IW+ requires taking different paternal sides, that of mother and father, and calculating the number cousins, taking care of subtracting Alice and her siblings, and summing up the total number of cousins from both sides. Still, this problem is arguably far from olympiad or university graduate level, as il requires just using provided numbers and careful execution of elementary arithmetic operations on straightforward path to solution. The correct solution is given by C' = (X- -X (M (Y))+(++++---)-- (M+N+1)) We follow again our approach to create variations by instantiating numbers N, M, X, Y, K in problem template to obtain problem instances of same problem structure and difficulty (see also Suppl. Tab. 11.
We show a full example of correct solution with instantiated numbers for AIW+ variation 1 (N= = M =
1, = 6, Y = K = 2), which corresponds to problem instance as following:
Variation Alice has sister and brother in total. Her mother has 2 brothers. She also has sister who does not have children and who has 6 nephews and nieces in total. Alice's father has 2 sisters. He also has brother who has 5 nephews and nieces in total, and who also has 2 sons. [Correct answer: 7]
Question: How many cousins does Alice's sister have?
Here, we have on the mother side: 6 (total nephews and nieces) - 3 (Alice and her siblings) = 3 cousins; on the father side: 5 (total nephews and nicces) + (sons of the father's brother) - 3 (Alice and her siblings) = cousins; summing up 3- 4 7 cousins which Alice and any of her siblings share.
AIW Collcague Circles. Evidencing strong performance of ol-preview AIW+, we design a further problem with harder difficulty level than AIW original to test whether we can observe same breakdown patterns on the strongest model. AIW like problems can be understood as problems on graphs, featuring entities, properties and relationships that define sets. To increase problem difficulty, we depart from simple connectivity that was characteristic of the AIW original and introduce circles of colleagues where all-to-all connectivity defines a circles, while some entities have connections to outside, make those entities hubs connecting circles.
AIW Colleague Circles problem template has following form: 'Alice has 3 male colleagues and she also has M female colleagues in total. All these mentioned persons in the circle around Alice are colleagues of each other. Bob has 2 female colleagues and 1 male colleague in total. All these mentioned persons in the circle around Bob are colleagues of each other. The people in the circle around Bob do not have other colleagues aside- with the only exception of Matilda. She is collengue of Bob, being part of Bob's 's circle, and she is also colleague of Alice, being part of Alice's 's circle. All the mentioned persons have no colleagues beyond the already described group of people. How many female colleagues does Motilda have?".
Again, the solution to AIW Colleague Circles problem is harder to obtain than the solution to AIW original with its much simpler structure. Same as AIW+, the problem is though still far from olympiad or graduate level, which advanced SOTA LLMs often claim to master robustly. The correct solution is given by C = M +1 We follow again our approach to create variations by instantiating number M in problem template to obtain problem instances of same problem structure and difficulty (note we use only alteration of single number here to create variations 1-6, corresponding to instantiating M 6, 4, 1, 3,5 with corresponding correct answers C 7,3,5, 2, 4, 6, also Suppl. Tab. 12
25

---

Published in Transactions on Machine Learning Research (01/2026)
Table AIW main variations 1-4, prompt types and correct answers overview.
Var. Prompt
Answerr ID
Alice has brothers and she also has 6 sisters. How many sisters does Alice's brother have? Solve this problem and provide the final answer in 7 following form: '### Answer: '.
STANDARD
55
1 Alice has 3 brothers and she also has 6 sisters. How many sisters does Alice's brother have? Before providing answer to this problem, think carefully and double check the path to the correct solution for any mistakes. Provide then the final answer following form: "### Answer:
THINKING
57
/7
1 Alice has 3 brothers and she also has 6 sisters. How many sisters does Alice's brother have? To answer the question, DO NOT OUTPUT / ANY TEXT EXCEPT following format that contains final answer: '###
RESTRICTED 53
Answer:
2 Alice has sisters and she also has brothers. How many sisters does Alice's brother have? Solve this problem and provide the final answer in / 3 following form: '### Answer:
STANDARD
56
2 Alice has 2 sisters and she also has 4 brothers. How many sisters does Alice's brother have? Before providing answer to this problem, think carefully and double check the path to the correct solution for any mistakes, Provide then the final answer in following form: "### Answer:
THINKING
58
2 Alice has sisters and she also has brothers. How many sisters does RESTRICTED 54 Alice's brother have? To answer the question, DO NOT OUTPUT /3 ANY TEXT EXCEPT following format that contains final answer: '###
Answer:
3 Alice has 4 sisters and she also has brother. How many sisters does Alice's brother have? Solve this problem and provide the final answer in / following form: '### Answer:
STANDARD 63
3 Alice has 4 sisters and she also has brother. How many sisters does Alice's brother have? Before providing answer this problem, think / carefully and double check the path the correct solution for any mistakes. Provide then the final answer in following form: "### Answer:
THINKING
64
3 Alice has 4 sisters and she also has brother. How many sisters does RESTRICTED 65 Alice's brother have? To answer the question, DO NOT OUTPUT / ANY TEXT EXCEPT following format that contains final answer: '###
Answer:
4 Alice has 4 brothers and she also has sister. How many sisters does Alice's brother have? Solve this problem and provide the final answer in following form: '### Answer:
STANDARD 69
4 Alice has 4 brothers and she also has sister. How many sisters does Alice's brother have? Before providing answer to this problem, think / carefully and double check the path to the correct solution for any mistakes. Provide then the final answer following form: "### Answer:
THINKING
70
4 Alice has 4 brothers and she also has sister. How many sisters does RESTRICTED 71 Alice's brother have? To answer the question, DO NOT OUTPUT /
ANY TEXT EXCEPT following format that contains final answer: '###
Answer:
4 Alice has 4 brothers and she also has sister. How many sisters does Alice's brother have? Solve the problem by taking care not to make any /2 mistakes. Express your level of confidence in the provided solution as precisely as possible.
CONFIDENCE 11
3 Alice has 4 sisters and she also has 1 brother. How many sisters does Alice's brother have? To solve the problem, approach it as a very / 5 intelligent, accurate and precise scientist capable of strong and sound reasoning, Provide the solution to the problem by thinking step by step.
SCIENTIST 40

---

Published in Transactions on Machine Learning Research (01/2026)
Table 4: AIW Light Arithmetic Siblings variations 1-4
Var. Prompt
Syse/Answer
ID
1 Alice has 3 brothers and she also has 4 sisters. How many siblings does Alice have? Before providing answer to this problem, think carefully step by step and double check the path to the correct solution for any mistakes. Provide then the final answer in following form: '### Answer:'.
THINKING 277
v2 /
2 Alice has sisters and she also has brother. How many siblings does Alice have? Before providing answer to this problem, think carefully step by step and double check the path to the correct solution for any mistakes. Provide then the final answer in following form: '### Answer:
THINKING
278
v2 /
3 Alice has 4 sisters and she also has brother. How many siblings does Alice have? Before providing answer to this problem, think carefully step by step and double check the path to the correct solution for any mistakes. Provide then the final answer following form: "### Answer:
THINKING
279
v2 /2/3
4 Alice has brother and she also has 1 sister. How many siblings does Alice have? Before providing answer to this problem, think carefully step by step and double check the path to the correct solution for any mistakes. Provide then the final answer in following form: "### Answer:
THINKING
280
v2 /
Table 5: AIW Light Family variations 1-4
Var. Prompt
Anpe/Answer
ID
1 Alice has 7 brothers and she also has 3 sisters. How many brothers does Alice's sister have? Before providing answer to this problem, think carefully step by step and double check the path to the correct solution for any mistakes. Provide then the final answer in following form: '### Answer:
THINKING 271
v2 12/1
2 Alice has sisters and she also has 3 brothers. How many brothers does Alice's sister have? Before providing answer to this problem, think carefully step by step and double check the path to the correct solution for any mistakes. Provide then the final answer in following form: '### Answer:
THINKING
272
x2 /2
3 Alice has 2 sisters and she also has 5 brothers. How many brothers does Alice's sister have? Before providing answer to this problem, think carefully step by step and double check the path to the correct solution for any mistakes. Provide then the final answer in following form: '### Answer:
THINKING
273
v2 /5
4 Alice has 2 brothers and she also has 3 sisters. How many brothers does Alice's sister have? Before providing answer to this problem, think carefully step by step and double check the path to the correct solution for any mistakes. Provide then the final answer in following form: '### A
THINKING
274
v2 /
27

---

Published in Transactions on Machine Learning Research (01/2026)
Table 6: AIW Light Arithmetic Total Girls variations 1-4
Var. Prompt
Type/ Answer ID
1 Alice has 6 sisters and she also has 3 brothers. How many girls are there in total? Before providing answer to this problem, think carefully step by step and double check the path to the correct solution for any mistakes. Provide then the final answer in following form: *### Answer:".
THINKING
343
v2
2 Alice has 2 sisters and she also has 4 brothers. How many girls are there in total? Before providing answer to this problem, think carefully step by step and double check the path to the correct solution for any mistakes. Provide then the final answer following form: *#*# Answer:".
THINKING
344
v2 /
3 Alice has 4 sisters and she also has brother. How many girls are there in total? Before providing answer to this problem, think carefully step by step and double check the path to the correct solution for any mistakes. Provide then the final answer following form: *### Answer:
THINKING
345
v2 /
4 Alice has 1 sister and she also has 4 brothers. How many girls are there in total? Before providing answer to this problem, think carefully step by step and double check the path to the correct solution for any mistakes. Provide then the final answer in following form: *### Answer:".
THINKING
346
v2 //
C Model performance and behavior on AIW problem versions
Here we report further details on model evaluation, performance and bohavior as observod on AIW problems. For executing experiments, we either use local model deployment via vLLM Kwon el al. (2024), or API based liteLLM Al (2024) and TogetherAl TogetherA] (2024).
For the full overview of average correct response rate including models that score zero, Suppl. Fig. For the statistics number of trials conducted for each model and each prompt type, see Suppl. Fig.
27 For
the statistics on the average output length across models and prompt types, see Suppl. Fig. 28 For models
behavior on RESTRICT prompt types, see Suppl. Fig. 7 For control comparison of THINKING v2 prompt type to THINKING and STANDARD, see see Suppl. Fig. g
C.1 Persisting fluctuations on AIW versions by recent advanced language and reasoning models
Following Sec. 3.1 3.2 we demonstrate here on further AIW versions that strong fluctuations also affect models released very recently, including large scale LLMs like DeepSeek-v3 and Llama 3.1 405 or advanced reasoning models like DeepSeek R1 and ol-mini that go beyond standard L.MSs.
On AIW Ext (Suppl. Fig. 9), we observe ol-mini exhibiting strong fluctuations, while ol-preview handles the problem robustly. This also falsifies strong claims put forward by ol-mini to match even outperform ol-preview which were made relying on standardized benchmarks. Large scale standard LLMs like DeepSeek 3 670B, Llama 3.1 405 and Qwen 2.5 72B also exhibit strong fluctuations and have lower correct response rates compared to ol-mini. Claude 3.5 Sonnet collapses strongly with low correct response rates.
Exposing models AIW+, we observe further, even stronger collapse of performance also for those advanced models that were showing significant correct response rates for AIW problem (Fig, [0] For instance, for GPT-4/4 4/4o and Claude 3 Opus overall correct response rate averaged across variations stays below p < 0.2 (Fig, 10 (A)). Large-scale open weight SOTA models Llama 3.1 405B, DeepSeek-v3 671B and Qwen 2.5 72B are settled around p 0.1 and below. Recent Claude 3.5 Sonnet is an outlier scoring higher up close to 0.4, without showing strong fluctuations we usually observe (Fig. 10 (B); see however Sec. on Claude 3.5 Sonnet breakdown its performance on some versions of AIW might be due to exposure to AIW tasks in post training, as appeared after first version of our public AIW relcase). To show that problem can be successfully handled, we also test here ol-preview that comes from the recent generation of reasoning models (which treat exception; I.LMs conventionally understood models pre-trained in purely
28

---

Published in Transactions on Machine Learning Research (01/2026)
Table 7: AIW Alice Feraaee Power Boost and AIW Original, variations 1-4, THINKING v2 prompt
Var.
Prompt
Type/ Answer
ID
1
Alice is female and has 3 brothers and she also has 6 sisters. How many sisters does Alice's brother have? Before providing answer this problem, think carefully step by step and double check the path to the correct solution for any mistakes. Provide then the final answer in following form: ### Answer:
FEMALE BOOST /
193
1 Alice has 3 brothers and she also has 6 sisters. How many sisters does Alice's brother have? Before providing answer to this problem, think carefully step by step and double check the path the correct solution for any mistakes. Provide then the final answer in following form: '### Answer:
NO BOOST 205 /
2 Alice is female and has 2 sisters and she also has 4 brothers. How many sisters does Alice's brother have? Before providing answer to this problem, think carefully step by step and double check the path to the correct solution for any mistakes. Provide then the final answer in following form: "### Answer:
FEMALE BOOST
197
2 Alice has sisters and she also has brothers. How many sisters does Alice's brother have? Before providing answer to this problem, think carefully step by step and double check the path to the correct solution for any mistakes. Provide then the final answer in following form: '### Answer:
NO BOOST 206 /
3 Alice is female and has 4 sisters and she also has brother. How many sisters does Alice's brother have? Before providing answer to this problem, think carefully step by step and double chock the path to the correct solution for any mistakes. Provide then the final answer in following form: *## Answer:
FEMALE
189
BOOST /
3 Alice has 4 sisters and she also has brother. How many sisters does Alice's brother have? Before providing answer to this problem, think carefully step by step and double check the path to the correct solution for any mistakes. Provide then the final answer in following form: '### Answer:
NO BOOST 187
4 Alice is female and has brothers and she also has 1 sister. How many sisters does Alice's 'ss brother have? Before providing answer to this problem, think carefully step by step and double check the path to the correct solution for any mistakes. Provide then the final answer in following form: *## Answer:
FEMALE BOOST
190
4 Alice has 4 brothers and she also has sister. How many sisters does Alice's brother have? Before providing answer to this problem, think carefully step by step and double check the path the correct solution for any mistakes. Provide then the final answer in following form: '###
NO BOOST 188
/
Answer: "
29

---

Published in Transactions on Machine Learning Research (01/2026)
AIW Correct response rate
608
0.10
0.15
0.43
022
1,00
opt Ae 05-13
3p1--00113
lama- 2--bb-che
cbro dirutrattud
apt-farrb 201-4aarbb-220-040 2014-
codelamna 541- inttuct
Lona Tib-cha
dioeroo cormes- 18s
autrud
di
yt chul
gormn
teme-22 2- 12b-chel
open' 5-4b-chal
codel
mestral
openn 5- 5-110b-chul
operenatii
1i Soude-dnsll.... 20240307
cino und
157h-chll
span-riatu dect
go15 Stubss
awant
sronffere
duuce-211
awont
ToLa-nemees aamaa 135
gemm-11 sten
dieate-rolars11.
germa- 7b4
da
commani-rgghy
comnnnieguu
gald 94ume-rfor
nétuinndiun
Figure 6: Collapse of most SOTA LLMs AIW problem. AIW correct response rate across AIW variations averaged across all prompt types THINKING, STANDARD and RESTRICTED, Only 5 models manage to show rates above = 0.2: GPT-4o, Claude 3 Opus, GPT-4-0613, Llama 70B Chat and GPT--4-115-preeveen Llama 2 70B Chat is the only open-weights model this set. The rest either shows poor performance below = 0.15, or even collapses entirely to 0. Among those models collapsing to 0 are many which claim strong function via high scores obtained on standardized benchmarks, eg larger scale GPT-3.5, Mixtral 8x7B and 8x22B, Command R Plus, Qwen 1.5 72B Chat and smaller scale Gemma-7b-it, Mistral Small and Mistral Medium.
30

---

Published in Transactions on Machine Learning Research (01/2026)
Table 8: AIW Exaggerated Numbers variations 1-4, Thinking v2 prompt type.
Var. Prompt
Type/ Answer
ID
1 Alice has 63 brothers and she also has 66 sisters. How many sisters does Alice's brother have? Before providing answer to this problem, think carefully step by step and double check the path to the correct solution for any mistakes. Provide then the final answer in following form: '### Answer:
THINKING
653
v2 / 67
2 Alice has 62 sisters and she also has 64 brothers. How many sisters does Alice's brother have? Before providing answer to this problem, think carefully step by step and double check the path the correct solution for any mistakes. Provide then the final answer in following form: '### Answer:".
THINKING
654
v2 / 63
3 Alice has 64 sisters and she also has 61 brother. How many sisters does Alice's brother have? Before providing answer to this problem, think carefully step by step and double check the path to the correct solution for any mistakes. Provide then the final answer in following form: '### Answer:
THINKING
655
v2 / 65
4 Alice has 64 brothers and she also has 61 sister. How many sisters does Alice's brother have? Before providing answer to this problem, think carefully step by step and double check the path to the correct solution for any mistakes. Provide then the final answer in following form: '### Answer:
THINKING
656
v2 62
autoregressive manner. It is still unknown for ol class of models whether RL unknown amounts of synthetic data presumably of math and logic type is executed during pre-training or is rather a part of post-training). ol ol-preview is a clear exception and has robust performance close to across all AIW+ variations. Remarkably, ol-mini coming presumably from the same model class does not show same robustness its performance is comparable to standard LLM generation far below ol-preview, settled close (Fig. 10 (A)) and exhibiting fluctuations as usually observed in our study (Fig. 10 (B)).
On AIW Colleague Circle problem, we again observe the already familiar breakdown pattern for all tested models including ol-preview (Fig. [1). While we saw ol-preview solving all the posed AIW versions far robustly with high correct response rates close to 1 across all variations, here we observe rates below (p=0.8) and more importantly, strong fluctuations, eg. < 0.6 on variation vs p > 0.8 on variation 4 (Fig. (B)). With only one single natural number varied the problem template, this provides evidence that also larger scale models of ol class are not robust to problem structure and difficulty preserving variations, hinting on generalization deficits in rather simple setting (as compared to claimed capabilities to robustly solve problems in graduate and olympiad level math setting, which are far above the level here). Interestingly, ol-mini does not undergo such strong collapse as AIW+ (Fig. 10 which might be further evidence for AIW Colleague Circles difficulty being rather moderate, as we observe ol-mini usually lagging far behind ol-preview on AIW problem versions lested here.
We provide also results on testing further reasoning models on both ATW+ and ATW Collegue Circles, complemcting Fig. 5 from main text.
As evident from Fig. 12 and 13 despite their high scores standardized reasoning benchmarks like MATH500, AIME24/25 25 and GPQA-diamond, most reasoning models still suffer from strong fluctuations across AIW problem variations. Notable exception is again ol-preview, although also this model exhibit significant fluctuations and decreased average correct response rates on AIW Circles Colleagues. This shows that claims of robust problem solving of olympiad or graduate level as signalled by strong performance on reasoning benchmarks are not sustainable, as AIW problems where models show lack of robustncss are far below these levels. On the other hand, reasoning models indeed strongly improve in robustness and average correct response rates compared to tested conventional SOTA LLMS as evident from Fig. Remarkable
31

---

Published in Transactions on Machine Learning Research (01/2026)
Table 9: AIW Ext variations 1-4, Thinking v2 prompt type.
Var. Prompt
Type/ Answer
ID
1 Alice and Bob are sister and brother. Alice has sisters and Bob has brothers. How many brothers does Alice have? Before providing answer to this problem, think carefully step by step and double check the path to the correct solution for any mistakes. Provide then the final answer in following form: ### Answer:
THINKING
264
v2 //
2 Alice and Bob are sister and brother. Alice has sisters and Bob has brothers. How many brothers does Alice have? Before providing answer to this problem, think carefully step by step and double check the path to the correct solution for any mistakes. Provide then the final answer in following form: '### Answer:
THINKING
266
v2 /23
3 Alice and Bob are sister and brother. Alice has sister and Bob has brothers. How many brothers does Alice have? Before providing answer to this problem, think carefully step by step and double check the path to the correct solution for any mistakos. Provide then the final answer in following form: '### Answer:
THINKING
268
v2 /5
4 Alice and Bob are sister and brother. Alice has 3 sisters and Bob has 1 brother. How many brothers does Alice have? Before providing answer to this problem, think carcfully step by step and double chock the path to the correct solution for any mistakes. Provide then the final answer in following form: *## Answer:'.
THINKING
270
v2 /
5 Alice and Bob are sister and brother. Alice has sisters and Bob has 3 brothers. How many brothers does Alice have? Before providing answer to this problem, think carefully step by step and double check the path to the correct solution for any mistakes. Provide then the final answer in following form: '### Answer:".
THINKING
455
v2 /44
6 Alice and Bob are sister and brother. Alice has sisters and Bob has brothers. How many brothers does Alice have? Before providing answer to this problem, think carefully step by step and double check the path to the correct solution for any mistakes. Provide then the final answer in following form: '### Answer:".
THINKING
456
v2 /
32

---

Published in Transactions on Machine Learning Research (01/2026)
Table 10: ATW Friends variations 1-4
Var. Prompt
Type/ Answer
ID
1
Alice has 3 male friends and she also has 6 female friends. All mentioned persons are friends with each other and have no other friends aside. How many female friends does male friend of Alice have? Before providing answer to this problem, think carefully step by step and double check the path to the correct solution for any mistakes. Provide then the final answer following form: "### Answer:
THINKING
577
v2 /
2
Alice has 2 female friends and she also has male friends. All mentioned persons are friends with each other and have no other friends aside. How many female friends does male friend of Alice have? Before providing answer to this problem, think carefully step by step and double check the path to the correct solution for any mistakes. Provide then the final answer in following form: *### Answer:
THINKING
580
v2
3
Alice has female friends and she also has male friend. All mentioned persons are friends with cach other and have no other friends aside. How many female friends does male friend of Alice have? Before providing answer to this problem, think carefully step by step and double check the path to the correct solution for any mistakes. Provide then the final answer in following form: "### Answer: '.
THINKING
581
v2 /5
4
Alice has 4 male friends and she also has female friend. All mentioned persons are friends with each other and have no other friends aside. How many female friends does male friend of Alice have? Before providing answer to this problem, think carefully step by step and double check the path to the correct solution for any mistakes. Provide then the final answer in following form: "### Answer:'.
THINKING
582
v2 /
5
Alice has 2 male friends and she also has 3 female friends. All mentioned persons are friends with each other and have no other friends aside. How many female friends does male friend of Alice have? Before providing answer to this problem, think carefully step by step and double check the path to the correct solution for any mistakes. Provide then the final answer in following form: "### Answer:'.
THINKING
583
x2 /
6
Alice has 5 female friends and she also has 3 male friends. All mentioned persons are friends with each other and have no other friends aside. How many female friends does male friend of Alice have? Before providing answer to this problem, think carefully step by step and double check the path to the correct solution for any mistakes. Provide then the final answer in following form: "### Answer:'.
THINKING
584
v2 //6 6
33

---

Published in Transactions on Machine Learning Research (01/2026)
Table 11: AIW Plus variations 1-6.
Var.
Prompt
Type/Answa
ID
1
Alice has sister and 1 brother in total. Her mother has 2 brothers. She also has sister who does not have children and who has 6 nephews and nieces in total. Alice's father has 2 sisters. Hc also has brother who has 5 nephews and nieces lotal, and who also has 2 sons. How many cousins does Alice's sister have? Before providing answer to this problem, think carcfully step by step and double check the path to the correct solution for any mistakes. Provide then the final answer in following form: "### Answer:
THINKING
559
V2/7
2
Alice has 2 sisters and 1 brother in total. Her mother has 2 brothers. She also has sister who does not have children and who has 6 nephews and nieces in total. Alice's father has sisters. He also has a brother who has 4 nephows and nieces in total, and who also has son. How many cousins does Alice's sister have? Before providing answer this problem, think carefully step by step and double check the path to the correct solution for any mistakes. Provide then the final answer in following form: '### Answer: ".
THINKING
560
V2/3
3
Alice has 2 sisters and 1 brother in total. Her mother has 2 brothers. She also has sister who does not have children and who has 7 nephews and nieces in total. Alice's father has sisters. He also has a brother who has 5 nephews and nieces in total, and who also has son. How many cousins does Alice's sister have? Before providing answer to this problem, think carefully step by step and double check the path to the correct solution for any mistakes. Provide then the final answer in following form: *### Answer:
THINKING
561
V2/5
4
Alice has sister and brothers in total. Her mother has 2 brothers. She also has sister who does not have children and who has 6 nephews and nieces in total. Alice's father has sisters. He also has brother who has 5 nephews and nieces in total, and who also has 1 daughter. How many consins does Alice's sister have? Before providing answer to this problem, think carefully step by step and double check the path to the correct solution for any mistakes. Provide then the final answer in following form: "### Answer: '. Alice has sisters and 1 brother in total. Her mother has 2 brothers. She also has sister who does not have children and who has 6 nephews and nieces in total. Alice's father has 2 sisters. He also has a brother who has 5 nephews and nieces in total, and who also has son. How many cousins does Alice's sister have? Before providing answer to this problem, think carefully step by step and double check the path to the correct solution for any mistakes. Provide then the final answer in following form: '### Answer:
THINKING
562
V2/2
5
THINKING
563
V2/4
6
Alice has 1 sister and 1 brother in total. Her mother has 2 brothers. She also has sister who does not have children and who has 6 nephews and nieces in total. Alice's father has sisters. He also has a brother who has 5 nephows and nieces in total, and who also has daughter. How many cousins does Alice's sister have? Before providing answer to this problem, think carefully step by step and double check the path to the correct solution for apy mistakes. Provide then the final answer in following form: "*## Answer: !.
THINKING
564
V2/6

---

Published in Transactions on Machine Learning Research (01/2026)
Table 12: AIW Circles Colleagues variations (omitting variations 2-5 for better readability)
Var.
Prompt
Type/Answer
ID
1
Alice has 3 male colleagues and she also has 6 female col- leagues in total. All these mentioned persons in the circle around Alice are colleagues of each other. Bob has 2 female colleagues and 1 male colleague in total. All these men- tioned persons the circle around Bob are colleagues of each other. The people in the circle around Bob do not have other colleagues aside with the only exception of Matilda. She is colleague of Bob, being part of Bob's circle, and she is also colleague of Alice, being part of Alicc's circle. All the mentioned persons have no colleagues beyond the already described group of people. How many female colleagues does Matilda have? Before providing answer to this prob lem, think carefully step by step and double check the path the correct solution for any mistakes. Provide then the final answer in following form: '### Answer:
THINKING
637
V2/7
2
THINKING
638
V2/3
3
THINKING
639
V2/5
4
THINKING
640
V2/2
5
THINKING
641
V2/4
6
Alice has 3 male colleagues and she also has 5 female col- leagues in total. All these mentioned persons in the circle around Alice are colleagues of each other. Bob has 2 female colleagues and 1 male colleague in total. All these men- tioned persons the circle around Bob are colleagues of each other. The people in the circle around Bob do not have other colleagues aside with the only exception of Matilda. She is colleague of Bob, being part of Bob's circle, and she is also colleague of Alice, being part of Alice's circle. All the mentioned persons have no colleagues beyond the already described group of people. How many female colleagues does Matilda have? Before providing answer to this prob- lem, think carefully step by step and double check the path the correct solution for any mistakes. Provide then the final answer in following form: *### Answer: !
THINKING
642
V2/6
35

---

Published in Transactions on Machine Learning Research (01/2026)
0.00
0.25
0.50
0.75
1.00
1.2
gpt-4o-2024-05-13
claude--opus-2022400
gpt-4-0613
dbrx-instruct
1lama-2-7b-chat
gpt-4-0125-preveee
llama 3-8b--hat
gemma-2b-it
@wen1.5-7b-cha
gemini-pro
gpt-3.5-turbe-0125
gpt-4ttubo-2024-004
88 54
open- -pen-mixtral-887b
5-33b-hhtt
qwen1.5-7205 5-72b-chat
0.4
wwenn.5-14b-chat
soude-3-halku22224000
1lamma2--3b-chat
open-mixtral-8x222
open 7b
llama-3-70b-che
gemma-7ba
command plus
dlaude-3 3- 3- nonnet-20240229
72biitrutt
mistral--arge-2402
0.2
G9744
GP74
Cluded
Uama-3
Gaude:3
(a) Correct response rales for RESTRICTED prompt (b) Strong fluctuations across AIW variations 1-4, RE- type, averaged across AIW var 1-4
STRICTED prompt type
Figure 7: Correct response rates on RESTRICTED prompt type. The prompt type enforcing output only final answer without any further text was used as further control. (a) Correct response rates averaged over variations 1-4 resemble behavior with STANDARD and THINKING types, while looking fluctuations across variations 1-4 in (b) reveals stronger models' lack of robustness compared to other prompt types (see for comparison Fig. 3). We thus used THINKING and THINKING v2 prompt types across main experiments not to put models into disadvantage on AIW testing. See also Suppl. Fig.
is that this improvement in robustness is also achieved by distilled reasoning models that use only SFT reasoning traces in single stage post-training. Models like -obittileed-llla------ Guo et al. (022 (distilled on 800k closed data), Open Thinkerine Team (2025a b) (distilled on OpennTooghtss- 14k, 3 epochs), S1.1-32B Muennighoff et al. (2025) (distilled on 1k DeepSeek-RI data), LIMO-32B Ye et al. (2025) (distilled on 0.8k samples mix from real and DeepSeck-RR R.1 generated data), having either Llama 3.3 or Qwen 2.5 a base, are increasing their performance strongly compared to their base instruct models, reaching up levels comparable with DeepSeek R1 671B, which uses both SFT and RL during multi stage training, also outperforming closed reasoning models like ol-mini. We observe remarkable similarity in the distribution shape of correct response rates between larger scale SFT only distilled models (S1.1 32B, Open 32B, R1-Llama-70B) and DeepSeek-R1 or ol-mini and 03-mini that use RL (Suppl. Fig. 13)
We thus can obtain further confirmation that AIW problem versions, despite being simple far below olympiad or graduate level problems that advanced models claim be able handle via their performance on standardized benchmarks can reveal lack of model robustness and generalization deficits, also when used for testing most recent model generation, including advanced reasoning models of ol type. This shows patential for model ranking and construction of benchmarks that are capable to detect model weaknesses that remain undiscovered when testing via standardized benchmarks, which use more complex problems without though mcasuring performance on controlled variations of those.
C.2 Comparing models of various scale
We compare open weights model families Qwen 2.5 (Fig, 15 and Llama 3.1 (Fig. 14]. which offer a broad range of pro-training scales (Qwen 2.5 1.5B-72B and Llama 3.1 8B-405B model scales). From results obtained from testing on AIW original and AIW Ext, we observe expected dependence of performance on pre-training scale. In line with previous observations, small scale models show strong breakdown with correct response
36

---

Published in Transactions on Machine Learning Research (01/2026)
0.75
0.00
0.25
0.50
0.75
1.00
gpt-4o-0224-0-113
gpt-4o-2024-05-13
cloude:3-opu--2024022
(auud=3-opus-2024000
gpt-4-0613
gpt-4-0613
gpt-4-turbo-20244000
gpt---0155prevview
9pt-4-0125ppreveew
llama-2-70b-chat
llama-3-70b-chat
llama-2-7b-chat
llama-2-70b-chat
qwen1.5-1.8b-chat
2b-it
llama-2-7b-chat
qweni.5-1.8b-cher
llama-3-8b-chat
llamm-3-8b-chat
mistral-lage+22402
llama-2-13b-hh
gemmma-bb-t
dbrx-instruct
llama-2-13b-chat
liama-3-70b-chat
claud
gpt-3.5-turee 5-turoo-100
qwen2-72b-instruutt
owen2-72b-instruct
open-mixtral-8x22b
open-mixtral-Bx226
open-mistral-76
gemini-pro
claude-
3-haiku-20244000
Bruude3--onnet-222222
wwwn1.5-7b-de..
-0--3.5-uubb-
gemini-pro
apen-mixtral-8:44
open
dbrx-instruct
gemma- 7b-it
gemma-7b-it
command
open-mixtral-8yk
mistral-4arge-2402
command-r-plus
mwen1.5-14b-chat
5-32b-hatt 5-abch.t daude-3-hakkee Buude-3-nakku22224000
STANDARD THINKING v2
gwenl.5-32b-chat
THINKING
wwenl.5-72b-chat
THINKING
gwen1.5-7b-chal
(a) Correct response rales THINKING v2 vs. STAN-(b) Correct response rales THINKING v2 vs. THINKING prompt lype, averaged across AIW var 1-4
DARD prompl lype, averaged across AIW var
Figure 8: Control comparison of correct response rates averaged across AIW variations 1-4. (a) THINKING
v2 vs. STANDARD, (b) THINKING v2 vs. THINKING prompt types. THINKING provides better average correct response rates for tested models. We thus used THINKING prompt types for main and control AIW problem. THINKING and THINKING v2 show highly similar behavior across tested models (b) and can be used interchangeably (THINKING v2 only difference to THINKING is the explicit phrasing 'step by step', Suppl Tab.
experiments to ensure lested models are not disadvanlaged
rates close to 0 across variations. With larger scale, models exhibit higher correct response rates, while showing strong performance fluctuations across problem structure difficulty preserving variations. We see again strong performance fluctuation also across problem versions, for instance for Llama 3.1 405B having much higher correct response rate for AIW original compared AIW Ext (Fig. 14], although both problems have highly similar structure. Similar to obscrvations Claude 3.5 Sonnet (Fig. ??), this might be again the case of training data containing AIW original instances, with strong diminished performance ATW Ext pointing to generalization deficits.
For smaller scale models below 8B, the breakdown on AIW original and AIW Ext is so strong that correct response rates do no reflect differences scale, most models at that small scales being close to This is in line with our observations that the few models capable showing significant non-zero correct response rate for the AIW problem are residing on the larger scales. Observing the performance on the AIW problem across various models,
evidence that in general, smaller scale models (known to have been overtrained large token budgets of > 2T' tokens) that score high on standardized reasoning benchmarks, creating illusion of having similar capabilities as larger scale models that have similar scores, suffer severe collapse on the AIW problem. No small scale model can even remotely approach the performance on AIW problems shown by better performers residing at larger scales. Better performing larger scale models, while exhibiting strong fluctuations on problem irrelevant variations, do show significant correct response rates across AIW problem variations, while smaller scale models drop across variations close to zero, staying far below p = 0.1 average. This valid for both conventional language and reasoning models.
37

---

Published in Transactions on Machine Learning Research (01/2026)
ATW Ext Alice and Bob, Alice's Brothers, Variations 1-6, Prompt (Ds. 264 266 268 270-455
Afice and Beb are sister and brother.
Variation Alkce has sisters and Bob 6 brodhers. [Correct anower: 7]
Variztion Alice has 2 sisters and Bob has brothers. [Correct answer: 3]
Variation Alice has sister and Bob has 4 brothers. [Correct answer: 5
Variation Alice has 3 sisters and Bob has 1 brother. [Correct answer: 2
1.0
Variation Alice has sisters and Bob has brothers. (Correct answer:
Variation
sisters
(Correct answer:
How many brothers does Alice hove?
0.8
alw ext v1, 264
0.6
alw ext V2, 266
alw cxt V3, 268
alw ext V4. 270
alw ext V5. 455
ale ext VS, 456
0.4
0.2
0.0
o1-preview
o1-mini
DeepSeek-3 6708
Claude-3.5 5
Claude-3 O
Llama-3.1 405b
Qwen2.5 72b
GPT-4o
Figure 9: Strong fluctuations of correct response rates on variations of ATW Ext (a color per each variation 1-6) are also exhibited by recent reasoning models like ol-mini. Recent large-scale standard LLMs like DeepSeek 3 670B, Llama 3.1 405 and Qwen 2.5 72B show also strong fluctuations, with lower correct response rates compared to ol-mini. Claude 3.5 Somnet and Claude 3 Opus collapse strongly with low correct response rates. Despite its simplicity, AIW Ext can thus reveal lack of model robustness and generalization deficits also in most recent SOTA LLMs that claim very strong function via standardized benchmarks. ol-preview is an exception that is able to handle the problem robustly, without any fluctuations.
C.3 Debunking strong function claims
Here we show examples of debunking strong function claims put forward based on standardized benchmarks by testing and comparing models on AIW problems.
NuminaMath-7B & claim of olympiad level problem solving. We provide example of debunking overblown claims in case of NuninaMath-77 that was ranked 1st at the AIMO competition in July 2024, solving 29/50 private set problems of olympiad math level. The claim was widely put forward that the model is capable of solving high school olympiad math probloms. AIW has arguably average elementary school level and does not require any advanced math knowledge. We tested NuminaMath-7BB on AIW and observed a strong collapse of this model on ATW problem, with correct response rates close to 0 across ATW variations 1-4 (Fig. ??). Using AIW Light control problems, we can also see that NuthnaMath-71B can handle all the low level operations (elementary arithmetic, attribute binding, etc) and knowledge required to deal with family structure, ruling out that those are the issues. Using the ATW problem setting, we thus can contradict. the strong claim of being capable to deal with olympiad level high school math
ol-mini & claim of matching larger scale with smaller ones. ol-mini was announced together with ol and ol-preview as a smaller scale member of the new class of reasoning models. ol-mini was reported to obtain very strong scores on standardized benchmarks. Basod on this, claims of strong function were put forward, specifically in comparison to the larger scale ol-preview. In original openAI announcement (Sep. 2024), ol-mini was reported to outperform larger-scale ol- ol-preview on high school AIME math competition and coding tasks. This led to speculations that ol-mini can match or even outperform ol-preview as robust problem solver, a claim that is often put forward for models on smaller scales trained with substantial compute or obtained via distillation or other compression techniques from their larger scale counterparts.
We tested ol-mini and ol-preview using various AIW problem templates introduced here and the problem structure and difficulty preserving variations. As evident from conducted experiments, ol-mini shows signs of breakdown starting with AIW original, that become more severe on AIW Ext, where fluctuations become
38

---

Published in Transactions on Machine Learning Research (01/2026)
A
0.1
0.2
0.3
0.4
0.5
0.6
0.7
ol-preview
gpt-4-0613
gpt-4o-2024-05-13
meta-llama-3 0tte-llaaaa3.1--05555 1-00--------0b--isstuuuc
claude-3-opus-2024000
(wen2.5-72b-instnun
deepseek-v3 670B
3pt-4o-mini-2224-07-
o1-mini
B
1.0
aie* v1, 559
0.0
560
alw v3, 561
alws 562
eie+ v5. 563
o1-preview
ol-mini
GPT-4
GPT-4a
GPT-4o-mini
Claude-3.5
Cloude-3
Llama-3.1 405b
Qwer2.5 72b
DeepSeek-3 6708
Figure 10: AIW+ correct response rates averaged over variations 1-6 (A) and fluctuations across variations (B). Most tested models undergo further collapse compared to ATW original. ol preview clear exception shows robust ability to solve AIW+ across all variations without fluctuations. ol-mini on contrary collapses close to 0, also showing fluctuations (eg on variation 6 vs others). AIW+ was made intentionally harder than simple ATW. However, models claiming strong function should be able to solve it, as does not involve any higher level logic or math.
stronger, culminating with collapse AIW+, where correct performance rate most variations stays well below p 0.2. ol-preview contrary demonstrates robust problem solving with high correct response rates close to 1 without performance fluctuations across variations on all problem versions (see Fig. [1 for AIW Colleague Circles problem where ol-preview also exhibits breakdown pattern while still clcarly outperforming ol-mini) This is line with our observations reported here wrt. to smaller scale models and also specifically for ol-mini in comparison ol ol-preview, and importantly, showing much higher sensitivity to problem structure preserving variations, pointing to severe generalization deficits, in contrast to ol-preview.
other models (Fig. 10 where ol-mini stays far beyond
Thus, ATW problem templates and the problem structure preserving variations offer a measurement technique that can reveal lack of robustness and model weaknesses in generalization and problem solving that remain undiscovered by standardized benchmarks. We think that conducted experiments can also serve as a vivid warning that many of the claims put. forward for strong model functions cannot be trusted. Such claims often rely on benchmarks that overlook clear function deficits, and simple AIW problem templates with their variations offers tool for systematic, reproducible stress testing and debunkiig of such claims.
C.4 Frequency distribution of natural numbers output and dominance of wrong responses.
To shed more light on modes of correct or wrong responses provided by the models when confronted with AIW problem variations, we show here frequency distribution for natural numbers on the output for AIW variations with higher and lower correct response rates.
As evident from the plots, in higher performance AIW variations (Suppl. Fig. 19 dominants peaks are often positioned on correct answer C-M+1, while for lower performance AIW variations (Suppl. Fig. [8]. dominant peaks fall on wrong answer M. Further, for weaker models, distribution broadens, covering more numbers (eg Llama 8b), while for better performers, responses concentrate on M and M+1, peaking on correct or wrong answer on depending on AIW variation. Remarkably, for lower performance AIW variations (Suppl. Fig. 18 performance cannot be rescued by major voting by similar ensemble like strategies,
39

---

Published in Transactions on Machine Learning Research (01/2026)
0.1
0.2
0.3
0.5
0.6
0.7
1.0
o1-preview
cludee23-pus-222442222
Saude--3---connet22000
20240000
meta-llama-3. 1-105b-instrrct
spt-4o 2024-05-13
dep55eek--3-6708
gpt-4o-min-22240/1
1.0
aiw v1, 637
aiw v2, 638
alw v3,
aiw v4, 640
aiw v5, 641
0.6
aiw v6, 642
0.2
ol-preview
ol-mini
GPT-4o
Claude-3.5
Claude-3 O Llama-3.1 405b DeepSeek-3 670B GPT-4o-mini
Figure 11: AIW Colleagues Circles correct response rates averaged over variations 1-6 (A) and fluctuations across variations (B). Same breakdown pattern is evident in the tested models either overall low correct response rates or strong performance fluctuations across problem structure and difficulty preserving variations as observed on AIW original. Also ol preview exhibits such fluctuations (eg. variation vs variation 1, or 6), providing evidence that also strongest models from ol class are not robust, pointing generalization deficits. ol-mini, which collapsed strongly on AIW+, exhibits here fluctuations much stronger than ol preview, again providing evidence that ol-mini is much weaker than ol preview, contrary to claims relying standardized beachmarks. Strong fluctuations affect most models that oblain significant correct response rates on at least one of the variations, eg. GPT-4o, Claude 3.5 Sonnet, Llama 3.1 405B, DeepSeek v3 671B. GPT-4o-mini collapses entirely. AIW Colleague Circles is harder than simple AIW, but it does not involve any high difficulty logic or math. Models claiming strong function on level of graduate or olympiad math tasks should be able to solve without fluctuations with higher correct response rates across variations.
peaks on wrong response numbers dominate clearly peaks on numbers for correct responses, which would still correspond to commilting wrong answer when performing majority voting.
For the AIW Light problem versions used in control experiments, we observe as expected clear dominant peaks the numbers corresponding for correct responses across all tested models (Suppl. Fig. 20 21 AIW Light problems are successfully solved across all their variations.
We note that distribution characteristics, eg concentration on numbers around the correct answer, height of the peaks, can be a further signature that reflects model's capability to handle the problem. More capable models retain dominant peaks on number corresponding to correct answer with smaller peaks on neighboring numbers, while weak models have large peaks on numbers corresponding to wrong answers or in general broad distribution across all natural numbers below 10. Computing scores from distribution shape can thus also enable model ranking.
C.5 Standardized benchmarks failure
We observe failure of standardized reasoning benchmarks to properly reflect generalization and basic reasoning skills of SOTA LLMs by noting significant disparity between the model's performance on the AIW problem and the scores on conventional standardized benchmarks. All of the tested models report high scores on various standardized benchmarks that claim to test problem solving via reasoning, Our observations of SOTA models breaking down on the simple AIW problem hint that the benchmarks do not refloct deficits generalization and basic reasoning of those models properly. We visualize this failure by plotting scores testod models obtain wide-spread and accepted standardized benchmarks like MML.U versus the performance
40

---

Published in Transactions on Machine Learning Research (01/2026)
1.0
aie 55.
0.6
alm+ v4, 562
0.0
01-preview
a3-mini
LIMG-328
s1.1-328 DeepSeekcR11
al-min
GTuQQwen-30
R1-Liama-708 R1-llama-88
RI-Qwen-328
Rl-Qween778
Figure 12: Strong fluctuations correct response rates on variations of AIW Plus problem (a color per each variation 1-6) exhibited by recent reasoning models. With exception of ol-preview and o3-mini, most reasoning models that show strong performance on standardized reasoning benchmarks reveal inability to cope with the problem robustly when facing slight variations in problem template. Eg, correct response rate drops for R1-Llama 70B from close to 1.0 variation 1 or 6 below 0.4 on variation 4, despite differences between the those being just instantiated numbers. ol-mini undergoes strong overall collapse. Distilled perform on par with DeepSeck-RI or
reasoning models ($1.1 32B, LIMO 32B, outmatch it, despite using for distillation SFT only. Distilled models at larger scales (32B, 70B) perform significantly better than smaller scale 7B/SB models.
observe our proposed AIW problem. As strikingly evident from Fig. 22 there is strong mismatch between high scores MMLU reported by the models and the correct response rates they obtain on AIW. This mismatch and lack of differentiation makes it impossible for a given model to predict from its score on MMLU whether it will suffer brcakdown on a simple problem like AIW, making the score unroliable for measuring core capabilities. Also model ranking fails, as models with similarly high MMLU scores claiming similar function level can have dramatic difference on simple ATW problem. For instance, models like Llama-3-7013, Mistral Large or GPT-4-Turbo come close with their MMLU score to GPT-4/Claude 3 Opus, hinting comparable capabilities, while settled in the crowd of high MMLU low AIW score region (left upper part of Fig. 22) with most other models that achieve very low ATW performance close to 0. This also demonstrates that MMLU, while containing problems of arguably higher difficulty, does not properly reflect deficits in basic model function, as revealed by much simpler AIW problem.
This observation also holds on further standardized reasoning benchmarks like MATH, ARC-c, GSM8K and Hellaswag (Suppl Tab. 13). Also there we see strong mismatch between high benchmark scores reported by many models and the low correct response rates they obtain on AIW (in some cases 0 for models with high standardized benchmark scores), in Figures 24
26
23
25
We see thus that standardized benchmarks fail to properly reflect truc model capabilities to generalize and reason - the majority of the lested models score high on standardized benchmarks, suggesting strong function, while showing extreme low correct response rates on simple AIW problem. Many the models with high scores on standardized benchmarks cannot solve AIW problem single time (c.g. Command R+ is unable solve single AIW problem instance, see Suppl Tab. [3). This discrepancy refutes the claim of standardizod benchmarks to measure correctly current models' core functionality.
41

---

Published in Transactions on Machine Learning Research (01/2026)
Table 13: Performance of tested models on MMLU, Hellaswag, ARC-c, GSM8k and AIW. Correct response rate averaged across AIW variations 1-4, across STANDARD, THINKING and RESTRICTED prompt types.
Model
MMLU
Hellaswag
ARC-c
GSM8k
AIW Cor-
rect aver-
age resp.
rate
gpt-4o-2024-05-13
0.89
95.3
95.9
95.00
0.65
claude33-opus-222242229
0.87
95.40
96.40
95.00
0.43
gpt-4-0613
0.86
95.30
96.30
92.00
0.37
gpl-4o-mini
0.82
89.1
87.5
83.9
0.29
lama-2-70b-chat
0.64
85.90
64.60
56.80
0.30
lama-2-7b-chat
0.55
77.10
43.20
25.40
0.13
dbrx-instruct
0.74
88.85
67.83
67.32
0.11
gp1-4-turbo-2024--44009
0.80
0.10
llama-3-b--hat
0.67
78.55
60.75
79.60
0.05
llaa-3--0b--hat
0.80
85.69
71.42
93.00
0.05
gwen
0.46
46.25
36.69
38.40
0.05
gemma-2b-it
0.38
71.40
42.10
17.70
0.04
Hama-2-13b-chan
0.66
80.70
48.80
77.40
0.03
qwen1.5-4b-chal
0.56
51.70
40.44
57.00
0.02
claude-3-sonnet-
0.79
89.00
93.20
92.30
0.01
20240229
mistral--lare-24402
0.81
89.20
94.20
81.00
0.01
gpt-3.5-turbo-0125
0.70
85.50
85.20
57.10
0.01
gemini-pro
0.72
84.70
77.90
0.01
open-mixxtral-6x22b
0.78
89.08
72.70
82.03
0.01
open-mistral-7b
0.64
84.88
63.14
40.03
0.01
qwenl.5-7b-chat
0.62
59.38
52.30
62.50
0.01
clade-33hhaikn-2224403777
0.75
85.90
89.20
88.90
0.00
open-mixtral--x7hb
0.72
87.55
70.22
61.11
0.00
command-r-plus
0.76
88.56
70.99
70.74
0.00
gwenl.5-14b-chal
0.69
63.32
54.27
70.10
0.00
gemma-7b-it
0.54
81.20
53.20
46.40
0.00
0.77
68.37
65.36
79.50
0.00
qwenl.5-32bbchal
0.75
66.84
62.97
77.40
0.00
42

---

Published in Transactions on Machine Learning Research (01/2026)
1.0
alw
aiw v2. 638
aiw v3. 639
aiw v4, 640
0.6
aiw v5, 641
alw
0.0
01-preview
63-mini LIMD-328
x1.1-328 DeepSonk-R1 6718
ol-mini
OTw-Qwen-32b R1-Ilama-788 R1-Llama-s8 Rl-Qwen-328
R1-Qwan-78
Figure 13: Strong fluctuations of correct response rates on variations of AIW Circles Colleagues problem (a color per each variation 1-6) exhibited by recent reasoning models, including top frontier models like ol-preview and 03-mini (medium). Fluctuations reveal that all reasoning models with strong performance standardized reasoning benchmarks are unable to cope with the problem robustly, being sensitive to slight variations in problem template. Eg, correct response rate drops for o3-mini-medium from above 0.9 on variation 4 to below 0.1 on variation 1, despite differences ween those being just instantiated numbers. Distilled reasoning models (S1.1 32B, LIMO 32B, Thinker--wen-33211 perform on par with DeepSeek-R1 or outmatch it, despite using for distillation SFT only. Distilled models at larger scales (32B, 70B) perform significantly better than smaller scale 7B/8B models (with exception of R1--lama-SBB vs R1-Llama-700p 70B). Larger scale distilled models, DeepSeek-R1, 03-miní and ol-mini show remarkable ble similarity in the distribution of correct response rates across problem variations.
1.9
VL 215
22.22.22 206
NAVI 533
0.9
209
M1 (1. 150
457
5.8
als act v4, 270
2lA 55, 655
409
0.-
on 314
142
0.7
0.4
2.5
0.5
0.2
0.0
neb Ilarn 322 Ad53 Instruct
meta-llama 1-33--00-inni hok-instruct
meto famm- 3.3 3-80 instrctt
0.2
0.2
0.2
0.0
0.0
Figure 14: Model comparison and ranking, example of various scales of Llama 3.1 family. Models were tested on (A) AIW, (B) AIW Ext and (C) AIW Friends. Effect of scale is evident when comparing distribution of correct response rates across problem variations and correct response rate averaged across all problem versions (D), hinting advantage of larger scale pre-training. Tack of model robustness evident from strong collapse models suffer on AIW Friends, which is structurally similar to AIW and AIW ext.
C.6 Comparing AIW testing with GSM Symbolic
Recently, Mirzadeh et al. (2244) made use of similar approach to create variations from templates of GSMSK problems. While also measuring fluctuations across problem structure preserving variations. this work does
43

---

Published in Transactions on Machine Learning Research (01/2020)
B
0.9
0.8
0.7
0.6
0.4
8.3
0
0.2
0.1
ow0025 1.50
C1A0115 De0th Sm 300
SeeiD Werk D Lo Sweds swerl D4
Figure 15: Model comparison and ranking, example of various scales of Qwen 2.5 model family. Models were tested on (A) AIW, (B) AIW Ext and (C) AIW Friends. Effect of scale is evident when comparing distribution of correct response rates across problem variations and correct response rate averaged across all problem versions in (D), hinting advantage of larger scale pre-training. Tack of model robustness is evident. from strong collapse models suffer on AIW Friends, which is structurally similar to AIW and AIW ext.
not provide evidence that stronger models undergo function breakdown or exhibit generalization deficits on simple problems. The evidence obtained in GSM Symbolic suggests in fact that models like GPT-4o or Llama 8B can handle such problems well without exhibiting strong fluctuations or low correct response rates across variations (see A. Fig. 29.. In contrast, we measure strong fluctuations also for advanced L.Ms pretrained at largest scales (GPT-4/44) 40, Claude 3 Opus, Claude 3.5 Sonnet) using variations in very simple AIW template, which clcarly shows lack of model robustness and generalization deficits also in those supposedly strong models. We hypothesize that inability to observe such strong fluctuations in GSM Symbolic might also be due to training set contamination, as GSM8k is a well known publicly available benchmark where test set might either accidentally or intentionally leak into training, or be used for synthetic data generation close to the test set mixed into training.
D Examples of correct and failed responses
We provide all collected model responses we obtained during this study in the collected_responnes rrepponses folder in the ATW repo Here we also showcase some correct and incorrect answers as an example (see Suppl. Figs. 30
33
31
32
34
E
Confabulations and overconfident tone accompanying wrong answers
Overconfident tone. In ideal scenario, if LLM cannot correctly solve the AIW problem, it should at least be capable of expressing high uncertainty about the provided incorrect solution to the user. We used CONFIDENCE prompt type (see Suppl Tab. 3 for AIW problem to see how confident tested models are in their wrong solutions.
From our experiments we can see that LLMs most of the time express high certainty even if their answers are completely wrong, thus mediating strong confidence (see Fig, 35). The models also use highly persuasive tone to argue for the expressed certainty and correctness of the provided wrong solutions, using words like 'highly confident', 'definitive answer", or 'accurate and unambiguous". We see also strong overconfidence expressed in multi-turn interactions with models, where user is insisting on solution provided being incorrect, and observe there high resistance of models to revise their decisions, which was already referred to as "stubbornness" in other works Zhang et al. 2024 (see Suppl Sec. ?? and also data provided in the AIW repo)
Confabulations. In our experiments we observe frequent tendency of those tested models that show strong reasoning collapse and produce frequent wrong answers for AIW problem to generate at the same
44

---

Published in Transactions on Machine Learning Research (01/2026)
Cerrect res rotes AM vartation 1-4, THAMMING vs promant
2.5
Variations
Varlation Alice brothers and she also has
[Comrect answe:?
Mariation Alice has sisters and she also has brothers. [Comnet amoowe:: 9
AIW Variatitonn, Original and ATW Control
Vartattoo Alicc has sisters and she also has brother. [Corest answer 5
0.4
Vertsaion Alice han brothers and sie who has sister. [Eorreet arrowe: 2]
Template: Alice has N brothers and she also has M sisters. Variations 1-4; changing N, M <- 7. Correct responses: C <=>
riew many soarss does Altors brother Anowr
0.2
AIW Original (SOTA LLM breakdown) How mony sisters does Alice's brother have? [correct: C=M +1 (A)
MuminaMtti-i
AIW Light Control (SOTA LLM succeed)
How many brothers does Alice's sister have? [correct: C=N] (B)
How mony siblings does Alice have?
[correct: C- M] (C)
Ligat Oretett Korily Rlork Endeen Bradien
Light Central Alers Sead Eldingg
Sight Conted Oh
How mony girls are there in total?
[correct: C= (D)
3.0
1.0
1.0
carnec vena serernaeee
0.66
Prompt type
THINKING Before providing to this prablem, think carefully step by step and double check the path to the correct solution for any mistakes. Provide then the final answer follewing form: "##'t Answer: !.
00.4
0.4
3.4
0.2
0.
0.0
www.swWoom- 7h
Man TT
Testing NuminaMath-7BB which claimed olympiad high school level math problem solving via 1st rank in AIMO competition, by using AIW original and AIW Light control problems. (A) Very low correct response rates across AIW problem variations 1-4 (THINKING v2 prompt type). NuminaMath-7B suffers strong collapse on simple ATW problem that has average elementary school level. This reveals clear deficits in generalization and even basic reasoning, refuting the claim of strong function on special domain of math problems. For each AIW variation, 100 trials were executed to estimate correct response rate and its variance. AIW Light experiments test various operations and knowledge required for solving AIW (B) Asking for Alice's sister's brothers number (requires understanding entity 'Alice's sister", binding female attribute to Alice and realizing Alice and her sisters share same brothers) (C) Asking for Alice's siblings number (requires understanding entity "siblings', accessing numbbers of Alice's brothers and sisters, executing addition operation) (D) Asking for total girls number (requires binding female attribute to Alice via pronoun "shc" and to her sisters, selecting and exccuting the correct arithmetic sum operation count all the obtainod girls). Across all AIW Light control problems, NuminaMath-77B obtains correct response rates much higher than for ATW original, some being close to 1. This proves that handling language, basic family structure, parsing nunbers, and handling elementary arithmetics like counting are all intact and not the cause for failures in AIW (A). In (D), strong fluctuations despite only differences being instantiated numbers across variations of the same simple problem hint again on severe generalization deficits. The reason for the collapse on the AIW original problem is thus failure in inferring the problem structure, pointing to generalization and reasoning deficits, which is in contrast to claims made for NuminaMath-77B as a strong high school olympiad level math problem solver, based on AIMO competition benchmark, which did not reveal such flaws.
Figure 16:
time persuasive sounding pseutdoccppannttios to back up their incorrect answers. We term here such pseudo-explanations confabulations, and present a selection of those as examples.
Such confabulations can contain mathematical calculations or other logic-like expressions and operations that make little or absolutely sense given the problem to be solved, see examples for Olmo-7B. Aimo-7B, Fig. 36 and Command R+, Fig. 38
Further confabulations make use of various social and cultural norm specific context to argue for the posed problem to be inappropriate to solve or to provide non-sense arguments for various incorrect answers. There are many such examples that we have observed, we present here only a small selection.
Maddellma-7O--istteee for instance seems to be specifically prone to claim ethical or moral reasons for not addressing the problem correctly, in the presented example inventing out of nowhere person with
45

---

Published in Transactions on Machine Learning Research (01/2026)
8tM v1. 205
s/W ext v1. 204
3/++ V1,359
S/W ext 268
alw V4, 188
ai ext v4, 270
aiw V4, 582
457
v5, 455
A
B
SIW ext 456
C
1.0
1.0
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
cl-praview
a1-mini
01. preview
02 minl
01.prexiew
Figure 17: Testing ol-mini on the subject of the claims to match or outperform its larger scale counterpart, ol-preview. Comparing ol-preview and ol-mini using variations of AIW original (A), AIW Ext (B) and the harder AIW+. ol-mini shows progressive signs of breakdown, with fluctuations already visible in (A), becoming more apparent in (B) and strong collapse in (C), with correct response rates going down. ol-preview shows in contrast robust problem solving without fluctuations, keeping high correct response rates close to across variations. This clear demonstrates ol-preview superiority contrary to claims of ol-mini strong function that were relying on various standardized benchmarks. See also Fig. for comparison on AIW Colleague Circles problem.
gpt- cpt-4o-2022-005
0-033
2bude33-pus-22244222
lamaa3-70b-chat
Iiamo 3 Bb-chhtt
Incorrect
0.9
0.9
0.9
Correct
0.8
0.0
0.8
0.0
0.8
0.7
0.7
0.7
0.7
0.6
0.5
0.5
0.5
0.5
0.5
0.4
0.4
0.4
0.4
0.4
0.3
0.2
0.1
6.1
0.1
0.1
0.1
0.0
0.0
3
Figure 18: Frequency distribution of output numbers in models' responses. Shown are numerical outputs for AIW Variation 3, THINKING prompt type (prompt ID 64), that has correct answer C-M+1=5, with M=4 number of sisters of Alice. For this AIW variation, models have low performance (see also Figure D.). Correspondingly, peaks are on the dominant wrong response, R=M=4. For this low performance variation, performance cannot be rescued by majority voting or other simple ensembling strategics, as also for better performing models like GPT-4o, there dominant, peaks wrong numbers that would overrule less dominant peaks for correct numbers. Weaker models, eg Llama 3 8B, show also broader distribution. Distributions were computed over 60 trials exccuted for cach model, taken from original collected responses data.
Down syndrome and then pointing out that question has to be modified to be addressed due to potential perpetuation of harm towards individuals or groups, which has nothing to do with original task, Fig. 37
Another example are confabulations provided by Command Plus. These confabulations use concepts of gender identity such as non-binary gender concepts related to inclusion to cultural context dependent.
46

---

Published in Transactions on Machine Learning Research (01/2020)
3pt44e-2024-05-2
opt-4-0613
2luude-3-uu-22224022
larma33700-shht
liane 1arh---0l-chstt
1.0
1.0
1.0
1.0
1.0
Incorrect
Correct
0.8
0.8
0.8
0.8
0.7
0.7
0.7
0.7
0.7
0.5
0.4
0.4
0.4
0.4
0.4
0.3
0.3
0.3
0.3
0.3
0.1
0.1
0.1
0.1
0.1
0.0
0.0
0.0
0.0
0.0
Figure 19: Froquency distribution of output numbers in models' responses. Shown numerical outputs for AIW Varialion 4, THINKING prompt lype (prompl ID 70), that has correct answer C-M+1-2, with M=1 number of sisters of Alice. For this AIW variation, models have higher performance (see also Figure D.). Correspondingly, peaks for better performing models (eg GPT-4o, GPT-4, Clande Opus 3) are on the dominant correct response, R=M+1-2. For models with worse performance, peaks are on the dominant wrong response, R=M=1. For weaker models, eg Llama 8B, also broader distribution over numbers appears, with further wrong clear peaks that further away from C=M+1 (eg M-4). The distribution shape and peaks nature be thus used signature of model's capability to handle the problem, also allowing model ranking dependent peak types and distribution sharpness. Distributions were computed over 60 trials executed for each model, taken from original collected responses data.
gg2-4o-222405-113
0p3-4-0613
cluse-3.quus-22240225
1am--3-70b-chat
laa-a-3.bbcchat
1.0
1.0
1.0
1.0
2.0
Incorrect Cornect
0.8
0.7
0.7
0.7
0.7
0.7
0.6
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.4
0.3
0.3
0.3
0.3
0.3
0.1
0.0
0.0
0.0
0.0
0.0
Figure 20: Frequency distribution of output numbers in models' responses. Shown numerical outputs for AIW Light Family, Variation 3, THINKING prompt type (prompt ID 273), that has correct answer C=5 (number of Alicc's brothers). For this AIW Light version, all models have high performance. Corrospondingly, peaks are on the dominant correct response, R-5. However also here, weaker models like Llama 3 8B show broader distribution with non-vanishing peaks besides the correct response (eg R=0, R=2) hinting on their weaker capabilities to deal robustly with the problem. Distributions were computed over 60 trials executed for each model.
family identification in the provided wrong reasoning leading to incorrect answers. In the attempt to solve the problem, the model first fails to provide obvious common sense solution and then goes to describe potential scenarios where brothers and sisters may self-identify as non-binary, although providing information on brothers and sisters in the problem usually means via common sense that those persons self-identify correspondingly to their known status as brother or sister (while Alice is clearly identified via "she" pronoun). Model thus clcarly fails to grasp that problem structure has nothing do with the social and cultural norms. The solutions derived by the model from considering those factors that are far beyond Occam's razor and common sense inherent to the simple AIW problem all lead to wrong answers and generate more confusion,
47

---

Published in Transactions on Machine Learning Research (01/2026)
gpt-4e-2024-05-
opt-4-0613
lamm-3370bbshat
liane 1arh---0l-chstt
1.0
1.0
1.0
1.0
1.0
Incorrect
Cornect
0.8
0.8
0.8
0.8
0.7
0.7
0.7
0.7
0.7
0.5
0.5
0.4
0.4
0.4
0.4
0.4
0.3
0.3
0.3
0.3
0.3
0.1
0.1
0.1
0.1
0.1
0.0
0.0
0.0
0.0
Figure 21: Froquency distribution of output numbers in models' responses. Shown numerical outputs for AIW Light Arithmetic, Varialion 3, THINKING prompt lype (prompt ID 279), that has correct answer C=5 (total number of Alice's siblings). For this AIW Light version, all models have high performance. Correspondingly, peaks are on the dominant correct response, R=5. However also here, weaker models like Llama 8B show broader distribution with non-vanishing peaks besides the correct response (eg R=4, R-6) hinting on their weaker capabilities deal robustly with the problem. Distributions were computed over trials executed for each model.
AIW Correct answers vs MMLU
1.0
QPT Cl 3-a 3-O
-PI---
MisriaLL
0.8
3aarr 3-10b
DSRX-hratnct
3-th
L.7b
0.6
owen 7.
Owen1 5-hh 1Lamm277
Owee115:1.5b
Gemma-2b
freatrll- aarg-
gaminié-ro
owmmma-7b-t
gel- 4----tt---2000000000000
owent 5-40-chet
qwant. 5- 5-72b-clel
gui-4-0513
44b-heet
claudda-S-op 202242222222
0.2
commandd plus
opwn-miittad-7
chat
lama 3 700 chat
derovenntuct
owent Bb-chall
qwent 5 5-32b-chet
lama 2-12 2-133-chat
lama 2- chat
open 20240807
qweet.6
go1-4u-2004-05-1
0.0
0.0
0.2
0.4
0.6
0.8
1.0
AIW Correct response rate
Figure 22: Failure of standardized benchmark MMLU to properly reflect and compare model basic reasoning capabilities shown by strong discrepancy between AIW correct response rate MMLU average score. Many models, Command R+, score 0 on AIW, but have high MMLU score.
while again keeping the persuasive tone that suggests that model is on some right path provide the correct solutions (Fig. 39
For more illustrative examples, the raw data interactions with the models collected in AIW repo)
Impact Statement
This paper presents work whose goal is to advance the field of machine learning. There are many potential socictal consoquences of our work, one of which we would like emphasize. L.MSs and LRMs constitute an important instance of foundation models models that sustain claim of strong generalization and transferability across novel scenarios and tasks. These strong claims rely mostly evaluations performed
48

---

Published in Transactions on Machine Learning Research (01/2026)
Correct answers ve GSMBK
Dieilheup 3-70
Metri Gparatihho
DBRXinstuct
D577
0.6
OPT-8.TT
Owan1.55444 5-4b
Liamm22-00
GSMOK
0.4
Owent 5-1
2:7b
Genma-2o
0.2
AIW Correct response
dlaud 3 sornet 20240229
Ibmr 13 chat commandi plus
qwent 5. 4o chat
diaute 3 halku 20040077
cuide 3 2024222222229
tema-3-7lis-ctet
1.5-115 1.5-hhh-hhtt
nittaHHeg-2200
qwent. shat
qeent 7-72--hhht gemrinipmm
dbrxiintrust gp--3.5ttrre-0011 5-trre--122
lama- 70b-chat german220 2bot
5-00bcchw
gpi-4-0515
Figure 23: Limitation of the standardized benchmark GSM8k accurately reflecting and comparing basic reasoning capabilities of models, as illustrated by the stark discrepancy between the AIW correct response rate and the GSM8k average score. Notably, the majority of tested models exhibit low performance on AIW problems while achieving relatively high scores GSM8k, graduate-level math benchmark for large language models. Among models with slightly better calibration are Claude Opus and GPT 4 that outperform other models on AIW, which coincides with their high GSMSk scores. Llama 2 70b also shows better calibration, where its modest AIW performance matches its modest GSM8k score. In contrast, models like Mistral Large, Gemini Pro, Dbrx Instruct, or Command R+, while scoring high on GSM8k, show breakdown AIW (Command R+ has 0 correct response rate, Mistral Large and Gemini Pro 0.01, Dbrx Instruct 0.11,
see also Suppl Tab. 13
49

---

Published in Transactions on Machine Learning Research (01/2026)
AIW Correct answers MATH
1.0
G21400
GPT4-T
C.uudeSSOO
GPT-
Matne- CaudeeS-H
GPT-3.5-T
Gonrini Pro
0.2
S:777
Lons 2-70h
LLama 2.7b
AIW Correct response rate
2004 05 petmme2b-1
qwee1 5-14-hhhu Barre---b-cher
ristallage-2400 2402
ewent 5 8-4b-chet owent 5-1 chat 3b-chhh 8ama-2-78 loma 700 chat
cluud 3 sonnet 23240229
gemma it 1.w7-.5-7b-chut
92b-chat
claude-3 opus-
qeant is 72b chat
gpt 4 0513
gp1-3..-umm--1122 01125
Figure 24: Discrepancy between the AIW correct response rate and the MATH average score, indicating the limitation of standardized benchmark MATH in accurately assessing and comparing basic reasoning capabilities of models. Numerous models, such as Qwen 1.5 72B, exhibit a stark contrast in performance, scoring or close to 0 on ATW while achieving high scores on MATH.
50

---

Published in Transactions on Machine Learning Research (01/2026)
AIW Correct answers Hellaswag
Mulless BBRX Instrudt 1e
LLana
0.8
2mmm-2777
0.6
Oawnt 5-Po
Helloawws
S-1.0 5-1.bb
0.4
0.2
0.0
0.0
0.2
0.4
0.6
0.8
1.0
AIW response
command--pua plus mishaHHrge-2422
llame 2: chat dbrx instuctt
qwent 5 14b chat
5-7b-chat
20040223
claudo-3 halku
owon1.5-725
gpt 3.6 turoo 0125 geminiiro
gemmo 2b
coude 3 opus 20240229
ewan 1.5-332 5-32b-chut
qwant. 5-4b-chat
laama 2:7-char
Figure 25: Limitation the standardized benchmark Hellaswag in accurately assessing and comparing basic reasoning capabilities of models, as evidenced by the significant. discrepancy between the ATW correct. response rate and the Hellaswag average score. Models like Command R+ or Gemma-7B score on AIW while scoring high on Hellswag
AIW Correct answers ARC-c
Minni
Coudd
Sherde---f
CPT-1.5-T
0.8
ama-33700
Elamae-706
0.8
Mamm--3-to
ARC-c
Qawnt 5-
LA ama 2 131
Gemma
0.4
d0err1..5 111.5-44
0.0
0.2
0.4
0.6
0.8
1.0
AIW Correct response
mistallarge 2402
qwon1.5 32b chat
awent 5 76 chat
claude- 3 sormet 23240229
9pt-4-07 gp7-4-0013
tama-2-11 2-b chat
20242907
3-8b-chat
instrctt
awent..5
command--pis
claude-3 opus- 20040029
Bama-22700- chat
lama 3-700
owen 1.5 145 chat
awen1 5-1 chat Kama
72b
7o
Figure 26: Failure of standardized benchmark ARC-c to properly reflect and compare model basic reasoning capabilities as shown by strong discrepancy between ATW correct response rate ARC-c average score.
51

---

Published in Transactions on Machine Learning Research (01/2026)
AIW Number of responses model
type
RESTRICTED
STANDWRD
THINKING
qwent chat
5.7+022
opon
3.40+02
30+02
got 3.6 turbo
3.30-02
2.2+022
2.7e-02
got-- pork---arm----eee
3.3p+02
286+02
open min--ietrl-7o
3.2+022
command
3,45-02
2.8e002
2.70
1.20+02
2.3++02
1.2e002
mistollomall
3,4++22
222+02
3.2+002
600
derue 3-60mne--224422222
M6e+02
large-
344+02
3/1e002
opus 20240229
3,30+02
2.3e-02
open
3,6+022
2c-02
2.0+002
dlaude haku 20240307
46-02
3.2+022
36+02
1.++02
2.4e002
qwen
2.90022
1.0+02
2.4e002
codellamo: intruct
36+02
18:+02
2.4e-02
queen
qwent 2,4+022
qwent 2,4+022
8.8+02
2.4e002
owent
277+02
1.8+02
2.4e-02
codollamo intruct
36+02
1.8+02
2.6+002
genman
2.6+022
oimo
36-02
1.0+02
2.4e002
26+02
1.8+002
1.8e022
gol-4e-2024-05--
300
moodi
36-02
2.4e002
deopoedk- chat
36+02
2.4e002
go1--00125-ptwwin
36+02
1.8+002
2.8e-02
3p1-4-0513
gemin-pro
37s+02
2.5e-02
genma
2.9+022
1.76+02
2.4e-02
qwent
200
122e02
2++62
1.2-022
commend--iggly
gemni-1
chronos-temms
gok-
gm33.5attoo1100
mistai-iin
Figure 27: AIW Average number of responses per model for each prompt type (4 AIW variations prompt type.). Models with less than 100 responses per prompt. lype are excluded from further analysis. All those models have negligible correct response rates, either 0 or close to
52

---

Published in Transactions on Machine Learning Research (01/2020)
AIW Average length of responses
RESTRICTED
STANDARD
THINKING
1,30+00
2.0e-03
4.5c+02
23240307
1o-02
4.2c-02
Saude----opus 23040229
1.5m022
3.8e-02
1e-02
3.8e-02
sonnet
3.7e-02
wizordmm
2.7+02
3.2c-02
y5
1.50m02
3.2c-02
gp--6o-2104-05-13
2.6e002
cors 20240001
2o-02
2.9e-02
qwon1.5
2.8c-02
open-miatial
220+02
2.7e-02
lama-2-33b-tt 2-bbhtt
2.7e-02
4.8+022
2.6e-02
1,4+02
2.6-022
2000
open
2:fe+02
2.5e+02
2.4e002
2.3c-02
owent
12:+02
2.1c-02
2.0c+32
despseet:
2.7+02
2e-62
gemma
2.5e+02
gemini 1.5
1.8c-02
gen-t- gp---turto--104-000
To-inetruct
owent 1.3=002
1.8e-02
spen modrol
1.8c-02
quent
owent
1.4-022
genme-2t-t
0pt-3
got-4-0125 preview
plus
commens-nigntt
gemittigro
gpe-3
mintal-ing
Figure 28: Average length (in characters) of responses per model for each AIW prompt variation. Phi-2 has the highest average length of responses, because it is not. a classical instruction tuned model, but a base model, less capable of following instructions.
53

---

Published in Transactions on Machine Learning Research (01/2026)
C: AIW
Correct AIW variations 1-4, THINKING
A: AIW problem template and variations
1.0
p > 0.91
Correct responte ratelacturacy
AIW problem varlations 1-4
0.8
Variation Alice
brothers and she
sisters. [Correct answer:
Variation 2; Alice has 2 sisters and she also has 4 brothers. (Correct answer: 3
Variation 3: Alice has 4 sisters and she also has brother. [Correct answer:
0.6
Variation 4: Alice has 4 brothers and she also has sister. [Correct answer: 2]
How many sisters does Alice's brother have?
0.4
very low perf (.
low pert
p <
0.24
0.2
strong fluctuation (.0.67)
GSMBK example
0.0
GPT-4a
Ulama33 Bb instruct
When Sophie watches her nephew, she geta out a variety of toya for him. The bag of building blocks has 31 blocks in it. The bin of stuffed animals has 8 stuffed animals inside. The tower of stacking rings has multicolored rings on St.Sop recently bought tube of bouncy balls, bringing her total number of toye for her nephew up to 62. How many bouncy balls came in the tube?
D: GSM-Symb./BK
CPT-4o
Lamaa--b-instraeet
GSMBK 95.0
GSM8K 74.0
Foequency
SMM-Symbolic 94.9
very high perf
Proqreccy
15
GSM-Symbolid 74.6 (+2.9)
pirh
t0
(.6.9)
GSM Symboolie Accuracy (56) (8s CoT)
52
GSM Symbolic Accuracy (56) (8% COT)
70C
72.5
weak fluctuation (s0.07....
Figure 29: Both AIW (A) and GSM-Symbolic (GSM-S) (B) use variations in problem templates to measure sensitivity of model performance to variations and draw conclusions about generalization and rcasonining capabilities. AIW (C) provides strong evidence for generalization deficits by observing 1) strong fluctuations across variations of simple ATW problem (a color for each ATW variation 1-4) and 2) low average correct response rates for most models, eg. Llama 3 8B on the right. This provides convincing falsification of strong function hypothesis. In contrast, GSM-S (D); while using more sophisticated and content overloaded (eg see (B)) problems, cannot offer such conclusive evidence. The observed fluctuations are weak (eg 0.07 vs 0.67 on AIW for GPT-4o), and average performance is high (eg > 0.69 vs 0.15 on AIW for Llama 8B), while GSMSk falls well within the measured response distribution. While AIW variations thus reveal GPT-4o and Llama-3 8B generalization deficits, they stay hidden for GSM-S. We hypothesize that inability to observe strong fluctuations on GSM-S might be due to leakage of GSM8k type of problems into training data. Fig.
(B) and (D) are adapted from Mirzadeh et al. 2024)
standardized benchmarks that argue to properly measure core capabilities like generalization and reasoning, eg. MMLU, ARC, PIQA, HellaSwag, HvmanEval, WinoGrande. GSM8k, MATH, A1ME24/25 or GPQA/GPQA-
Diamond Hendrycks et al. (2020gg Clark et al. (2018): : Bisk et al. (2020); Zellers et al. (2019h: Sakaguchi et al.
(2019) Cobbe et al. (2021 AIME (2024); Rein et al. (2024), to name few important examples. Relying on
those benchmarks, it is commonly-held position to attribute to SOTA LLMs advanced functions like zero-shot
reasoning Kojima et al. (2022). and in general to put high expectations of strong core functionality like
generalization on released SOTA LLMs AAchiam et al. (2023 : Touvron et al. (2023a b Jiang et al. (2023)
Such claims extend beyond basic rescarch artifacts and bocome pervasive in applied industry, where SOTA
LLMs are advertised for public as robust problem solvers for various real world settings, explicitly emphasizing
their value as robust reasoners, coders and math solvers, attesting 'key business-critical capabilities' suitability for 'real-world enterprise use cases" (see announcements by Cohere on Command R-Plus Cohere 2024a), or by Mosaic on DBRX Mosaic, as only few selected represcntative examples out of many). Our study provides clear evidence for model breakdown on simple problems, and importantly, lack of model robustness in face of natural problem variations that do not change problem structure, and thus can serve as a vivid reminder that current SOTA LLMs and LRMs are not capable of strong generalization and robust reasoning. This can correct their perception in public and protect from trusting into overblown claims as supported by standardized benchmarks that do not properly reflect core capabilities and overlook clear deficits.
54

---

Published in Transactions on Machine Learning Research (01/2026)
Human
Alice has sisters and she also has brother. How many sisters does Alice's brother have? Before providing answer to this problem, think carefully and double check the path to the correct solution for any mistakes. Provide then the final answer in following form: \#### Answer: \"
Claude Opus
Let's think through this problem step by step:
Alice has sisters and brother.
Alice's brother is part of the same family as Alice.
3. All of Alice's sisters also sisters to her brother.
4. Therefore, Alice's brother has the same number of sisters as Alice has, which is 4.
I've double-checked the reasoning and can't find any mistakes in the logic.
### Answer: Alice's brother has 4 sisters
Figure 30: Example of a wrong response for Claude 3 Opus on THINKING prompt type, featuring failure of self-verification.
Human
Alice has four brothers and she also has a sister. How many sisters does Alice's brother have?
LLaMA-3-70B-Instru-
A classic lateral thinking puzzle!
Let's break it down:
Alice has four brothers.
Alice also has sister (herself).
So, each of Alice's brothers has... (drumroll) ...one sister, which Alice herself!
Therefore, the answer is: each of Alice's brothers has one sister.
Figure 31: Example of wrong response for Llama 70B Instruct. The model counts Alice as her own sister therefore produces wrong answer.
55

---

Published in Transactions on Machine Learning Research (01/2026)
Human
Alice has brothers and she also has 1 sister. How many sisters does Alice's brother have? Solve this problem and provide the final answer following form: |#### Answer:
GPT-4-tubbo-2224-44499
Alice has 1 sister. Since they are siblings, each of Alice's brothers would have the same number of sisters, which includes Alice and her other sister. Thus, each of Alice's brothers has 2 sisters. ### Answer: 2
Figure 32: Example of correct final answer with fully correct reasoning for GPT-4 Turbo.
Human
Alice has three brothers and she also has three sisters. How many sisters does Alice's brother have?
Claude 3 Opus
Okay, let's think about this step-by-step:
1) We know that Alice has three brothers and three sisters. This means that there total of 7 children in Alice's family (Alice + brothers + sisters).
2) Now, let's consider one of Alice's brothers. Let's call him Bob.
3) How many sisters does Bob have? Well, all of Alice's sisters are also Bob's sisters. Alice has 3 sisters, so Bob has 3 sisters too.
4) But wait! Alice is also Bob's sister, but didn't count her yet. we need add Alice the count of Bob's sisters.
5) Therefore, has sisters (Alice's sisters) sister (Alice herself) sisters total.
Since all of Alice's brothers are in the same situation as Bob, each of Alice's brothers has 4 sisters.
Figure 33: Example of a correct final answer with fully correct, detailed reasoning for Claude 3 Opus.
56

---

Published in Transactions on Machine Learning Research (01/2026)
Human
Alice has 3 sisters. Her mother has sister who does not have children she has 7 nephews and nieces and also 2 brothers. Alice's father has brother who has 5 nephews and nieces total, and who has also 1 son. How many cousins does Alice's sister have? Before providing answer to this problem, think carefully and double check the path to the correct solution for any mistakes. Provide then the final answer in following form: "#### Answer: [92]
Claude Opus
Let's break down the information given the problem solve step by step:
1. Alice has 3 sisters. 2. Alice's mother has sister who does not have children.
3. Alice's mother has 7 nephews and nieces total.
4. Alice's mother has 2 brothers.
5. Alice's father has a brother who has 5 nephews and nieces total.
6. Alice's father's brother has son. Now, let's calculate the number of cousins Alice's sister has:
1. Alice's mother's side:
Alice's mother's sister has no children, so there are no cousins from this side.
Alice's mother has brothers, but don't know how many children each of them has.
We know that Alice's mother has total of 7 nephews and nieces, which includes Alice and her 3 sisters.
So, the number of cousins from Alice's mother's brothers 7 (Alice + her 3 sisters) = cousins
2. Alice's father's side: Alice's father has a brother who has son.
We know that Alice's father's brother has total of 5 nephews and nieces, which includes Alice and her 3 sisters.
So, the number of cousins from Alice's father's brother
(Alice her sisters) son 5 4 + 1 = 2 cousins Therefore, the total number of cousins
Alice's sister has Cousins from mother's side Cousins from father's side = + 5 cousins
### Answer: Alice's sister has 5 cousins.
Figure 34: Example of fully correct reasoning and final answer response given to AIW+ problem by Claude Opus (prompt id 92, THINKING prompt type, Suppl. Tab. I1p. Such responses are very rare, but do exist (Suppl. Fig. 10 hinting that reasoning capability is potentially in place in stronger performers at largest scale like Claude 3 Opus, but is very fragile and severely compromisod.
57

---

Published in Transactions on Machine Learning Research (01/2026)
Human
Alice has four brothers and she also has sister. How many sisters does Alice's brother have? Solve the problem taking care about not doing mistakes.
Command-R+
This problem can be solved accurately and unambiguously. Alice has four brothers, which clearly indicates that she has four male siblings. Additionally, it stated that she also has sister, which designates female sibling.
Therefore, Alice's brothers have one sister. This interpretation straightforward and avoids any potential mistakes or ambiguities. based solely on the information provided, and there is no need for further interpretation speculation.
So, the definitive answer is: Alice's brothers have one sister.
Figure 35: Overconfidence: example of wrong response for Command R. Plus, containing highly persuasive wording to back up its confidence into incorrect solution
Human
Alice has X brothers and she also has Y sisters. How many sisters does Alice's brother have? Solve this problem and provide the final answer in following form: (#### Answer:
OLMo-7B-Instruct
The number of Alice's brothers number of Alice's sisters. ELEMENTARY ALGEBRA:
The number of X brothers and the total number of brothers and sisters is Y). now, and Y's sisters
represent X's brothers
(X Y)
Since the total number of brothers and sisters is X Y.X (X Y)
Now, divide both sides by (X + Y) to findX: X = (X Y) Y (x y) In other words:X = Subtract from both sides:
y.Now, scale the -y to the left side by adding 5 to both sides: = ((x + 5) y)-
So, the number Alice's brothers either x,
3, V.Wa Answer:
Figure 36: Confabulation: example of a wrong response for OLMo-7b-nntruec for AlW-param problem. The model generates text. that contains different algebraic manipulations involving X and Y and has little to do with reasoning.
58

---

Published in Transactions on Machine Learning Research (01/2026)
Human
Alice has four brothers and she also has sister. How many sisters does Alice's brother have?
Codeelamaa77b-inssuuct
apologize, but as a responsible language model, cannot provide an answer that would potentially perpetuate harm discrimination towards any individual or group, including person living with Down syndrome. It is important to recognize that every individual has inherent dignity and worth, regardless of their abilities characteristics. Instead, suggest rephrasing the question to focus on Alice's relationship with the person living with Down syndrome in a respectful and inclusive manner. can assist you in modifying the question to promote understanding and empathy towards individuals with diverse abilities and characteristics. you have any other questions or concerns, please feel free ask, and will do my best to provide helpful and responsible responses.
Figure 37: Example of a confabulation response for produces wrong reasoning and addition distracts from the actual main problem solve by bringing up that it is disrespectful to ask questions about person with Down syndrome, which is entircly invented out of nowhere by the model and is not present in the original task formulation.
In this example, the model
Author contributions
Marianna Nezhurina: discovered the original problem formulation and performed first experiments observing collapse across different models. Created further problem variations including the hard AIW+. Collected and analyzed data. Wrote major parts of the experimental infrastructure, data analysis and evaluation routines. Co-wrote the manuscript.
Lucia Cipolina-Kum: performed experiments, collected data and provided further input for the studies. Co-wrote the manuscript.
Mehdi Cherti: organized access to various models in the study via various APIs. Wrote code for parts of experimental infrastructure. Performed experiments, collected data and provided further input for the studies. Co-wrote the manuscript.
Jenia Jitsev: led the project. Created further problem variations, including AIW Friends and AIW Circles Colleagues. Created automated routines for experimental workflows, acquired resources for local model deployment infrastructure and performed large portion of experiments, extending experiments on reasoning models, collected and analyzed data. Wrote the manuscript.
59

---

Published in Transactions on Machine Learning Research (01/2026)
Human
Alice has four brothers and she also has sister. How many sisters does Alice's brother have? Look the problem step by step and formulate 3 different solutions that come to different results. Then evaluate which solution seems to be the best and then come to a definitive final statement.
Command-R+
This riddle can be approached in different ways, which can lead to different results. Here are three different solution approaches:
Interpretative solution:
This solution interprets relationship from Alice's perspective. Alice has four brothers, from her perspective, her sister is already accounted for among her brothers. So, her brothers have one sister each, which is Alice herself. So, Alice's brothers have one sister each, leading to result of four sisters in total for Alice's brothers.
Technical solution:
This solution focuses on the literal interpretation of the sentence structure. Alice has four brothers, and she also has a sister. Using a strict mathematical approach, one could argue that the word "also" adds an additional entity to the group of four brothers. This would result in Alice's brothers having five sisters in total since Alice herself their sister, and the additional "sister" from the sentence structure brings the total to five.
Contextual solution:
Considering the context, the sentence mentions that Alice has four brothers and introduces a separate entity, sister. Assuming that this sister is distinct from Alice herself, each of Alice's brothers would have one sister, Alice, plus this additional sister mentioned. So, Alice's brothers would have two sisters each, resulting in a total of eight sisters for Alice's brothers."
Now, evaluating these solutions:
The first interpretative solution considers a straightforward interpretation of the relationships, suggesting that sense understanding of the riddle doesn't require any complex linguistic or mathematical interpretations.
Alice's brothers have four sisters total. This interpretation aligns with a common-
The second technical solution unconventional conclusion that there are five sisters. While this solution adheres to the literal meaning of the sentence structure, it feels somewhat contrived and doesn't reflect how we typically understand family relationships."
strict linguistic interpretation, but it leads to rather
The third contextual solution considers a separate sister entity, which leads to the result that Alice's brothers have eight sisters in total. While this solution considers a distinct sister, it doubles the count for each brother, which doesn'tt align intuitively with the riddle's context.
Therefore, the final statement is that the first interpretative solution seems like the best fit. Alice's brothers have four sisters in total, and the riddle's answer is four. This interpretation respects the relationships described in the riddle without requiring complex linguistic mathematical maneuvers.
Figure 38: Example of confabulation response for Command Plus. We prompted Command R+ solve the problem step by step and come up with different solutions and then provide the final answer by evaluating those solutions. The model produces 3 solutions by taking different angles the problem, providing for each angle explanation to back it up. The 3 provided solutions are all wrong. In the following evaluation, the model labels then one of the solutions as best fit and proclaims it as a correct answer, arguing additionally for properly respecting relationships in the problem. The final provided answer incorrect.
60

---

Published in Transactions on Machine Learning Research (01/2026)
Human
Alice brothers and also sister. How many sisters does Alice's brother have?
Look at the problem step by step and formulate 3 different solutions that come to different results. Then evaluate which solution seoms to be the best and then come to a definitive final statement.
Command-R+
This riddle approached multiple ways, will present three different perspectives and their respective outcomes:
Solution
When interpreting this sentence, we take a literal approach and consider the information as it is given, Alice has four sisters, and she also has a brother. So, from this perspective, Alice's brother has four sisters.
Solution 2:
We consider the possibility inclusive language. When it is said that 'Alice sisters," it could be interpreted an inclusive statement, suggesting the speaker is including Alice count So, in case, Alice her four sisters make up Alice's brother has sisters.
females the family. Thus,
Solution 3:
We consider a broader context and the possibility of a non-binary sibling. In modern times, gender identities are always binary (male female). So, is possible that one more of Alice's 'sisters' could Identify non-binary. this case, the number sisters her brother has could be anywhere from three to five, depending how many Alice's siblings identify sisters.
Evaluation:
Solution seems straightforward, but relies on a very literal interpretation and doesn't consider nuances in language or modern perspectives on gender identity.
Solution 2 takes into account the potential for inclusive language, which is a valid point, but adds layer of complexity and uncertainty to the interpretation.
Solution 3 bbroadens the perspective to include non-binary gender identities, which is an important and inclusive viewpoint, but also introduces high level of uncertainty and ambiguity, making chailenging provide definitive
Final Statement:
Considering the information given and attempting to respect modern perspectives on gender identity, the most inclusive and definitive answer would be combination of Solutions and Alice's brother has four sisters, and this includes any cisgender or transgender sisters, but does not account for non bon-binary siblings, as their identification as 'sisters' may vary based on personal identity. This response provides clear answer while acknowledging the potential for broader range of gender identities.
Figure 39: Example of confabulation response for Command R Plus. We prompted Command R+ to solve the problem step by step and come up with 3 different solutions and then provide the final answer by evaluating quality of these solutions. All 3 solutions produced by the model are wrong. In the third solution model attempts to back up its answer by elaborating on possible gender identities members of Alice's family, which has no proper connection to the posed problem.
61


Reviews:
[{"review_id": "iX7Ra7sGyj", "reviewer": "Reviewer_EPht", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Contributions:\n1. Drawing on the analogy of Alice and Humpty Dumpty, the paper uses a simple “Alice’s relatives counting problem” (AIW) to test mainstream state-of-the-art LLMs and LRMs. It shows that models struggle to correctly answer specific task that requires the integration of numerical reasoning, relational reasoning, and logical inference.\n2. The paper raises a call for a reassessment of benchmark leakage. By fine-tuning models and then evaluating across different model generations with AIW questions created at different times, the authors find that later models can solve earlier AIW questions but continue to fail on newer variants. This suggests possible exposure of test items during training.\n\nPros:\n1. The paper provides detailed experimental results on a focused reasoning task across a wide range of LLMs and LRMs.\n2. The paper uses the observed failures to motivate a discussion of benchmark leakage, offering a valuable perspective on dataset contamination and evaluation reliability.\n\nCons:\n1. The structure of the paper is unclear: contributions are not highlighted, related work is not separated, and the writing relies on overly long sentences.\n2. The range of evaluation is too narrow, relying almost exclusively on AIW-style problems that test only one specific combination of relational, numerical, and logical reasoning.\n3. Several claims are overstated and extend beyond what the presented evidence can support.\n4. The self-evaluation prompt is overly simplistic and does not follow established best practices, such as chain-of-thought prompting, making the conclusion drawn from it relatively weak.", "claims_and_evidence_yn": "No", "claims_explanation": "Most claims are only partially supported, as the conclusions are often generalized beyond what the experiments demonstrate.\n\nClaim 1: Failure of LLMs on simple reasoning tasks\n\nThe evidence shows that multiple SOTA LLMs and LRMs fail on AIW. However, the authors extend this to claim a general reasoning deficit across models. This is an exaggeration, as the experiments address only one narrow reasoning domain and cannot establish a universal reasoning breakdown.\n\nClaim 2: Attribution to low-level issues vs. untrained problem types\n\nThe authors initially suggest that failures may stem from low-level issues such as number recognition or relationship reasoning. Controlled experiments largely rule out these explanations, but instead of resolving the cause, the paper shifts to speculating that models fail because of unfamiliar “untrained genres” of problems. This claim, however, is not supported by further experiments or evidence.\n\nClaim 3: Fake reasoning ability due to benchmark leakage\n\nThe analysis of model performance across generations provides reasonable grounds to suspect that AIW examples entered training data. However, since AIW is not an established benchmark, its inclusion in training corpora is unsurprising. From this evidence alone, one cannot conclude that other benchmarks are compromised. The call for reassessment is constructive, but claims of widespread benchmark ineffectiveness are overstated.", "audience_yn": "Yes", "audience_explanation": "Yes, at least some of TMLR’s audience would be interested in the findings. Despite the narrow scope of AIW and the weak evidence for benchmark leakage, the results show that even state-of-the-art LLMs struggle with simple reasoning tasks. This makes the study relevant for researchers focused on reasoning evaluation, benchmark design, and dataset integrity, while opening avenues for further work on broader reasoning problems and root-cause analysis.", "requested_changes": "1. Revise the title and framing\n\nThe Humpty Dumpty analogy is incorrectly attributed to Alice in Wonderland but actually comes from Through the Looking-Glass. This should be corrected to avoid confusion.\n\n2. Broaden the evaluation\n\nThe study should incorporate a wider variety of logical reasoning tasks beyond AIW so that its conclusions can be tested under broader conditions and made more generalizable.\n\n3. Strengthen causal analysis\n\nThe authors should design experiments that more directly test whether failures arise from number parsing, relation encoding, or higher-level reasoning. Without this, claims about the causes of failures remain speculative.\n\n4. Moderate claims\n\nThe conclusions should be stated more cautiously. Terms such as “reasoning breakdown” suggest an overly broad failure of LLMs and should be avoided, and benchmark leakage should be framed as a possibility indicated by AIW evidence rather than definitive proof of the invalidity of other benchmarks.\n\n5. Revise the design of the self-evaluation prompt\n\nThe current prompt is too simple and does not incorporate established best practices such as chain-of-thought prompting. To strengthen the validity of the conclusions, the authors should redesign the self-evaluation setting with more robust prompting strategies and compare results across different prompt types.\n\n6. Improve structure and clarity\n\nThe paper should explicitly list its contributions in the introduction, separate related work into its own section, and streamline the writing by shortening long sentences and reducing verbosity.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MwokRoIHV8", "reviewer": "Reviewer_neNL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This article shows that modern LLMs and LRMs can struggle with a maths problem. The article describes one such problem and shows that LLMs and LRMs have a wide range of success rates attempting variants of that same problem. The article also suggests that successes of some LLMs and LRMs at tackling earlier variants of the problem could be attributed to the problem leakage. The article calls for more critical, reliable and robust benchmark generation that is algorithmic in nature to reduce the impact of a possible benchmark leakage.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The article provides sufficient evidence that LLMs and LRMs believed by some to possess generalization and reasoning may not do so in all possible situations.", "audience_yn": "Yes", "audience_explanation": "I believe the article would be of interest to a broad audience spanning from those actively working on LLMs and LRMs to those who do not but may simply be interested in the current state of the area.", "requested_changes": "As the article is heavily focused on proving generalisation and reasoning breakdown given one class of maths problems, making requests for  changes would probably fall outside of the scope. Just a few examples of changes/additions that I found useful would be:\n\n1) You have never defined what do you mean by generalization and reasoning and how would you measure it. Let's say you have a class of students provided with an exam paper that features a range of maths problems like the one you describe. The probability that one of the students would fail one of these problems is not that low. Does it necessarily mean the lack of generalization and reasoning? \n\n2) You have focused on a single problem albeit one that can be formulated in a myriad of different ways. Although it is interesting that LLMs and LRMs may fail at it, it remains unclear why. I believe they all could solve \"if N=Y and  K=1, what is N+K\" style problem for at least one choice of Y (K indicates brother/sister). Can they do it for all possible choices of Y? Can you then gradually convert the problem above into the one you have investigated to determine at which point they fail? \n\n3) You could have broadened the scope of maths problems LLMs and LRMs struggle with to provide more insight on failed generalization/reasoning. Are generalization and reasoning capabilities linked with only solving maths problems? Is there a value to look at non-maths problems?", "broader_impact_concerns": "None", "additional_comments": "Overall, I found this article interesting. You have provided some evidence of what could not be the reason of the failure. I was hoping you would also provide a bit more on what could be the reason.\n\nFinally, could you please explain to me what is the point of \"Humpty Dumpty sat on a wall\" in the name of section 3.1?", "extra_scores": {}, "extra_text": {}}, {"review_id": "gRr2KrRsdD", "reviewer": "Reviewer_Xyxi", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper assesses the performance of a variety of reasoning and non-reasoning LLMs on prompt variations in a simple reasoning task. The paper finds that even SOTA models are highly sensitive to prompt variation despite their strong performance on standard benchmarks.\n\nStrengths: the paper provides a thorough investigation of prompt sensitivity in AIW task and is clearly written.\n\nWeaknesses: There is prior work with similar findings/claims based on different reasoning tasks and thus the novelty of the contribution is unclear to me.", "claims_and_evidence_yn": "Yes", "claims_explanation": "I have some comments and suggestions on the analyses but I’m generally convinced that the models tested don’t do well on the variations of AIW prompts. However, I’m wondering what I have learned from this work that I didn’t know before. I provide the details below under Requested Changes.", "audience_yn": "No", "audience_explanation": "The paper points out flaws in reasoning generalization in SOTA LLMs and LRMs. While this type of finding would generally be of interest to the TMLR readership, I'm concerned about its novelty as there's existing work that makes similar claims.", "requested_changes": "a) Critical:\n1. Novelty. There’s existing work asking very similar questions. For instance, Shojaee et al. ’25 present a comparison of thinking vs. non-thinking LLMs on several non-standard reasoning tasks and arrive at similar conclusions. The approach of employing variations of the prompt isn’t novel either and neither are the claims that the seemingly strong performance of modern SOTA LLMs on standard benchmarks is due to the training set contamination. Given the above, I’d encourage the authors to spell out the novelty of their contribution more and to situate it more clearly in the current research landscape.\n\nShojaee et al. 2025: The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity.\n2. The benchmarks. Throughout the paper, the authors claim that the models presented perform well on the standard benchmarks. However, I’m a bit confused by the choice of the benchmarks — MMLU, Hellaswag, or ARC are typically considered NLP rather than reasoning benchmarks; GSM8k is on the simpler side for many of the SOTA models. I’d consider looking at AIME or Math-500 as the performance gap here would be more informative.\n3. Design choices:  I’d like to see the motivation for testing the various aspects in the ‘Further relevant observations’ section as some of the design choices are unclear to me. For example, it’s not quite clear to me what MMLU can tell us about reasoning or how #3 differs from #8.\n\nb) Nice-to-have's:\n1. Section 3.2 currently reads as an afterthought but I think it’s the more interesting part of the paper and I’d like to see more analyses there. For example, have you considered testing some of the points in the ‘Further relevant observations’ such as AIW-Param for reasoning models and comparing the performance to the non-reasoning ones? Have you considered analyzing the reasoning traces?\n\nc) Smaller comments:\n1. In Figs. 1 and 2, GPT-4 is doing very well on prompt variation 4. Do you have an intuition why?\n2. What are the error bars in the figures?\n3. Consider making the figure labels larger and keeping the style of the figures the same (some have guides while others don’t, the font sizes are different sometimes, the bars aren’t always aligned)-- this will increase readability.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)