CogniLoad: A Synthetic Natural Language Reasoning Benchmark With Tunable Length, Intrinsic Difficulty, and Distractor Density
Authors: Daniel Kaiser
Arnoldo Frigessi
Ali Ramezani-Kebrya
Benjamin Ricaud
Venue: ICLR 2026 Poster (2026)
Keywords: benchmark
LLM
reasoning
long-context reasoning
Cognitive Load Theory
CLT
synthetic benchmark
natural language benchmark
intrinsic difficulty
extraneous load
needle-in-a-haystack
Primary area: datasets and benchmarks
TL;DR: CogniLoad offers a novel approach to LLM evaluation through natural language logic puzzles with independently tunable parameters (length, intrinsic difficulty, distractor density) grounded in Cognitive Load Theory

Abstract: Current benchmarks for long-context reasoning in Large Language Models (LLMs) often blur critical factors like intrinsic task complexity, distractor interference, and task length. To enable more precise failure analysis, we introduce CogniLoad, a novel synthetic benchmark grounded in Cognitive Load Theory (CLT). CogniLoad generates natural-language logic puzzles with independently tunable parameters that reflect CLT's core dimensions: intrinsic difficulty ($d$) controls intrinsic load; distractor-to-signal ratio ($\rho$) regulates extraneous load; and task length ($N$) serves as an operational proxy for conditions demanding germane load. Evaluating 22 SotA reasoning LLMs, CogniLoad reveals distinct performance sensitivities, identifying task length as a dominant constraint and uncovering varied tolerances to intrinsic complexity and U-shaped responses to distractor ratios. By offering systematic, factorial control over these cognitive load dimensions, CogniLoad provides a reproducible, scalable, and diagnostically rich tool for dissecting LLM reasoning limitations and guiding future model development.

COGNILOAD: A SYNTHETIC NATURAL LANGUAGE REASONING BENCHMARK WITH TUNABLE LENGTH, INTRINSIC DIFFICULTY, AND DISTRACTOR DENSITY
Daniel Kaiser*t Arnoldo Frigessi* Ali Ramezani-Kebrya" Benjamin Ricaud
daniel. kaiserauit.
ABSTRACT
Current benchmarks for long-context reasoning in Large Language Models (LLMs) often blur critical factors like intrinsic task complexity, distractor interference, and task length. To enable more precise failure analysis, we introduce CogniLoad,, a novel synthetic benchmark grounded in Cognitive Load Theory (CLT). Cognil ood generates natural-languse
logic puzzles with independently tunable parameters that reflect CLT's core dimensions: intrinsic difficulty (d) controls intrinsic load; distractor-to-signall ratio (p) regulates extraneous load; and task length (N) serves as an operational proxy for conditions demanding germane load. Evaluating 22 SotA reasoning LLMs, CogniLoad reveals distinct performance sensitivities, identifying task length as a dominant constraint and uncovering varied tolerances intrinsic complexity and U-shaped responses to distractor ratios. By offering systematic, factorial control over these cognitive load dimensions, CogmiLoad provides reproduuible, scalable, and diagnostically rich tool for dissecting LLM reasoning limitations and guiding future model development.
INTRODUCTION
Cognitive Load Theory (CLT) (Sweller 1988) characterizes three types of cognitive load on human working memory when solving problems (Sweller 1988 Paas et al. (200) Lieder and Griffiths 2020):
intrinsic (ICL), extraneous (ECL), and germane (GCL). ICL stems from the inherent complexity and element interactivity of the task (Halford et al.) 1998). ECL induced by suboptimal task presentation requiring the processing of elements that not task-relevant (Chandler and Sweller [199I). GCL
denotes the proportion of working-memory memory resources productively allocated to constructing and maintaining task task-specific schemas (Paas et al. 2003 Sweller 2010). Unlike ICL and ECL, which properties of the task material, GCL reflects how the learner allocates resources; must therefore be operationalized indirectly, through task conditions that demand sustained, constructive engagement with task-relevant schema.
Large language models (LLMs) demand analogous computational resources when solving reasoning tasks. The essential element interactivity of a reasoning chain mirrors ICL, distractor elements reflect ECL, and sustained engagement with intrinsically relevant information over a long reasoning process acts as an operational proxy for GCL-the constructive effort to build, maintain, and repeatedly apply a coherent problem- state schema over many sequential steps.
To the best of our knowledge, no study has based the evaluation of problem-solving capacities of LLMs in CLT by distinguishing these three load types, and existing benchmarks often confound them: LongBench (Bai et al.) 2024a) and L-Eval (An et al.] 2024) vary context length but not necessarily the intrinsic reasoning depth; LogicBench (Parmar et al.) 2024) probes ICL. with minimal demands on ECL or context-induced load; BABBLongg (Kuratov et al. 2024) mixes multi-step reasoning with fixed distractor ratios, obscuring precise failure attribution.
We introduce CogniLoad, controllable synthetic benchmark for long-context reasoning, inspired by
CLT, that operationalizes these load types through tunable parameters randomized natural-language
IItegreeat - Norwegian Centre for knowledge-driven machine learning Norway University of Oslo
fUT The Aretic University of

---

100%
80%
60%
40%
10
20
50
100
250
10
50
80
Intrinsic officutty
Tesk leigth
Neade-to-ay (86)
DeeptSeebu/l 5-2538
DeepSieei 10w-hr1
DoopSook-R1 R1-2 31 Lana 709
EXAONE Deop 52B
OwO 82B
0-e 3-778
ga1.5 20250025560 002250000000 00.07
08-2025- 04
SheppSeeko/1--bae QuennStzt
Phadi mwa---eeaoornn--ssu
garrt122 1-2---2-5ppp 5-pno
www--------122-------- 08647
DeepSeek-R 1-Dieill Qwen-7B
-wee -3-22--A2B
gem.t122222 E-fas
Figure 1: The average accuracy of models across the evaluated parameter space for d E 31,3,5, 10 (left panel), N {20, 50, 100, 250} (center panel), and (5, 95) (right panel). Each plot selects one dimension for the X-axis and averages the accuracy of all evaluated puzzles for the other two dimensions relative to it.
logic puzzles: (i) Intrinsic Load via Intrinsic Difficulty d controls the number of interacting entities, attributes, and logical clauses, directly manipulating ICL by varying essential element interactivity and reasoning depth. (ii) Extraneous Load via Distractor Density P dictates distractor density; lower increases irrelevant elements, manipulating ECL. (iii) Germane Load Proxy via Task Length N: increasing the number of sequential statements scales the sustained schema maintenance required without changing per-step complexity distractor density, serving as an operational proxy for conditions demanding germane-like cognitive work.
Our key contributions are summarized as follows:
We introduce CogniLoad, the first benchmark grounded in CLT that independently controls intrinsic load (d), extraneous load (p), and an operational proxy for germane load (N), scaling to arbitrarily long contexts.
2. We provide an algorithm for automatic randomized generation and evaluation of puzzle instances, enabling large-scale, reproducible, and contamination-resistaat comparison of LLM capabilities.
3. We report empirical results 22 state-of-the-art (SotA) reasoning LLMs (Figure re- vealing distinct failure regimes across (d, N, p) and highlighting specific targets for model improvement.
Together, these contributions translate CLT into a precise diagnostic framework for understanding and advancing long- contect reasoning in LLMs.
1.1 RELATED WORK
Long-context Benchmarks (Working Memory Capacity). line of work starting with Long- Range Arena (LRA) (Tay et al.) 2020) and followed by several recent benchmarks probe LLM performance on long sequences, often framed as testing "memory load" or context utilization. Earlier studies such as SCROLLS (Shaham et al., 2022). BookSum (Kryšciński et al. 20211) and QMSum (Zhong et al. 202 scale document length without manipulating intrinsic difficulty. LongBench (Bai et al. 2024a b) and L-Eval (An et al. 2024) aggregate multi-task сorporа up to 200k tokens, while BABILong (Kuratov et al.) 2024). LongReason (Ling et al. 2025). RULER ER (Hsieh et al., (022)) ZeroSCROLLS (Shaham et al., 2023), and Michelangelo (Vodrahalli et al.) 2024) increase context while the inherent difficulty of individual sub-tasks (ICL) may vary misystematically and distractor density (ECI.) is often not controlled variable. Consequently, performance degradation could be due
2

---

to sheer length overwhelming processing capacity, or inability to sustain germane-like cognitive work over extended relevant information, but the precise cause of failure is not clear.
Logical-reasoning Benchmarks (Intrinsic Load). A complementary line of benchmarks focuses on ICL by presenting tasks with high inherent complexity but often within minimal context lengths
or distractors. Notable classical suites include ReClor (Yu et al. 2020), LogiQA (Liu al. 2020), and BIG-Bench-Hard (BBH) (Suzgun et al. 2022). AutoLogic (Zhu et al. 2025 is a benchmark
that explicitly focuses on scaling ICL through controllable complexity. LogieBench (Parmar et al.) 2024). CLUTRR (Sinha et al. 2019), and ZebraL.ggic (Lin et al. 2025) also exemplify this by formulating symbolic logic puzzles that demand processing many interacting elements (e.g., multi- step deductions, handling negation, and constraint satisfaction). LogiEval (Liu et al. (025) tests deductive, inductive, analogical, and abductive reasoning but forgoes independent dimensional control and synthetic generation. Similarly, mathematical reasoning datasets, e.g., GSM8K (Cobbe et al.. 202T) and abstract rule induction tasks, e.g.. ARC-AGI (Chollet et al.) (2024) primarily escalate ICL by increasing the complexity of essential rules and their interdependencies.
Needle in a Haystack Benchmarks (Extraneous Load). Needle in a haystack (NIAH) designs (Gkamradi 2023) specifically target ECL by embedding relevant facts ("needles") within large volumes distractor text ("hay"). Variants such as Sequential NIAH (Yu et al. (202)) and Nolima (Modarressi al.) 2025 investigate the impact of such distractors, which constitute non-essential cle- ments requiring processing for filtering, thereby imposing ECL. While these benchmarks effectively isolate the impact of distractors on information retrieval, the "needle" tasks themselves typically involve low ICL (e.g., simple fact lookup).
Evaluation. CLT highlights the interplay of ICL, ECL, and germane processing under finite working memory (Paas et al. 2003). Existing LLM reasoning benchmarks, however, typically manipulate only one dimension without systematic, independent control over the others. Even benchmarks like MIR-Bench ( (an ct al., (025) which combine high ICL with extensive input, do not offer the factorial control needed disentangle these loads, hindering precise diagnostics. Similar to our work, GSM-x (Zhou et al.) (025) allows manipulating noise and difficulty. However, these parameters are not adjusted independently of task length.
Need for
BENCHMARK DESIGN: COGNILOAD LOGIC PUZZLES
2.1 PUZZLE DEFINITION AND CONSTRUCTION
CogniLoad is a family of Lantuual- logic-grid puzzles explicitly crafted to probe sequential reasoning capabilities of L.Mss. The design goals are threefold: each puzzle (i) necessitates sequential multi-step deduction where order fundamentally matters; (ii) embeds a controllable number of relevant "needle" facts within the context of controllable number of "hay" distractor statements; and (iii) provides parameters that control distinct dimensions of cognitive load. This section formalizes the task and describes the puzzle generation process, the control parameters, and key design choices.
Each puzzle in CogniLoad (see Figure 2 consists of a set of people with independent and mutable attributes. A series of statements, applied in strictly sequential order, updates these attributes according to conditions specified in each statement. The puzzle generation is parameterized by three key parameters: intrinsic difficulty d, total number of statements N, and needle-to-hay ratio p.
2.1.1 BASIC PUZZLE CONSTRUCTION
puzzle is formally characterized by the following components:
People: A set P = {p1. (pp.PP...... } of persons in the puzzle, and n max(d, 2).
Person of Interest (Pol): A randomly selected person p* E aboou whom the final question is asked.
Attribute Categories: A set A = {cI, c2. ... Ca) of attributes randomly selected from predefined taxonomy of 12 categories. Each category takes values in a Value Domain with a given finite cardinality, smaller or equal to 10.
3

---

(i) Puzzle Instruction: Solve this logie puzzle. You MUST finalize your response with a single sentence about the asked property (e.g., "Peter is in the livingroom.", "Peter is wearing blue socks",. ... ). Solve the puzzle by reasoning through the statements in strictly sequential order.
(ii) Initial State:
(iii) Update Statements:
Brent is wearing green socks is wearing pur-
1. The people wearing green socks listen to elec- tronic music.
ple gloves and last listened to classical music.
Anthony is wearing purple socks and is wearing
2. The people who last listened to classical music and wearing purple gloves put on yellow gloves.
yellow gloves and last listened to disco music.
(iv) Query: What color socks is Brent wearing?
Figure 2: Example CogniLoad puzzle with intrinsic difficulty d = 3, statements N = 20, and needle-to-hay ratio 50%. Only a subset of the initial state and update statements is shown.
Value Domains: For each category E A. a value domain V. = fee V........ Co where le = + for d or le 3 when d See Appendix G for the complete ontology.
State Function: (p, c) represents the value of attribute e for person p at step t. Each person has values for the d attribute of the selected attribute categories A, thus the state value represents a vector of dimension d.
2.1.2 PUZZLE INITIALIZATION
puzzle starts with initialization statements (t = 0) that assign unique attribute values to each person: Vp E P, Ve E A So e) V. such that Vp;, 2 E P, + j. 3e E A So (p:, c) j So(p, c). This initialization guarantees a maximally diverse starting point in which every pair of persons differs in at least one attribute. preventing trivial early states and ensuring that both needle and hay statements remain non-degenerate throughout generation.
2.1.3 STATEMENT GENERATION PROCESS
For each step from to N, statement is generated that changes the state person. If updates the Pol, the statement is called needle. An update for non-Pol is called hay.
Statement Type Selection: Given N and p. let neelle and mhay be the remaining numbers of needles and hays satisfy the desired proportion p in the complete puzzle. The probability of selecting a needle statement is then P(T, = needle) = mreedte/ /NN t). The total number of needle statements in the puzzle is caleulated as ne max(1, min(Ni round(V p/ o/100))).
2. Reference Person Selection: Given the selected statement type Tú, the algorithm selects the reference person Te: if T, needle - Ft = p" and if hay - Tr Uniform( ( {p" }).
3. Statement Structure:
CogniLooad samples a number of conditions k; ~ Uniform(1,.. d} (the number of attribute categories that must simultaneously match for the update), a number of state updates m, Uniform (I...,, d} (the number of attribute categories modified when the rule fires), and uniformly samples attribute categories C. S A, |C,| h2 and state updates U, C A, [UL] = m--
4. Condition and Update Value Specification: For each category E Ct. the condition value set by the reference person's current state: Ve, := S c). For needles, these conditions target the Pol (and can possibly involve other people), while for hays the conditions can match anyone except the Pol. For update values, if T. = needle = Uet Uniform(V,) and
T, = hay -
We. Uniform(V. (S-11pp", ())
Logical Form: The statement step has the logical form:
Vp E P: (A S-1(p.c) vor) oEC,
(A St Moe)) cEU,
Attributes not mentioned in the update set remain unchanged Vp E P. E A U, S,(,,) = Sp-1pp,e) c). This is not specified in the prompt but implicitly assumed by the L.Mss.
4

---

Each statement is converted from its logical form into natural language via deterministic string templates (c.g., "The people wearing (color) socks listen to (genre) music."), ensuring full repro- ducibility without reliance an LLM for text generation. The complete generation code is provided in the accompanying repository (see Section
2.1.4 VALIDATION CONSTRAINTS
sequence of validations verifies that the generated statement does not result in a state that prevents the generation of further needles and hays. If all validations pass, the statement is appended to the puzzle; otherwise new statement is generated.
For hay statements (re * p"): After the update, the state of affected non-Pols must not become identical to Pol Vp P) {p*} such that Ve E C₁ St-1(P, c) = Ve,,, 3e E A S,(,,) c) + S((p", and the update must not affect the Pol 3c C. 1 7 Ce,
Rationale. Hay cannot affect the Pol and cannot collapse all non-Pols onto the Pol: this ensures a hay statement is truly extraneous and that at least one non-Pol retains different state from the Pol, permitting further hay generation.
For needle statements (r; =)"): The update must not affect all non-Pol people 3p E P {p") Ee E C. : Sp (p. c) * Vo.t and after the update not all non-Pols have identical states as the Pol Bp E P [= A (p,, c) + S,(p",c).
Rationale. Needles cannot affect everyone and cannot collapse all non-Pols to the Pol: this preserves state diversity and ensures that further genuine hay statements remain constructible.
To prevent the distractors from becoming too trivial to track at lower difficulties, we require that hay statement does result in all non-Pols become identical the two persons with distinct attribute values. CogniLoad construction ensures that each hay statement T. hay affects least one non-Pol Ep [p-} Vc E C. Sp-1 c) Ue.t-
{p"} must contain least
2.1.5 FINAL QUESTION GENERATION
After all N statements have been generated, the puzzle concludes with question about random attriiute of the Pol, sampled as a random category c2 ~ Uniform(A). The correct answer to the puzzle is (x , G,) obtained from the final state of the Pol.
2.1.6 EVALUATION METRICS
We evaluate each puzzle by exact-matching of the queried attribute value in the model output, accommodating minor phrasing and common lexical variants (see Appendices D and E for the full pipeline). Specifically, we accept three outcome types correct: (i) the gold value in the last sentence that names the Pol and contains a category qualifier; (ii) the gold value the last sentence naming the Pol; and (iii) the gold value in the final sentence of the output. This grad- uated matching mitigates sensitivity to minor formatting drift while maintaining a deterministic, reproducible evaluation signal. The accuracy of model M across the evaluation set is calculated as acc(M) . E1ez answer (2) = (w(p", ,c)] where (p c2) represents the final state value of the queried attribute og for the Pol p" after all N statements have been processed.
2.2 TUNABLE PARAMETERS
To systematically probe long-context reasoning, CogniLoad employs three independent parameters that operationalize CLT's load dimensions (Paas al. 2003) and allow the creation of puzzles with varying characteristics. Together, they define the load profile of puzzle instance.
Intrinsic Difficulty (d) for d (1,3,5,7 10) controls multiple facets of puzzle complexity (see Table I directly manipulating ICL which according to CLT hinges on element interactivity (Halford et al. 1998). Increasing d increases ICL via: (i) combinatorial growth in state space (== (d+11)) (11) increased interactivity between persons, attributes, and values, and (iii) increased rule complexity (up to d conditions/updaate per statement).
5

---

Table 1: Key parameters controlling the puzzle generation.
Symbol
Name
Definition
Cognitive Load Affected
d
Intrinsic Difficulty
Controls cardinality of people set PP|
ICL. Element interactivity, state space/rule complexity.
max(d, 2), attribute categories |A| = d.
for each category E A the cardinal- ity of value domains |Val max(d + 1,3), and the distribution of conditions and updates per statement: k.m
(1..orm(1.......
N
Task Length
Total number of sequential state transi- tions in the puzzle.
GCL Proxy Task Length:
Scales the number of sequen- tial schema applications
Needle-to- hay Ratio
Percentage of statements directly influ- eneing the Pol (needles) versus distrac- tor statements (hay)
ECL: Distractor density challenges filtering, selective attention, and imposing load from processing non- essential elements.
Task Length (N) for N E {20, 50, 100, 250} sets the total number of sequential state-update state- ments. N serves an operational proxy for GCL: the schema to be maintained is the Pol's attribute vector across d categories, and each statement requires a sequential update of this vector. Increasing N scales the number of such updates without changing per-step element interactivity (d) or distractor density (p), thereby isolating the demand for sustained, constructive schema maintenance (Ericsson and Kintsch 1995) the hallmark of germane processing CLT (Sweller) 2010).
Needle-to-hay Ratio (p) for p (5,... ..., 95} sets the percentage of Pol-relevant ("needle") versus distractor ("hay") statements, directly manipulating ECL. ECL arises from processing non-essential elements (Chandler and Sweller 1991). Decreasing o increases ECL via increased distractor density which challenges filtering. Increasing P controls ECL by focusing resources on relevant information. Critically, CogniLoad's "hay" statements are syntactically similar to "needles" and involve valid state updates for non-Pols, imposing more challenging ECL than easy to distinguish distractor text.
RESULTS
We have evaluated the performance of 13 open weights LLMs on 100 random CogniLoad puzzles per (d, N, p) configuration resulting in 14'000 puzzle instances per LLM in total. In addition, we have evaluated the proprietary Gemini-2.5 and gpt-s[
10 Cognil puzzles per configuration (i.e., 1'400 puzzle instances). The maximum context length (input + output) was set 32K tokens and LLMs run with their preset default decoding settings and system prompts.
mpo-e-models models and DeepSeek-F -1-ck------28
Figure shows mean accuracy across models as each load dimension varies with trends corroborated by our regression analysis (Section 3.I. Error bars (90% Wilson confidence intervals) for all curves are provided in Appendix C
Intrinsic Difficulty (d) Performance declines monotonically with for most models, although few mid-tier models show small bumps at d = 3 (c.g., QwQ-32B: 0.62+;0.69 +0.69; DS-Qwen-32B: 0.63 (066).. Top models degrade only slightly from d 1.00->0.97,, while smaller or distilled models drop by 0. 10-0... in the same range. By d = 5, 12 of 22 models fall below 50% accuracy. Beyond d 2 7 the marginal decline flattens for the majority of models: the strongest models maintain their performance even at d 10 (gpt-5: 0.82: o3: 0.80), and the weakest ones approach 0.10-0.15.
d 3 (o3: 0.96 0..6; gpt-5:
Task Length (N) This parameter remains the dominant stressor. Most models exhibit their steepest decline between N 20 and = 50 (c.g., OS-Llama-70l 70B: 0.89 Qwen3-8B: 0.71 00.40)
The gpt-5 family models were evaluated using the "medium" reasoning effort setting.

---

Table 2: Per-model quadratic-p GLM estimates with Wald statistic for p-values alongside derived
50% load load-capacity thresholds (see Section 3.1.3). The value for NT indicates that no real root exists [0, 1]. "DS" abbreviates "DeepSeek" in the model names. *** ==p=00001. "0p01,"*<0000 *pp-0.01, pp=0.05
k
Ap-
ECL.00
NTso
DD50
gemini-2.5
22.51
0.41
9.15
-1.67
1.76
153.3
12.74
gemini-2.5-flasht
18.14
-0.44am
-7.56
-1.79
2.16
111.5
8.56
gemini-2.5-llas--itt
3.19
-0.30
-1.22
-2.88**
3.82**
8.8
0.93
1.53
gpt-5-2025-08-077
17.34
0.399
5.11
-7.04"
5.62
382.8
14.78
2015-min--2025-08---
11.09
0.22
33.6
-4.87**
5.10't
164.1
11.72
Opt-5-nano-2025-08-00
6.50
-0.31
-2.43
-4.30
4.12
35.7
0.94
2.87
03-2025-04-16
12.83
-0.22
4.26
-2.72
2.07
356.9
19.07
04- 04-mini-20 mimii-025--00-
2005-04-12- 04-16
13.00
-0.23
4.89
-3.9"*
6.37
132.1
10.86
DS-RI-0522 RI-0528
13.70
-0.39
- 5.28
-4.13
4.19
104.6
7.51
DS-Llama-
8.36
-0.30
33.8
-3.00
3.92
69.8
0.53
5.14
DS- DS-Qwen-32B
5.15
-0.19
-2.12
-2.7
2.29
0.78
3.95
DS-Qwen-7B
1.71
0.23
0.96
-0.5
0.58
2.9
0.53
DS- DS-Qwen-1.5B
-0.35
0.200
0.33
0.47
-0.14
0.0
3.95
Phi Phi-4-reasoning plus
9.52
0.45
3.58
4.211
3.41
45.7
0.16
3.68
Phi-4-reasoning
9.11
0.39
3.35
4..2
3.99
52.3
4.08
Phii--mini-reasooinng
1.70
0.244
01.1
-1.40
1.45 **
1.3
0.55
EXAONE-Deep-32B
4.09
0.26
1.55
33.16
2.45
14.1
1.0
QwQ-32B
5.68
-0.21
2.08
3370
3.07
48.0
0.95
3.56
Qwen3-32B
7.22
-0.29
-2.55
-3.21
2,75
0.94
4.1
Qwen3-30B-A3B 30B-BBBB
5.83
-0.30
2.25
-2.66
2.87
36.7
0.99
3.05
Qwen3-8B
5m40
-0.26
-2.19
-2,77
2,66
30.8
2.2
Qwen3-1.7B
0.62
-0.17
-0.46
-1.53
1.24
0.0
-4.07
while the best performing ones show relative resilience (gpt-5: 1.00-+0.98; o3: 0.99~>..88). Accuracy declines with longer sequences
N 100, several models roughly halve their N 20 accuracy (e.g., DS-LLama- 70B: 0.48: Qwen3-322 3-BB: 0.38) and at N = 250 only two models show above 50% accuracy (i.c., gpt-5 at 0.76 and o3 at 0.68) while most perform at 0.20-0.30 accuracy.
Extraneous Load Needle-to-hay Ratio (p) A characteristic U-shaped response emerges, with perfor- mance lowest around o [25, 50]% and recovering at higher p. Recovery equals or exceeds the low-p baseline for several models (e.g.. DS-Llama-700 70B: 0.61 0.61->0.64; gemini-2.5- 2.5--fasshfhihit 0.38 (88-),53), Frontier models show smooth, marginal variations (gpt-5: 0.97+ 89-00.91), while some models recover only partially (Phi-4-reasoning-plee 0.59->0.45; EXAONE-Deep-322 0.47-x.32)..
Two competing effects underlie the U-shape. Increasing p simultaneously (a) reduces the number of distractor statements, making filtering easier, and (b) increases the number of Pol state transitions, making sequential tracking harder. The accuracy difference Ap = Acc(low p) Accghigh p) reveals which effect dominates per model: Ap 0 (e.g., DeepSeck-R1, gpt-5-mini) indicates the two effects balance; A, > 0 (e.g., gpt-5, o3) indicates noise tolerance; Ap < 0 (e.g., Gemini-2.5- flash, DS OS-RI-Llama) indicates distractor sensitivity outweighs the benefit of fewer essential steps. Appendix Appendix AA] further investigates this U-shape at different levels of d and N, showing that increasing accentuates the U-shape while increasing N shifts curves downward and progressively flattens them.
3.1 LOAD-SENSITIVIT REGRESSION
To quantify model-specific sensitivities of the accuracy to load dimensions and derive interpretable capacity thresholds for each model, employ regression-based approach that allows to isolate the impact of each type of cognitive load (see Table 2)
3.1.1 REGRESSION MODEL SPECIFICATION
We model the performance of LLMs using a binomial generalized linear model (GLM) with a logit link function:
Pi(Y=1)= dBo Ba d + By logio N 8pp 8,2 )²),
7

---

where the binary outcome Y represents exact-match accuracy (Y when the model solves the puzzle correctly), a(-) is the inverse logit function, and the coefficients Sa. By and 3p quantify sensitivity to intrinsic difficulty (ICL), task length (GCL), and distractor ratios (ECL), respectively. The inclusion of quadratic term for is motivated by the characteristic U-shape observed in the third panel of and based on improved Akaike Information Criterion (AIC) value for 18 out of the 22 fitted models when included (see Appendix F). Since N ranges up to 250, we apply 1og10 keep it at a similar scale the other parameters of the regression.
3.1.2 SIGNIFICANCE OF MAIN EFFECTS
In all models, Ba and By are significant and highly negative, confirming performance degradation with increased ICL and GCL. The quadratic term for is also significant (except for two models) confirming the U-shaped response for most models: models typically perform worst intermediate values and recover as approaches either extreme. Five models exhibit statistically insignificant coefficients for P terms, reflecting poor bascline performance for the smallest models and indifference to distraction for strong models (i.c., o3, Semini-2255ppre (emini--2.5-lashh)
3.1.3 CAPACITY POINTS AT 50% ACCURACY
The GLM coefficients (Table 2 allow us to derive interpretable capacity thresholds. These represent the point which a model's accuracy is predicted to drop to 50% when varying single load parameter, while holding other load parameters at their estimated mean values:
ECL,0 (Effective Context Length): Maximum number of statements model can process while maintaining 50% accuracy. Large ECL50 values indicate superior context handling.
NT (Needle-to-hay Threshold): Minimum proportion of relevant information required to maintain 50% accuracy. Crucially, small values indicate greater robustness to distractors. If the estimated 50 missing, then the model accuracy is not expected to cross the 50% threshold for any value 0 S p S 1, under mean conditions for d and N.
1o (Intrinsic Difficulty): It is the maximum intrinsic complexity (number of interacting entities/at- tributes) that a model can handle while maintaining 50% accuracy. Negative values indicate failure to reach 50% accuracy even at the lowest difficulty setting under mean conditions for N and p.
Mathematically, these thresholds are derived by setting the logit in the GLM equation to zero (for Pr(y = 1) 0.5) and solving for the parameter of interest, e.g.i
ECL50 10 -0 9ad++ 1/Bx; ID50 -30 + Ivlogio N +Bp0
)/3a )/a
For NT.0, we solve the quadratic equation Bu + 3yd + 8 N + 3p + 8ppp² = 0 for p.
3.1.4 MODEL CAPACITY
The regression analysis and estimated capacity thresholds (Table reveal clear variations among models that be grouped into three classes:
Frontier/High-capacin Models: gpt-5 and o3 lead by a wide margin (ECL50 > 300), followed by gemini-2.55pro. gpt-5-mini, o4-mini, and DS-R1-0528. The high baseline performance of gemini- 2.5-pro (6o = 22.5) together with the large N of -9.15 5 is consistent with the uniquely large amount long context errors as N increases (as illustrated in the Appendix EE.
Mid-capacity Models: DS-L.ama--70B, thii--reasoning-plus (wen3-33B-A-3) Qwen3-32B, DS- DS-QWen-32RB, QwQ-32B, Phi-4-reasoning, 2op-5--nao-2225-08-00 and Qwen3-8B form broad middle
tier with good performance at moderate N and d values.
Low-capacity Models: DS-Qwen-7B, Phi-4-mini-reasooing DS-Qwen-1.5B, and Qwen3-1.7B ex- hibit minimal effective context handling capacity failing to reach 50% accuracy even under mean context/distractor conditions, deteriorating rapidly under slightly increasing load.
3.1.5 DIFFERENTIAL SENSITIVITY TO LOAD DIMENSIONS
The estimated coefficients further reveal distinct sensitivity profiles:
8

---

Sensitivity to context length (3x): Universally negative and highly significant, larger models often show greater relative degradation compared frontier models arise from the combined effect of So. Py indicating comparisons are best made via FCLLs0 and not Ay in isolation.
to their higher baselines. Yet large ECLso values for
Sensitivity intrinsic difficulty (Ba): Negative across models with narrow range, suggests more uniform effect. Despite some steep Sa values, high baselines (e.g. gemini-2.5-flash) yield large 1D5 values unlike smaller models with similar Ba (e.g., Phi-4-reasoning-plus)
Sensitivity to information relevance (3p and : Confirms the U-shaped response, but NT values reveal nuanced distractor robustness variations masked by aggregate scores (e.g., DS-Llama-70B vs. Qwen3-32B), For frontier models, the absence of NT50 indicates achieving above 50% accuracy while for weak models the same absence indicates remaining below 50% accuracy.
3.1.6 INTERACTION EFFECTS AMONG LOAD DIMENSIONS
To probe whether load dimensions compound, we fit per-model logistic GLMs augmented with all pairwise and three-way interactions among d, log10 N, and p (full specification and coefficient table in Appendix B). Three robust patterns emerge:
Super-additive difficulty-enggth coupling. A negative d N interaction is significant (PLR < 0.01) in 17/22 models, indicating that longer, more complex tasks degrade accuracy beyond what either factor alone would predict. Frontier models (gpt-5, 03) show no detectable d N interaction, consistent with near-separable responses at current loads.
Difficulty amplifies distractor harm. A negative d' p is significant 13/22 models: higher intrinsic complexity makes models more vulnerable to extraneous load, consistent with more complex schemas being more fragile under noise.
Length-distructor trade-off' is model-dependent. The N p interaction is significant in subset of models and its sign separates two regimes: positive values (c.g., Qwen3-30B-A3B, EXAONE- Deep-32B) indicate that reducing distractors mitigates the length penalty, while negative values (e.g.. DS-Q (S-Qwen-7B) indicate that longer sequences amplify distractor harm. A positive three-way dx N x p in 12/22 models shows that increasing the needle share is most beneficial when tasks are simultaneously long and complex.
3.2 FAILURE MODES ACROSS MODELS, LENGTH, AND DIFFICULTY
We analyze error categories from the evaluation pipeline (see Appendix D) to identify failure modes and provide the complete distributions and per-model breakdowns in the Appendix
State-tracking mistakes dominate under load. Across models, the most common non-context failure is wrong final attribution in the last valid Pol sentence (valid-logic), consistent with mis- tracking sequential updates rather than formatting issues. For example, at N=250, Qwen3-32B has 2'541 valid-logic cases, DS-Llama-70B 2 2'465, and QwQ-32B 2'092. These logic errors also increase monotonically with d for nearly all models.
Long-context budget overflows are a model-specific failure at extreme N. The max-context errors grow sharply with N for some models: gemini-2.5-faas (280 errors in 350 samples at N 250) or gemini-2.5-pr (268/350). OpenAl models also make these errors at N = 250 but at much lower levels (gpt-5: 32/350; o3: 24/348). Token-length distributions (Appendix Appendix H show that these overflows stem from Gemini's 's substantially higher output verbosity rather than the puzzles not generally being impossible be solved within the provided 32K context limit. Because overflows are concentrated in specific model-condition pairs (primarily Gemini 2.5 at N = 250) while the vast majority of models and conditions remain well below the 32K budget, we report them as separate error category. Gemini's ss accuracy N 250 should accordingly be interpreted as a lower bound on reasoning capability. For frontier models such as gpt-5 and o3, total token usage sits comfortably below the budget the same (N, d) conditions, confirming that their degradation reflects reasoning under load rather than serving limits.
mnswer-formattin drift emerges under higher N and d, mainly in smaller models. While poi-logic stays near zero for most models, last-logic increases notably for compact models (e.g.. Phi-4-mini-e menir-nnnnnn 400 last-logic at N 250; DS- DS-Qwen-7B: 116), indicating that under load,

---

models fail to state their answer in the required format (a single sentence naming the Pol and the queried attribute), even when they reason about the Pol elsewhere in their output.
"Other" failures rise with sequence length in small and mid-tier models. At . 250, DS- Qwen-1.5B has 605 "other" cases (often claiming the puzzle is unsolvable) and DS DS-Qwen-7B 461 indicating shift from precise (but wrong) answers non-answers load grows.
DISCUSSION
By operationalizing CLT, CogniLoad enables multi multi-dimensional LLM evaluation that reveals nuanced failure patterns obscured by single-dimensional benchmarks. Task length (N) emerges as the dominant constraint; models exhibit distinct sensitivities to d versus p. with characteristic U-shaped responses to intermediate distractor densities; and estimated capacity thresholds provide concise "cognitive fingerprints" for diagnostic LLM evaluation. We discuss limitations below.
Nuances of the CLT-LLM Analogy While CLT provides a powerful analogous framework, it crucial to acknowledge that "cognitive load" in LLMs manifests as computational constraints (e.g., attention saturation, representational bottlenecks) rather than biological working memory limitations. Our operationalization of N proxy for conditions demanding GCL, for example, is an abstraction. Future research should aim to bridge CLT concepts with direct, mechanistic measures of the underlying computational processes in LLMs to refine this analogy.
Scope of Reasoning and Generalizability Cognil Load focuses sequential and pure deductive reasoning without requiring domain knowledge. While this reasoning type is fundamental to various subject areas (e.g., code, math), it is distinct from alternative reasoning paradigms like inductive, abductive, or analogical reasoning. Extending the CLT-grounded multi-dimensional evaluation to other reasoning types and evaluating it in other languages is a promising next step.
Beyond Accuracy and Main Effects The current evaluation relies on exact-match accuracy. Future iterations could incorporate richer metrics (e.g., step-wise reasoning fidelity, solution coherence, uncertainty of solutions). Reinforcement learning on verifiable rewards (Guo et al. (025) presents promising application of CogniLoad in LLM training, as its generated metadata enables precise verification of reasoning steps despite limited data of this kind.
Despite these considerations, by decomposing the "task difficulty" into principled, controllable di- mensions inspired by CLT, CogniLoad provides a more insightful perspective than single-dimensional benchmarks. It allows a more differentiated understanding of LLM reasoning capabilities and limitations, paving the way for more targeted development of robust and generalizable Al systems.
5 CONCLUSION
We introduced CogniLoad, novel synthetic benchmark grounded in CLT for multi multi-dimensional evaluation of LLM long-context reasoning. By independently controlling parameters for intrinsic cog- nitive load (d), extraneous cognitive load (p), and task length (N), CogniLoad offers unprecedented diagnostic precision. Our evaluations revealed task length as a dominant performance constraint and uncovered unique "cogmitive fingerprints" of LLM sensitivities to different load types, providing actionable insights beyond single-dimen mngle-dimensional benchmarks. CogniLoad offers a reproducible, scal- able, and petrreicalllyggoondde the development of more capable and robust AI systems. While human and artificial cognition are mechanistically distinct, applying frameworks such as CLT to AI evaluation can provide valuable perspectives for understanding and characterizing their operational differences and capabilities.
tool to systematically dissect L.LM reasoning limitations and guide
6 AUTHOR CONTRIBUTIONS
D.K conceived the idea, designed and implemented the generation algorithm, prepared the dataset, scored the LLMs, analysed the empirical results, designed and implemented the evaluation pipeline, and wrote the first draft of the manuscript. B.R. compiled the first draft of the literature overview and acquired the computational resources for the evaluation. All authors contributed refining the benchmark design and in editing the manuscript towards the final version.
10

---

7 LLM DISCLOSURE
LLMss were used only in the early stages of writing the paper to refine phrasing and correct grammar. During coding they were used update the chart formatting code, and to translate the manual implementation of the generation algorithm Elixir to python for reproducibility.
REPRODUCIBILITY
The code for generating CogniLoad puzzles according the algorithm described in this paper is provided at: https: /ggthuu......... com/ kaiserdan wwwpeeerdnnnnnnnonooooooooeoaaoe cogniload, The dataset of the puzzles on which the LLMs were evaluated for the results presente in this paper is provided on HuggingFace: bttps://huggingfaee http://ececcaaaaatt/////oooooooo
ACKNOWLEDGEMENT AND SUPPORT
This work supported by the Research Council of Norway through its Centres of Excellence scheme, Integreat Norwegian Centre for knowledge-driven machine learning, project number 332645.
Ali Ramezani-Kebryy was supported by the Research Council of Norway through FRIPRO Grant under project number 356103.
We acknowledge NRIS Norway for awarding this project access to the LUMI supercomputer, owned by the EProHPC Joint Undertaking, hosted by CSC (Finland) and the LUMI consortium through Sigma2, project number nn 2027k.
REFERENCES
C. An, S. Gong, M. Zhong, X. Zhao, M. Li, Zhang, L. Kong, and X. Qiu. L-eval: Instituting standardized evaluation for long context language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pages 14388- 14411, 2024.
Y. Bai, X. Lv, J. Zhang, H. Lyu, J. Tang, Z. Huang, Z. Du, X. Liu, A. Zeng. L. Hou, et al. Longhench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3119-3137, 2024a.
Y. Bai, S. Tu, J. Zhang, H. Peng, X. Wang. X. Lv, S. Cao, J. Xu, L. Hou, Y. Dong, et al. Longbench v2: Towards deeper understanding and reasoning on realistic long-contex: multitasks. arXiv preprint arXiv:2412. 15204, 2024b.
P. Chandler and J. Sweller. Cognitive load theory and the format of instruction. Cognition and instruction, 8(4::293-332 1991.
Chollet, M. Knoop, Kamradt, and B. Landers. Are prize 2024: Technical report. arXiv preprint arXiv:2412.04664 2024.
K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
K. A. Ericsson and W. Kintsch. Long-term working memory. Psychological review, 102(2):211, 1995.
Gkamradt. Needle in haystack pressure testing 1lms, 2023, URL https::/githut. gkamradt LLLMTesM Needle IneddeenAAaayttackk
D. Guo. D. Yang. H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu. S. Ma, P. Wang, X. Bi, al. Deepseck-rl: Incentivizing reasoning capability lmms via reinforcement learning. arXiv preprint arXiv:2501. /2948, 2025.
11

---

G. S. Halford, W. H. Wilson, and S. Phillips. Processing capacity defined by relational complexity: Implications for comparative, developmental, and cognitive psychology. Behavioral and brain sciences, 21(6):803-82) 831, 1998.
C.-P. Hsieh, S. Sun, S. Kriman, S. Acharya, D. Rekesh, F. Jia, Y. Zhang, and B. Ginsburg. Ruler: What's the real context size of your long-context language models? arXiv preprint vrXiv:2404.06654 2024.
W. Kryšcińsski N. Rajani, D. Agarwal, C. Xiong, and D. Radev. Booksum: A collection of datasets for long-form narrative summarization. arXiv preprint arXiv:2105.08200 2021.
Y. Kuratov, A. Bulatov, P. Anokhin, I. Rodkin, D. Sorokin, A. Sorokin, and M. Burtsev. Babi- long: Testing the limits of Ilms with long context reasoning-in---hhaytacc Advances in Neural Information Processing Systems, 37:106519-10055 2024.
Lieder and T. L. Griffiths. Resource-rational analysis: Understanding human cognition as the optimal use of limited computational resources. Behavioral and brain sciences, 2020.
B. Y. Lin, R. L. Bras, K. Richardson, A. Sabharwal, R. Poovendran, P. Clark, and Y. Choi. Zebralogic: On the scaling limits of lims for logical reasoning. arXiv preprint arXiv:2502.01 100, 2025.
Z. Ling, K. Liu, K. Yan, Y. Yang, W. Lin, T.-H. Fan, L. Shen, Z. Du, and J. Chen. Longreason: A syn- thetic long-context reasoning benchmark via context expansion. arXiv preprint arXiv:2501.15089 2025.
H. Liu, Y. Ding, Z. Fu, C. Zhang, X. Liu, and Y. Zhang. Evaluating the logical reasoning abilities of large reasoning models, arXiv preprint arXiv:2505. 11854, 2025.
J. Liu, L. Cui, H. Liu, D. Huang, Y. Wang, and Y. Zhang. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. arXiv preprint arXiv:2007.08124, 2020.
A. Modarressi, H. Deilamsalehy, Dernoncourt, Bui, R. A. Rossi, S. Yoon, and H. Schütze. Nolima: Long-context evaluation beyond literal matching. arXiv preprint arXiv:2502.05167, 2025.
F. Paas, A. Renkl, and J. Sweller. Cognitive load theory and instructional design: Recent developments. Educational psychologist, 38(1):1-4, 2003.
M. Parmar, N. Patel, N. Varshney, M. Nakamura, M. Luo, S. Mashetty, A. Mitra, and C. Baral. Logichench: Towards systematic evaluation of logical reasoning ability of large language models. arXiv preprint arXiv:2404.15522 2024.
U. Shaham, E. Segal, M. Ivgi, A. Efrat, O. Yoran, A. Haviv, A. Gupta, W. Xiong, M. Geva, J. Be- rant, et al. Scrolls: Standardized comparison over long language sequences. arXiv preprint MrXiv:2201.03533 2022.
U. Shaham, M. Ivgi, A. Efrat, J. Berant, and O. Levy. Zeroscrolls: A zero-shot benchmark for long text understanding. arXiv preprint arXiv:2305. 14196, 2023.
K. Sinha, S. Sodhani, J. Dong, J. Pineau, and W. L. Hamilton. Clutrr: A diagnostic benchmark for inductive reasoning from text. arXiv preprint arXiv: 1rXiv:1908.000 /6077 2019.
M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.0925 2022.
J. Sweller. Cognitive load during problem solving: Effects on learning. Cognitive science, 12(2): 257-285, 1988.
Sweller. Flement interactivity and intrinsic, extraneous, and germane cognitive load. Educational psychology review, 22:123-138, 2010.
Y. Tay, M. Dehghani, S. Abnar, Y. Shen, D. Bahri, P. Pham, J. Rao, L. Yang, S. Ruder, and D. Metzler. Long range arena: benchmark for efficient transformers. arXiv preprint arXiv:201 2011:0000.00000 140006, 2020.
12

---

K. Vodrahalli, S. Ontanon, N. Tripuraneni, K. Xu, Jain, R. Shivanna, J. Hui, N. Dikkala, M. Kazemi, B. Fatemi, et al. Michelangelo: Long context evaluations beyond haystacks via latent structure queries. arXiv preprint arXiv:2409. 12640, 2024.
K. Yan, Z. Ling, K. Liu, Y. Yang, T.-H. Fan, Shen, Z. Du, and J. Chen. Mir-bench: Benchmarking I1m's long-context intelligence via many-shot in-context inductive reasoning. arXiv preprint arXiv:2502.09933, 2025.
W. Yu, Jiang, Dong, and J. Feng. Reclor: reading comprehension dataset requiring logical reasoning. arXiv preprint arXiv:2002.042: 2020.
Y. Yu, Q.-w. Zhang, L. Qiao, D. Yin, F. Li, J. Wang, Z. Chen, S. Zheng, X. Liang, and X. Sun. Sequential-niah: A needleein--ahhayttack benchmark for extracting sequential needles from long contexts. arXiv preprint arXiv:2504.04713 2025.
M. Zhong, D. Yin, T. Yu, A. Zaidi, M. Mutuma, R. Jha, A. H. Awadallah, Celikyilmaz, Y. Liu,
X. Qiu, et al. Qmsum: A new benchmark for query- based multi-domain meeting summarization. arXiv preprint arXiv:2104.0555 05938, 2021.
Y. Zhou, H. Liu, Chen, Y. Tian, and B. Chen. GSM-infinit: How do your LLMS behave over infinitely increasing context length and reasoning complexity?, 2025. URL httts::/aaxa org/abs/ 2502 05252
Q. Zhu, F. Huang, R. Peng, K. Lu, B. Yu, Q. Cheng, X. Qiu, X. Huang, and Lin. Autologi: Automated generation of logic puzzles for evaluating reasoning abilities of large language models. arXiv preprint 2XXv:2502211606 2025.
13

---

APPENDIX
The supplementary material is organized as follows:
Appendix
investigates p relative to N and d.
Appendix
analyses and discusses interaction effects among d, N, and p
Appendix
charts model performance reported with error-bars.
Appendix discusses the evaluation, scoring pipeline, and error taxonomy.
Appendix reports the distribution of error types across models and different levels of and N.
Appendix compares the AIC values of linear vs quadratic p in the GLM regression.
Appendix details the complete attribute ontology.
Appendix lists the token-length distribution of prompts and output models per model by
d and N.
A INVESTIGATING RELATIVE TO N AND d
In Figure 1 of the paper we discover characteristic U-shape of the performance of LLMs on CogniLoad relative to p (i.e.. the ratio of distractors to essential elements). The following figures investigate this U-shape at different levels of difficulty (Figure and statement length (Figure
d=
1.0
0.8
Accursicy (mean)
0.6
0.4
50
Heyd [%6]
Needle-to
Needle-to-Hay
DeapSeak- R1-0628
2hhi4-reassning.oo plus
Owen3-a8
gpt 2pp-5-2025-08- 20125-08-07
DaapSeak-RR www.R---iil--llaa--00
Phi4--hiirrssrring
Owen3-1178
Deep Seee-eeee-eeiiieee Bww.--tDDiil----------
Phi-4-eeseaning
gemini-2.5.pc pro
gpl-S-nano 2025-08-07 08-07
DeepSeak- R1- R1-DRRil----eeeeeeeeeeeee Di-Deell-Qwer-7
QwQ-320
gemini-2.5 flahhshh
03-2025-04-16
DeepSeak-RR R1-D--R--------- Distll-wenn
13-32B
gemini-2.--tash-
o4-mini-2200 04-15
EXAONE-Desp-528
13-30BBA3B
Figure 3: Distractor ratio p relative to d for d {1,5, 10)
Figure 3lott mean accuracy as a function of the needle-to-hay ratio (higher means more signal per distractor) under three difficulty settings d 1, 5, 10. For the easiest tasks (d = left) accuracies are high and largely flat; most models show at most shallow U-shape, with mild dip around p = 5. centre) the trough 10 % to 25 % and bottom out across = 25 %-50 % before rebounding at high p. notable exception EXAOONE-Deep-220 which trends downward as p increases for d 2 5. Under the hardest condition (d 10, right) capability tiers separate. Frontier models remain robust and still recover at high p averaged over GPT-5 0, 821, 03 0, 804, GPT-5-mini 0, 675, 04-mini 0, 646, and Gemini-2.5-Pa 0, 654 254-whereas mid-size and small models 70B 0, 361; QwQ-32B 0, 300; Qwen3-32B 0, 280; Phi-4-reasoning 0, 256; Owen3-1,,BB 0. 110). Overall, the panels show that (i) accuracy shifts downward and the U-shape becomes more pronounced as increases; (ii) recovery high p is strongly model-dependent and favours larger, reasoning-tume
25 %-50 % and a small recovery by p > 75 %. At intermediate difficulty widens: many curves drop sharply from
stay low despite a late uptick (e.g.. DeepSeek-RR-eee R1-0528 0, 475; DeepSeek-R1- Di-tssi-----iiill----ea
14

---

systems; and (iii) the moderate-p p regime (p = 25 25%50 remains the most adversarial, where balancing signal and noise is hardest.
100
N 250
1.0
0.8
Accuracy (mean)
0.6
75
Needle-to-Hol [%]
Noedle-to [%]]
Noedle-to-Hay [%)
DeepSeek- B18-8-66--1
Phi-4-messoning-luu
Owan3-58
gpl-5 2pk-5-2025-08-07
Rt-Dietil-Uama---
Phi-t-mirirressrinn
3pt-5-minn-0025-00000
DeepSeak-RY
Beeeeak------e------- Dietil-Qwen-B2
Phi4-reasaning
gemini-2.5ppr
spt-5 2025 05-07
DaepSoak- Rt--iddl--Geer--7
gemini-2.5-nash
03-2025-04-5
DeepSeek-R11 Www.RRtDDiil--wi---. 1.5B
Qwer3-332
gamini22.5ffaitt--e
04-mim--2025-04-16
EXAONE-Deep-S2B
Qwen B3en-3-30BAA3B
Figure 4: Distractor ratio relative to N for N E (20, 100,250)
Figure shows mean accuracy function of the needle-to-hay ratio p for N 20 (left), N 100 (centre), and N = 250 (right). At N = 20 many systems still display a clear U-shape: accuracy begins near 1, 0. dips at medium clutter around p = 25-50 %. and then rebounds as needles dominate. As the context grows, this dependency compresses and then largely disappears. At N = 100 the strongest models are nearly p-invariant (e.g., curves remain between about 0, 93 and 0, 97), while mid-tier models retain only shallow trough. By N 250 most curves are flat or gently increasing with p at much lower levels; only the frontier models exhibit a modest late uptick.
Taken together, the two figures show that difficulty d and length N both depress accuracy but modulate its dependence on p in different ways. Increasing accentuates the classic U-shape (i.e., many models drop from م 10 % into the mid-range p 25 %-50 % and then recover needles dominate) whereas increasing N shifts the whole curve downward and progressively flattens it. These stresses reveal clear stratification: frontier models remain comparatively robust (retaining and partial p-invariance). a mid-tier exhibits a pronounced but narrowing mid-p trough as N grows, and small models settle at low, nearly p-insensitive performance. In short, the mid-p band 25-50 % is the most ad adversarial at fixed N and d, but at long contexts the primary failure mode becomes N itself rather than p.
high-p gains
B INTERACTION EFFECTS AMONG d, N, AND P
To probe whether load dimensions compound each other in predictable ways, we fit, for each model. logistic GLM that augments the main-effects specification with all pairwise and three-way interactions among intrinsic difficulty d, effective length log N, and the needle ratio while retaining quadratic main effect for to capture the observed U-shape:
logit P(correct) = Bo + [d, log N, p] + [d:log1o N, d:p, logı0 N:p] + [d: N:pl + p².
In Table 3 we report for each interaction the coefficient 3 and ANOVA-style likelihood p-value (PLR) obtained by dropping that interaction from the full model.
B.1 GLOBAL PATTERNS
Three robust patterns emerge across models:
A strong d x interaction is widespread and negative: 17/22 models show PLR. < 0.01 with Bax (e.g., QwQ-32B: -0.39, PLR 1.25e- 33; Phi-4-reasoning: -0.45.
15

---

PLR 1.94c-30). This indicates that increases in intrinsic difficulty and sequence length compound to reduce accuracy additively. Frontier systems (gpt-5, o3) show no detectable d x N interaction (all PLR 0.3), consistent with their near-separable responses to d and at current loads.
A clear d interaction present 13/22 models (all 9xxp PLR e.g., QwQ-32B: -0.96, EXAONE- EXAANE-Deepp-E difficulty exacerbates the harm of distractors. Several strong models remain p-invariant (c.g.. gpt-5, o3; PLR 0.2), and DS-Llama-708s is marginal (PLR 0.055).
PLR S 0.01 in those cases; -0.89). This confirms that higher intrinsic
The N x interaction is model-dependent and less ubiquitous (significant in 4/22). Two qualitatively distinct regimes appear: for some mid-tier models the coefficient is posi- tive and significant (Qwen3-30B-A33) +1.16, PLR 7.18e-5; 5; EXAONE- -.ONE-Dee--33B +0.56, PLR = 0.038), indicating that increasing the needle ratio attenuates length-related degradation (i.e., more signal per distractor helps especially longer sequences). In contrast, other models show negative and significant N x IDS-Qwen-7BB -0.81, PLR 0.0025; gemini-2.5-pro: -8.61, PLR 0.0021), consistent with long contexts amplifying sensitivity to distractors. Several additional models exhibit borderline positive evidence (0.05 < PLR 0.10; e.g., Qwen3-32B, Qwen3-8B).
16

---

Table 3: Interaction analysis across intrinsic difficulty d, effective length log N. and needle ratio p. Coefficients (9) are from a per-m ser-model logistic GLM with all
pairwise and 3-way interactions plus quadratic main effect for pt PLR are likelihood-ratio p-values from dropping each interaction from the full model. Stars: #5 p 0.001, #* 0.01, * p < 0.05.
N p)
0.997
0.014"
0.211
13
07**.
15
08
0.00696
0.0199
3.72e 12
07
0.134
0.000104*
05
0.67
0.0582
0.00891
0.0762
0.838
0.569
0.0922
0.81
8.14e
9.41e
2.12c
1.08c
3.55e
5.36e
PER (d
NaxNNa
-0.00
0.14
0.08
0.36
0.26
0.42
0.30
0.18
0.15
0.36
0.27
0.08
0.21
0.21
0.16
0.31
1.61
11.0
-0.06
-0.10
-0.62
0.07
0.421
0.433
0.402
0.702
0.00246
0.0378
0.886
0.0917
0.124
0.652
0.887
055**
0.0965
0.0767
0.133
0.171
0.00211*
0.197
0.756
0.0603
0.131
0.979
PLR (N x
7.18e
1.41
0.28
-0.25
-0.1
-0.81
0.56
-0.04
0.59
0.54
-0.13
0.04
1.16
0.53
0.50
3.05
1.20
0.00601* ** -8.61
5.23
0.48
1.82
1.72
0.05
PLR (d p)
0.89
0.0552
0.195
1.84e - 17
1.41e - 07
20
05 TH
0.000821
3.77e - 23
09
0.00919*
6.88c - 08
- 07
0.681
0.00686**
0.276
0.985
0.929
0.212
0.856
2.73c
2.4c
2.06c
7.48c
8.97e
Bax
0.07
-0.21
-0.15
-0.79
00.51
0.89
00.62
-0.0
-039
-0.96
11 -0.56
-0.5
-0.56
0046
-0.3
-0.9
3353
1.41
0.01
-000
0.12
PLR x .)
0.864
05
0.706
32
07"*
17
05
30
25
33
09
10
14
0.0011
0.00554*x
08
0.685
0.00034**4
0.393
0.353
0.00829*
3.4le
4.07e
7.63e
1.28c
4.96c
1.94c
4.28e
1.25c
7-33e
4.57e
1.76c
6.63e
1.84 7.8le
94x N
-0.03
-0.15
-0.01
-0.37
-0.66
0.27
-0.13
-0.45
-0.41
-0.39
-0.20
-0.9
-0.22
-0.24
-0.58
-027
0.13
-0.56
0..9
-.19
0.45
DeepSeck-Ne R1 Disttill- Llama--aa---- 70B
DeepSeok -Distill-- Distill-Qwen-Il Sal---nnnnii 1.5B
DeepScok-B R1 Distill-Qwenn33 32B
DeepSeek RI -istiill-QQeee Qwen-TQeee 7B
apt-5-nano-225 2025-08-77
DeepSeek- RI -0528
EX AONE- Deep- 32B
4- mini mimiiiiiineoooo reasoning
4hi-4 reasoning plus
gemini-2 2.mini-225--0 -255-255 f--ffahh--iit lite
gpt-5-minn 11--5-mnii-222-- 20ii-2222-0--00
o4-mini- 2025-04-55-
Qwen3 33B-A3B
gpt-5 00-5--222-08-- 08-07
ww---namo-2025--11
04-mini-2025-04--
Phi Phi-4-re reasoning
gemini-2.55ffass 3-fllshh
gemini-2.5- pro
10-2025-04-16
Qwen3-1.7 1.7H
Qwen3-322 -32B
QwQ-32B 32B
Qwen3-8B
Model
17

---

B.2 THREE-WAL WAY EFFECTS
A positive three-way interaction d x N x is significant in 12/22 models (e.g., DS DS-Qwen-322: +0.36, PLR 8.1e-13; miiuronnnn- +0.30, PLR 1.1e-8; QwQ-32B: +0.36, PLR 3. 12)..1). Given the quadratic main effect in this should be read as moderation of the slope with respect to p: when tasks are simultaneously long and intrinsically complex, changes in p tend to have larger beneficial marginal effect (i.e., increasing the share of needle statements mitigates the compounded X N burden more than when only one of or is high). Frontier models show near-zero slightly negative three-way coefficients (gpt-5: 1.05, PLR 0.076; o3: -0.62, PLR = 0.092), aligning with their weak p-dependence overall.
B.3 PRACTICAL IMPLICATIONS
Beyond main-effect sensitivities, many models exhibit genuine interaction structure: high d and high N jointly depress accuracy more than additively, and high d amplifies distractor harm (d p < 0). For substantial subset of models, increasing the needle share (p) yields greater gains specifically when tasks are both long and complex (positive d N x p): providing a concrete knob for mitigating compounded failure modes. It is important to note however that for models with long-context budget issues (e.g., Demini-225-ppre see Appendix Appendix H). N x estimates can be dominated by those failures, inflating effect sizes.
C MODEL. PERFORMANCE WITH ERROR-BARS
Due the small size and the large amount of models we compare in Figure of the main paper we omit error-bars to improve legibility. In this section plot larger versions of the charts in the panel with vertical error bars representing 90% confidence intervals for the mean accuracy of each model-condition pair.
We compute these confidence intervals with the Wilson score method for a binomial proportion. Specifically, for given point with & correct answers out of trials we p k/n and the 90 % standard-normal quantile 1.644853627 to obtain
(1 p) + 2² /4n)
2n
-
21
[4, u]
2
The resulting interval [7, u] marks the range that would contain the true underlying accuracy in 90% of repeated experiments with the same sample size. Wider bars correspond to greater sampling uncertainty, whereas tighter bars indicate more stable estimates.
The narrow error bars we observe substantiate the discussion of the results in the main paper and highlights a significant difference between the curves at 90% significance and the presence of the characteristic U-shape.
D EVALUATION, SCORING PIPELINE, AND ERROR TAXONOMY
This section fully details the evaluation and scoring pipeline functionally in text and in pseudo code.
D. INPUTS, FILES, AND NAMING
The scorer consumes three artefacts per puzzle instance 2:
1. Model output of the reasoning trace response (plain text).
2. Puzzle text (for the question): The last line is the natural-language question.
3. Puzzle metadata (JSON state): providing the person-of of-interest (Pol), gold label, needles, difficulty, statement count, and category domains.
18

---

DaepSeakR1-052
DeepSeek-R1-
DeepSookK1:
EXAONE-Deep-
Pht-A-mindreessonnng
Phi 4ineassannnn
Accuracy (mean)
PH1-4/raasoonnngguu
QwQ-32B
Owen3-1.76
Gwen3-33B-
Gwen3-328
Qwen3-8B
ganiri-2.5-fahh
gemins--2..ffaahhbbe
gemini-2.S-pra
gpt-5
2pp-6-mmi-2225-0--22
gof-5-rero-2225-08-48
0.0
o3: 03-2025-04-18
06-mir-2992-00-11
Intrinsic difficulty
Figure 5: Accuracy relative to difficulty d for d E (1,3,5,7,10) on the X-Axis
DeepSeekRR10522
Dislll-Owwe-322
PeepSeekRtt-istll--eee
0.8
EXAONE
Ph1-4-iiri-rassonnng
Phi-4-ressonnng
Accuracy (mean)
Phi-t-reaspninnpite
0.6
Qwen3-1.76
0.4
Gwen3-32B
Owwn3-8B
gemini-22 2--iii-2.--lassh
gemini-2-ffashllte
0.2
gamini-2.55pr
get-5- 9x-5-2025-08--7
got-5-mim--225-008-77
gst- 1ot-8-raan-222 02-88rano-2025-08-07
2025-08-07
03-2025-04-15
10
25
50
75 90
95
04-mirl-225-044
Needle-to-Hap Hay [%]
Figure 6: Accuracy relative to distractor-ratic p for p {5, 10, 25, 50, 75, 90, 95) on the X-Axis
D.2
CATEGORY MAPPING AND QUALIFIERS
The target category is resolved from the last line question using deterministic templates:
s startswith ("Where is") - location
startswith ("What color shirt") chothes_shirt
('artswith""Whtt color pant") clothes_pant
startswith("What color hat") clothes_hat
startswith ("What color of socks") + clothes_socks
(aaatswwih("Whaae color of gloves") + clothes_glooess
startswith ("What color of underwear") + +ccothhs__nderwees
startswith ("What is the final hair color") hair
endswith ("most recently eat?") recent_eat
endswi th ("recentll watch?") reccnt_watch
19

---

DaepSeekR1-0520
DeepSeek-R1-
DeepSookK1:
EXAONE-Deep-
Pht-A-min-reesoonnng
Phi 4ineaasasnnnnngg
Accuracy (mean)
PH1-4/raasoonnngguu
QwQ-32B
Owen3-1.76
Gwen3-33B-
Gwen3-328
Qwen3-8B
ganiri-2.5-fahh
geminn-22..fishh-ll
gemini-2.S-pra
gpt-5
1pp-5-mmi-2225-00-42
gof-5-rero-2225-00-44
0.0
o3: 03-2025-04-18
o4-mini ol-min-2925-04-18
Update length
Figure 7: Accuracy relative to total statement length N for N {20, 50, 100, 250} the X-Axis
endswith ("recently listen to?") + recent _isten
endswith ("recently read?") + recent_reat read
Qualifiers (used to find "valid Pol sentence"):
location: fat, located, in]
clothes_shirr (shirt, wear]; clothes_ pant: |pant, wearl; clothes hat: [hat, wear]; sootthes_gloveee (glove, wear]; sothhe_sscckss (sock, wear); clotthes_udderwea (underwear, wear)
hair: (hair)
recent_eat:
(eat, ate): recent_watch, (watch, watched, movie); recent_listen (listen, listened, music); recent_reaa (read, book)
The Pol name and the domain values for the target category are read from the puzzle metadata file.
D.3 NORMALIZATION, SENTENCE WINDOWS, AND CONTEXT BUDGET
Normalization:
Lowercase the full model output; split by newlines.
If the last line starts with "(" and ends with """, drop that line because it's usually just clarification that follows the final answer.
Drop empty lines; if no lines remain, keep single empty line.
Sentence windows:
1. valid_poi Lac__poollsstttt sestence: last line containing Pol and any category qualifier; take TakeLast
2. moi_last Dast_sentenee last line containing Pol; take TakeLast of that line.
3. sentence: apply Takelasttentencee to the last remaining line; below.
TakeLLsstSentenco(s):: let p = split(s, "."). If (en(p)) 2 2, return p[-2]: else return s.
Context limit failure: mark wwwng if any of the following holds: 32'768,
last_sentence "" empty output.
20

---

The 20 additional tokens added to the response _ooeessttteeeeeeeee is necessary because when models usually produce incomplete responses a few tokens before the maximum-outputttoker. limit they prompted to decode with.
D.4 MATCHING, SYNONYMS, AND ALTERNATIVES
Since some models respond with the answer attribute starting with a special character instead of an empty space (like a word) the containment accepts the following alternative word boundries (code CheckAnswez wer (term, sentence) returns true if any holds);
sentence. stwrtswiiiii..... th (term) or sentence = term
substring appears prefixed by one of: space, "T", com coger "."""
Since some attributes in the attribute taxonomy could be spelled alternatively, relax Synonym whitelist for them in the evaluation (code CheckCorrect (gold, sentence)):
reaggea: accept reaggea or reggae
sci fi: accept sci-fi, science fiction, science fiction
camp: accept camp, campground
potatoes: accept potatoos, potato
market: accept market, marketplla
livingroom: accept livingroom, living room
otherwise: accept exactly gold by boundary-tolerant containment
We further ensure no alternative value is present in the last sentence and therefore compute (A (AltFlag (sentence, alts, gold)) as:
1. Let lowercase(sentencce) g lowercase(gold), and A lowercase alternatives domain values \gg)-
2. Define present(t) = CheckCorreettnssee(t..
3. If no alternative is present, return false.
4. If both gold and at least one alternative are present, keep the alternative flag only if some alternative's character span encloses the gold's span in ("spans" test); otherwise clear the flag. This done to accommodate attribute values like "non-fiction' which contain alternative values like 'fiction'.
5. Since sometimes models respond by mentioning also previous value of the attribute (eg sentence like "... the final step explicitly converts gray to green, resulting in **green*** x") we do not set the flag if the gold flag is the last mentioned attribute the sentence.
D.5 BUCKETS, PRECEDENCE, AND ACCURACY
Let:
lastt_ill__lpi__s____e_eeeeeee Cvalid cheekooreettasseeeoooddaaa...... Looi__lstt__eee last_sentence),
Cpoi = cheetborrccaaverrrooopp....eeeer
Clast Cheer/oocrettaneerrrrooaaaasesee
avalid = a.a.Thaahhallgppe.aaaaee.eeeteec
Apoi AttePlofoollaatteeeeenneaaaoos_p
Ojast ALIFLLag(aatteenttnneeaa......
Classification (first applicable wins):
1. if Gralid and not Ovalid' correct_val
21

---

2. else if Cpoi and not Opot correct_poi
3. else if Clast and not Glast correct_last_sentent
4. else if not Cvaldd and avalid and wadivalddpo__ssttseeteeeee # wrong_logic
5. else if not Cpoi and Opii and
"": wrong_logic_pol Alstt and last_sentence * "":
6. else if prong_logiicllastsennee
Clast and
7. else: wrong_other
Note:
When sentence contains both gold and alternative that spans the gold, this does not trigger
srong_logiis bucket unless the corresponding "not c_ and a_"" guard holds; such cases can fall through wrong_other, matching the imple- mentation. The main figures/regressions of the paper use accuracy with correct ccorrect_valice correct_ correct_lasteeeeeeeeeeeeeessssse jast_sentence] and correct as the comple- ment.
D.6 PSEUDOCODE
Using pseudo code, figure Deeeeeeepppl explains the data preparation, figure[explains the scoring of the results and success/failure types, and figure BguueBBettil details the used utility functions.
Algorithm 1: Pseudocode the data preparation.
// Parse identifiers and gold from puzzle name:
N c
(N, d, puzzllld,,, gold) < PARSE(ans_patt))
// Load question and resolve category
question -- [
category <- NagghesttiorTooatttgor//qgggsnn
// Load metadata needed for evaluation
(poi, domains) RRA.D/OO(staateeesoooooooooo
poi + LOWER domain Values + domains (cateoor::
quals + GetQuualiflerreFo (catteoorr::
// Read and normalize model output
resp READ(ans_path))
lines < SPLLT_IINESSLLOWWRRRRRE
STARTSWWHH(ieees-1] (") (NDSWITH((ineS(1))) then
DROP (ine_
lines < FILTER BICEER_NNNMPPP(Miin
(FN(Nline)) then lines < [']]
E
ERROR DISTRIBUTION
The following tables illustrate the error-type distribution per difficulty 4 and length 5 The purpose of this distinction is to illustrate three primary error types (i.e., long context error, logic error, other error) and distinguish among the logic errors how well the model followed the instruction in its response formatting. The other error captures cases where the model responds but doesn't mention any attribute, usually this cases where the model erroneously assumes the problem can't 't be solved with the provided information. For conciseness the error types in the table headers are abbreviated as following:
Wrong max-context -
Wrong logic (last valid POI sentence) valid-logic
22

---

Algorithm 2: Pseudocode of the CogniLoad scoring pipeline.
// Prepare data
// Extract windows
last_sentence - (inee
poi_lines - {s lines: (ONTAINS(s, poi)];
poi_lass + LLLLLAOOREMPT(ppiiiiieee linee);
valid poi Jinel {s E poi lines CONTAINSANY(S,.quil)
loo_lastttaaappp valid poi last - LASTOREMTTOO // Budget//emptinesss guard
poi [ii_i
if Excceddddoottext (ranttte_mmmmmmtetti V =os__sentence="" = V LEN((esp) = 0 then
L return wrong_ maaa_aootttea
// Gold matches
Cralid + CheckCorrecttAsswer(ggl valid_ poi last;;
Cpoi < CheckCoorrecttnswwerggd... pao_llsst
Gast ChhcckCorreettnsswrrrrrre
last_ last_sentence);
// Alternative flags with apan rule
alts {v E NomminnVluuee * gold);
avalid - AltFlaggvalid poi last, alts, gold);
apoi AltFlag last, alts, gold);
Dast - AltFlag(last_eenn sentence, alts, gold);
Buckets (first match wins)
Coalid A -0valdd then
return correct_valid
else if poo A poo then
L return correct_poi
else if Gust -0jast then
L return correct_last_sentee
else -calid A avalid A valid_poil lacliddoppllsttteellee last + "D then
return wrong_logis
else if -Col A apol A poi last then
L return wrone_logic_pot
else if -Cast/ 4jst A last last_sentence * nn then
L return
else
return wrong_other
Wrong logic (last POI sentence) poi-logic
Wrong logic (last sentence) last-logic
Wrong other + other
If the total number of evaluated samples does not precisely match the expected number of samples (e.g 103-2225-0416 with difficulty resulted 279 vs 280 expected examples), this usually indicates the puzzle was not processed due to API error (e.g. security refusal). These few missing cases were attempted to be evaluated a second time but refused again.
Table Error-type distribution by difficulty d per model (aggregated over
N and p).
Model
Dim
Samples
Acc [%]
ctx
valid-logic
poi-logic
last-logic
other
DeepSeek-R1-0528
280
88
3
31
0
0
0
DeepSeek-R1-OS5
3
280
78
11
52
0
0
DeepSeck-RI -1-0528
5
280
67
13
80
0
0
DeepSeek-RR R1-0528
7
280
55
16
109
0
0
DeepSeek RI R1-0528
280
48
19
128
0
0
23

---

Table Error-type distribution by difficulty d per model (aggregated over N and rho). (continued)
Model
Dim
Samples
Acc [%]]
ctX
valid-logic
poi-logic
last-logic
other
Deepstek-RI-DDtiilllllmmm-
1
2800
82
0
507
0
2
9
ReesSeek-KI-Distil--L-mm---
3
2800
68
11
877
3
DeepStek-KI-DDittilLLLam----
5
2800
54
12
1260
0
8
2
DeepSeaak-RI--Diiiilllaaaa-----
2800
48
28
1424
0
4
DeepSeckkRR-DDisillllammaa---
10
2800
36
33
1747
0
4
5
DeepSeaa-RR--Disiiil--aaaaa-
2800
561
1140
4
81
154
DeepSeckkKL-Disstll--aaa----1
3 2800
19
543
1305
3
82
347
DeepSsekkKK--Diiiii..eeee.....
5
2800
12
613
1381
4
63
413
DespSSekkKR--Disiii---eeee...
7
2800
10
695
1323
7
60
443
DeepScckKRR--DDttii--wveeL..LL
10
2800
8
717
1371
7
62
407
DeepSeak-KL-Diisil---va-=--
2800
63
1023
0
3
DeppSckkKI--iistll---w----2
3
2800
66
46
908
0
2
I
DeepSeek
SrepSeek-KI-Diitill----wa..
5
2800
52
78
1275
DeppSeak-RI-Diitillooeaaa.
7
2800
42
94
1532
0
0
3
2800
1813
0
DeepSeek-RR-Dissill-Oouee
1
2800
46
47
1454
5
4
15
DoepSeckkRl--isttll--waaar
3
2800
32
124
1561
5
70
134
DeepSeck-RI--Ditill--wwwnnn
DeepSeck-RI
5
2800
20
246
1706
3
65
206
DeepSeea-RKi-Diisiillowwwaa//
7
2800
14
317
1802
8
49
223
DeepSeek-RL--Dsstil-Weeeen/
10
2800
12
352
1770
6
60
290
EXAONE-
2800
62
0
995
5
2
EXAONE-Deep-522B
3
2800
42
2
1595
2
27
10
EXAONE-Deee-S32B
5
2800
29
10
1938
2
45
EXAONE-Decp-32B
7
2800
24
14
2060
42
EXAONE-Deep-320
10
2800
17
16
2257
40
13
gemini-2.5-flasi
280
89
6
24
gemini-2.5-flass
3 280
71
64
17
0
0
gemini-2.55flan
5
280
63
93
gemini-2.5-flass
7
280
56
116
8
0
gemini-2.5-fR
280
52
110
25
gemini-2.5-flass
1
280
68
0
91
2
gemini-2.5-flass)
3 280
48
5
138
semini-2.5-fass-liit
5 280
36
3
176
gemini-2.5-flash--iie
7
280
22
8
209
gemini-2.5-flash-il
10
280
16
227
2
gemini-2.5-pro
1
280
95
10
5
gemini-2.5-pro
3 280
74
69
gemini-2.5-pm
5 280
72
71
8
gemini-2.5-pro
7
280
69
79
gemini-2.5-pro
10
280
65
88
9
gpt-5-2025-08-07
280
100
gpt-5-2025-08-07
3 280
98
0
gpt-5-2025-08-07
5
280
94
0
17
gpt-5-2025-08-07
7
280
89
3
26
gpt-5-2025-08-07
10
280
82
29
21
-
gpt-5-mini-2025-08800
1
280
92
0
14
gpt-5-mini-2025-08-07
3 280
83
0
46
$pt-5-mini-2025-08-00
5
280
78
3
52
gpt-5-mimi-2025-08..
7
280
74
12
58
0
3pt-5-mini-2025-08-07
280
68
27
63
gpt-5-nano-2025-08.
1 280
76
0
65
gpt-5-nano-2025.44
2025-08-07
3
280
0
133
gpt-5-nano- 2025508807
5
280
48
0
145
0
10455-man-22225-0-.1
7
280
34
0
183
0
I
24

---

Table Error-type distribution by difficulty d per model (aggregated over N and rho). (continued)
Model
Dim
Samples
Acc [%]]
ctx
valid-logic
poi-logic
last-logic
other
gpt-5-nano-2025-08-90
10
280
21
0
219
0
03-2025-04-16
279
96
0
11
0
03-2025-04-16
3
280
93
0
20
0
0
03-2025-04-16
5
279
90
1
28
0
0
03-2025-04-16
7
280
89
2
30
0
0
03-2025-04-16
280
80
21
34
0
0
04-mini-2025-04-16
279
88
0
31
0
0
2
04-mini-2025-04411
3
280
81
0
53
0
0
06-mini-2025-04116
5
280
79
3
57
0
0
04-mini22025004-16
7
279
70
5
78
0
0
04-minn-2025-04-16
10
280
65
13
86
0
Phi-4-mini--cassoiing
2800
49
4
1222
16
48
126
Phi44-mini-reasoninng
3 2800
1613
126
178
Phi-4-mini-reasooiing
5
2800
20
12
1733
8
286
189
Phi-4-mini-reassoningg
2800
16
1785
354
203
Phi-4-mini-reasooning
10
2800
11
18
1856
5
415
187
Phi-4-rcasoning
2800
83
14
417
0
0
42
Phi-4-rcasoning
3 2800
63
79
921
0
0
33
Phi-4-reasoning
5
2800
47
246
1219
0
30
Phi-4-reasoning
7
2800
36
522
1240
0
0
29
Phi-4-reasoning
2800
26
857
1200
0
Phi-4-reasooning-plu
1
2800
83
100
347
37
Phi-4-reassoning-pluss
3 2800
58
182
938
0
0
43
Phi-4-reassoming-luu
5 2800
40
471
1153
46
Phi-4-reasoning-plee
7
2800
30
936
1001
2
0
27
I
Phi-4-reasoonng-plaa
2800
1198
994
Qwen3-1.7B
1
2800
33
1840
1
11
17
Qwen3-1.7B
3
2800
28
2010
2
Qwen3-1.7B
5
2800
19
O
2268
0
I
8
Qwen3-
7
2800
14
0
2396
2
10
1
Qwen3-1.7B
10
2798
11
0
2483
0
5
Owen3-30B-A3B
2800
75
18
652
3
0
Qwen3-30B-A31B
3 2800
54
48
1237
Qwen3-30B-A3B
5
2800
40
1587
2
0
10
Qwen3-30B-A3P
7
2800
31
25
1894
2
0
Qwen3-30B-AMe
10
2800
23
19
2133
3
I
I
8
Qwen3-32B
1
2800
76
626
Qwen3-32B
3 2800
61
1075
10
Qwen3-32B
5 2800
48
1434
3
Qwen3-32B
7
2800
38
0
1721
0
6
9
Qwen3-32B
2800
1994
0
10
Qwen3-8B
1 2800
68
77
739
2
0
76
Qwen3-8B
3 2800
50
136
1190
3
18
49
Qwe33-8B
5
2800
36
106
1619
3
19
37
Qwen3-8B
7
2800
30
101
1791
0
27
39
Qwen3-8B
10
2800
22
78
2046
0
39
25
QwQ-32B
1
2800
62
2
1044
0
5
QwQ-32B
3
2800
69
12
850
0
3
QwQ-32B
5
2800
52
52
1279
0
3
QwQ-32B
7
2800
39
112
1596
0
1
QwQ-32B
10
2800
30
273
1674
0
5
9
25

---

Algorithm 3: Pseudocode of the CogniLood scoring pipeline utility functions. Utilities.
ExceededContext: return true if prompt prompt_tokens- response_tokenn + 20 2 32 768. Takel Take LastSentence((s: p = (P "."). (P) 2 2, return p[-2]] else
CheckkCorretttnnseerrod s):
CheckAnswer(t, 5) true if any holds: STARTS t), t, or contains one of: "space+t", "[t", t", "*t", "_",, "I t","""" "(t".
If gold = reaggea: return CheckAnnswer(reaggea) V CheckAnswwer(regaae)
If scci-fl return CheckAnswer(sciit)) V CheckAnswer(science fiction) V CheckAlnsverrscienn-fficooo
If camp:
CheckAnswer(camp) v Checkk
If potatoes: CheckAAsswer(potatoes) V CheckAAnswer(pottto)))
If market: CheckAnswer(markkt) V CheckAnsserr(manketppaces
livingroom: CheckAnswerlivvinggnnn V (ieckAnsswer(ivingg room).
Else: CheckAnswer(gold)
AltFlag(s, gold):
1. LOWER(S). LoWER(gold), A +- (LOWER(a) alts, g).
2. alt present +- VAEA CheckCorreetAnsser((...
3. g_idz s.)F
4. if g g_idr + -1 and alt present: let g_endded------ = gdid + LEN(g); set spans false. For each a E A: a_id s. S.RFIND(a): if __dirrr + and a a_idr= I g_idr and a_idæ + +LN((a) 2 g_end, set spans -true. If -spans, set alt alt_eet false.
5. Return ale_eretent.
Table 5: Error-type distribution by puzzle length per model (aggregated d and p).
Model
Dim
Samples
Ace [%]
ctx
valid-logic
poi-logic
last-logic
other
DeepSeek-R1-0528
20
350
97
7
2
0
0
0
DeepSeek-R110522
50
350
87
14
30
0
0
DeepSeck-R1-0528
100
350
56
25
128
0
DeepSeck-R1-0028
250
350
27
16
240
0
0
DeepSeek-RI-Disse -I----iiil-iilaaaaaa
20
3500
89
0
372
0
1
50
3500
66
2
1167
0
4
5
DeepSeck-RR1-Disiill-Llmmm
100
3500
2
1811
0
6
DeepSSek-RR--Distilllllaaaa
250
3500
27
80
2465
0
10
11
DeepSees-KK--Disiil---eeee
3500
19
749
1676
8
58
349
DeepScck-RR--Dsttii-QQeen....
50
3500
16
752
1749
5
85
360
DeepSeckRRl-Disslll-vwwa----.
100
3500
15
698
1749
4
90
450
DeepSekkKI--Dittlloyyaa---...
250
3500
14
930
1346
8
115
605
20
3500
78
760
0
0
DeepSeekKRI-Dissll--waaa--222
50
3500
55
1583
0
2
DeepSeek-KI-Diisiillovvee---2
100
3500
41
0
2061
0
I
250
3500
29
323
2147
3
DeepSeck-KI-DDittill--vv---
20
3500
37
117
2012
4
14
63
DeepSeck-RI--istill-oe
50
3500
24
207
2298
43
123
DeepSeck-RI-Dissilloowaaaa
100
3500
20
311
2177
11
75
221
DeepSeek-R1--issill--vaaae
250
3500
19
451
1806
116
461
EXAONE-Deep-32B
20
3500
57
7
1451
4
35
20
EXAONE-Deep-32B
50
3500
33
17
2255
2
56
19
SLAONE-Deep-330
100
3500
25
7
2556
4
34
21
26

---

Table Error-type distribution by puzzle length N per model (aggregated over d and p). (continued)
Model
Dim
Samples
Acc [%]]
ctx
valid-logic
poi-logic
last-logic
other
EXAONE-Deep-32B
250
3500
24
11
2583
1
31
25
gemini-2.5-flash
20
350
0
0
gemini-2.5-flash
50
350
91
10
22
0
0
gemini-2..5flass
100
350
99
39
0
0
gemini-2..5flass
250
350
14
280
20
0
0
gemini-2.5-fllah
20
350
55
0
154
0
0
semini-2.5-flas--iit
50
350
35
0
223
gemini-2.5-flash--
100
350
32
235
0
0
gemini-2.5-flash-lin
250
350
29
16
229
gemini-2.5-pro
20
350
99
0
2
gemini-2.5-pro
50
350
98
gemini-2.5-pmm
100
350
82
48
15
gemini-2.5-pro
250
350
268
10
gpt-5-2025-08-07
20
350
100
0
gpt-5-2025-08-07
50
350
98
0
gpt-5-2025-08-07
100
350
96
0
12
gpt-5-2025-08-07
250
350
76
32
52
gpt-5-mini-2025-0--77
20
350
96
0
12
gpt-5-mini-2025-08--07
50
350
93
0
23
$pt-5mmini-2025-08-07
100
350
84
0
54
gpt-5-mini-2025--0807
250
350
44
42
144
3p+55-nano-2025-008007
20
350
75
86
1in---mao--20550082
50
350
48
0
181
1
ppt-5-mano-2025-08-0
100
350
37
217
gpt-5-namo-2025-008-99
250
350
25
0
261
200000000
3
03-2025-04416
20
350
99
0
03-2025-04-16
50
350
98
0
03-2025-04-16
100
350
94
0
22
03-2025-04-16
250
348
68
24
88
04-mini-2025-044
20
350
99
0
1ammini-2025-4444
50
350
93
0
24
04-mini-2025-04-11
100
350
76
0
85
0
04-mini-2025-04-16
250
348
38
21
193
0
0
Phi-4-mini-reasoning
20
3500
37
8
1765
7
213
215
Phi-4-mini-rcasoning
50
3500
24
2163
289
208
Phi-4-mini-rcasooiing
100
3500
21
13
2198
6
327
222
Phi-4-miin--resooiinn
250
3500
17
2083
400
238
Phi-4-reasoning
20
3500
84
11
496
0
0
38
Phi-4-reasoning
50
3500
56
106
1393
0
Phi-4-reasoning
100
3500
40
332
1736
0
43
Phi-4-rcasoning
250
3500
24
1269
1372
0
Phi-4-reasoningg-phee
20
3500
81
12
586
0
55
Phi-4-reasooing-plue
50
3500
50
382
1326
3
0
Phi-4-reasoning-plus
100
3500
35
668
1563
I
45
O
Phi-4-reasoning-plus
250
3500
20
1825
958
0
Qwen3-1.7B
20
3500
27
2546
5
12
Qwen3-1.7B
50
3500
20
2796
3
13
Qwen3-1.7B
100
3500
19
0
2818
6
Qwen3-1.7B
250
3498
18
0
2837
0
3
18
Owen3-30B-AAB
20
3500
72
25
942
I
12
Qwen3-30B-A3B
50
3500
48
40
1764
2
0
19
Owen3-30B-AAE
100
3500
34
44
2228
0
Qwen3-30B-AR
250
3500
25
41
2569
7
0
16
Qwen3-32B
20
3500
82
0
607
0
I
27

---

Table Error-type distribution by puzzle length N per model (aggregated over d and p). (continued)
Model
Dim
Samples
Acc [%]]
ctx
valid-logic
poi-logic
last-logic
other
Qwen3-32B
50
3500
55
1
1577
0
3
10
Qwen3-32B
100
3500
38
0
2125
0
Qwen3-32B
250
3500
26
1
2541
2
15
31
Qwen3-8B
20
3500
75
912
0
18
Qwe3--8B
50
3500
40
108
1893
1
31
53
Qwen3-8B
100
3500
31
117
2198
3
27
Qwen3-8B
250
3500
23
198
2382
4
27
85
QwQ-32B
20
3500
76
8
810
0
4
7
QwQ-32B
50
3500
55
27
1550
0
1
QwQ-32B
100
3500
42
36
1991
0
5
5
QwQ-32B
250
3500
29
380
2092
1
2
F AIC-COMPARISON
In this section compare the Akaike Information Criterion (AIC) of the GLM model containing pure linear term with a GLM model with squared term added for The equations for the models following with indicating a correctly solved puzzle:
Linear model
Pi(Y=1) (B0 Bad + BN logio N P)
Quadratic model
Pi(Y=1) = oBo + Ba d BN log10 N N 8pp + 8,2
To assess whether the quadratic specification provides a statistically significant improvement over the linear GLM, compare their maximised log-likelihoods (i and fquad- The likelihood-ratio statistic D = 2(fquad - (im) follows, under the null hypothesis that the extra term is unnecessary, a chi-squared distribution with one degree of freedom because the quadratic model introduces exactly one additional parameter.
The p-luuue reported Table 6 is the upper-tail probability p - Pr(xi D). p-values below 0.05 indicate that the quadratic term yields a statistically significant gain in fit and therefore justifies its inclusion.
We find that for all except for two models (i.c., DeepSeek-RR -R RRI-Distll-QWeer 1.5e and DeepSeek-R1- Distill- (waill-Qeen-BBB) the quadratic term for results significantly improved AIC value. Therefore we include the quadratic term in the GLM specification within the paper.
G COMPLETE ATTRIBUTE ONTOLOGY
The following section lists the full attribute ontology of all values available for the categories.
people Peter, Paul, Mary, John, Mark, Jeff, Craig, Daniel, Anna, Amoldo, Ali, Benjamin, Joe. Donald, Mitch, Chuck, Jack, Lucas, Jeniffer, Adam, Greg, Allan, David, Ellen, Fred, Hank, Hubert, lan, Ingrid, Rebecca, Ken, Lewis, Michael, Nathaniel, Oliver, Russ, Steve, Sandy, Ted, Tanya, Veronica, Vincent, Wesley, Brad, Sam, Igor, Sue, Jan, Jeffrey, Jacques, Debby, Olivia, Benedict, Chris, Charles, Harry, Eli, Mahmoud, Chen, William, Linda, Elizabeth, Robert, Jennifer, Emily, Joseph, Thomas, Patricia, Anthony, Jessica, Brian, Lisa, Kevin, Karen, Laura, Eric, Stephanie, Michelle, George, Andrew, Joshua, Amber, Timothy, Victoria, Richard, Cynthia, Brandon, Megan, Matthew, Nancy, Jacqueline, Gary. Dorothy, Edward, Kimberly, Scott, Sara, Justin, Brittany, Ronald, Deborah, Janet, Christopher, Alexander, Samantha, Oscar, Cindy, Frank, Carl, Paula, Irene, Theresa, Dennis, Ralph, Gerald, Martin, Terry, Bryan, Lance, Corey, Casey, Brent, Derek, Travis, Austin,
28

---

model
PLR
AlCiinear
AlCquad
DeepSeek-R1-D0 --0528
0.000
1032.432
1019.177
DeepSeet-RI-Disiil-LLmee
0.000
13888.760
13703.165
1.ven-1.BB
0.657
11582.864
11584.667
0.000
16487.488
16409.647
-eepSee--R--Disilll- -ilee---eeeBB
0.041
14227.155
14224.975
EXAONE-Deep-3225
0.000
15469.929
15388.288
Phi-4-mini-reasoninn
0.000
14462.835
14438.087
Phi-4-reasoning
0.000
13344.906
13161.306
Phi-4-re Phi-4-reasoning plus
0.000
12603.062
12477.924
QwQ-32B
0.000
16357.613
16218.181
Qwen3-1.7B
0.000
13700.396
13684.334
Qwen3-30B-ABH
0.000
15363.614
15252.219
Qwen3-32B
0.000
14818.588
14720.097
Qwen3-8B
0.000
15522.392
15425.100
(eminii-2.5-fae
0.077
807.553
806.420
gemini-2 1omini-2.5-ffae 2.5-flash-lite
0.000
1574.306
1555.886
gemini-2.5- 2:5-pi
0.201
642.222
642.583
gpt-5-2025-00--9
0.001
484.186
474.503
gnt-5-mini-2025-068.1
0.000
1044.689
1023.604
3pt-3-mamo-2025-08-07
0.000
1520.921
1500.138
03-2025-04-16 04-mini-2025-04-16
0.121
692.757
692.356
0.000
1002.927
970.994
Table Model comparison: linear vs. quadratic fit. A bold number indicates a significant improve- ment of AICquad over AICiinear with p 0.05.
Victor, Jesse, Zachary, Kyle, Aaron, Betty, Connie, Holly, Donna, Gloria, Carla, Isabel, Sylvia, Evelyn, Doris, Arthur, Raymond, Harold, Lawrence, Neil, Brenda, Tracy, Simon, Wendy, Zoe, Ethan, Calvin, Sean, Ruth, Sheila, Miriam, Lorraine, Fay, Sophie
clothes_socks blue, red, yellow, green, purple, pink, orange, black, white, gray
clothes_shin shirt
blue, red, yellow, green, purple, pink, orange, black, white, gray
clothes Sotthes_pant tt blue, red, yellow, green, purple, pink, orange, black, white, gray
clothes hat blue, red, yellow, green, purple, pink, orange, black, white, gray
dlthees_gooee blue, red, yellow, green, purple, pink, orange, black, white, gray
clothes underwear blue, red, yellow, green, purple, pink, orange, black, white, gray
hair blue, red, yellow, green, purple, pink, orange, black, white, gray
recent_eat
pizza pasta burrito sushi taco burger toast egg banana potatoes
recent listen rock, pop, country, electronic, folk, jazz, blues, classical, funk, ska, rap, synth, disco, reaggea
recent. watch drama, comedy, thriller, romance, adventure, horror, sci-fi, action, western, fantasy, documentary, mystery, crime, musical
recent read fiction, mystery, novel, thriller, biography, sci-fi, non-fiction, essay, encyclopedia, dictionary
29

---

location bathroom livingroom kitchen basement toilet balcony garden pool bedroom store university farm office bank tree museum school airport Zoo train bus park butcher library restaurant mall mountain tunnel church river pond harbor taxi gallery bar pizzeria beach gym elevator insurance embassy police hospital festival monument laboratory observatory valley motorway viewpoint synagogue factory castle cave stadium arena cabin plaza amphitheater bridge pier vineyard forest cliff desert creek bay lighthouse orchard resort camp inn motel aquarium bazaar chapel monastery lookout campground retreat dock depot consulate manor theatre cathedral casino lodge mill bakery spa station diner gazebo terrace arcade boardwalk winery hill plateau ridge port oasis market fairground quarry mine grove auditorium cemetery dunes courthouse prison fort granary ranch promenade coliseum field tower pavilion silo bistro labyrinth cafe saloon brewery carnival marina estate safari cottage courtyard waterpark island greenhouse meadow lagoon ford hacienda village marketplace grotto maze golfcourse atrium academy waterfront peninsula cove summit plains
H TOKEN-LENGTH DISTRIBUTIONS BY TASK LENGTH (N) AND DIFFICULTY (D)
We report token-length distributions per model tokenizer across the full CogmiLoad grid in Table 7 T T and Table I0 For each model and each (N, d) condition, we show the mean and the 90th percentile (p90) of both the input (prompt) tokens and the output (reasoning) tokens, compactly formatted as Pmean[Pp90) I Omean[Op90] For example, a cell like P3995[4683] O1 014778[25212] indicates that the mean prompt length is Rs 3995 tokens with p90 Rs 4683, and the mean output length is about s 14778 tokens with p90 ~ 25212 for that condition. Presenting both prompt and output side by side makes it possible to separate the load imposed by the puzzle itself from model and tokenizer-specifie verbosity in decoding.
Across all models, prompt length grows monotonically with N and increases with d, as expected from the construction of the puzzles. At fixed (N, d), the variation in prompt tokens across tokenizers is small (i.e. typically a few percent) which indicates that models received essentially the same input burden for a given condition. For example, at N 250 and d 10 as the highest load condition, prompt means cluster around 16000-17000 tokens, while at d = I they are around 3500-4000 with these patterns recurring for every tokenizer. This confirms that differences in performance at fixed (N, d) do not stem from large differences in presented input length.
On the contrary, output lengths vary substantially across models and show heavy-tailed behavior in several cases, especially at larger N. Gemini 2.5 models tend to produce the longest outputs: at 250 and moderate d, their output p90 values are frequently in the mid-20000s while prompt p90 values are often around 6500-7000, so totals approach or exceed the 32768- token budget. For instance, N N= 250 and d 5, gemini-2.55 2.r shows approximately P9474[1008, and gemini-2.5-flash is in similar range. These totals explain the higher rates of max-context overflows we observed for Gemini at high N indicating a serving limitation rather than genuine reasoning error. By comparison, o3 and the gpt-5 family are more restrained in their output lengths at the same (N, d): their output p90s are typically around 11000-19000 at N = 250 depending on d, leaving more headroom when paired with prompt p90s near 6500-7000. Mid-tier open Qwen3-32B, QwQ-32B, and EXAONE-Deep-525
023221[243211
models such as DeepSeek-R R1- --Distill--lma--7000 generally produce shorter outputs still, for example, Qwen3-32B at N = 250 and = 7 shows roughly P12419[13058) 08034(8 153], a total far below the context limit. Smaller models often have shorter outputs on average but can occasionally exhibit very long tails. Inspecting their transparent reasoning traces we notice those outliers being consistent with the error taxonomy' 's degeneracy categories (e.g.. wrong-max- max-cottex-ttttteteeee or "other"), where models infinitely repeat tokens or drift into infinite reasoning loops rather than concluding. One peculiar case is Gemini-2.55 2..minn-..5-flas----e f-flash-lite at the cell N=250/d=10 resulting in P1 P16683[17372] 0366[9]. Here this particular model doesn't even try to reason through the problem but usually just guesses the response with barely any reasoning.
Taken together, these distributions show that prompt length scales predictably with N and d and is closely matched across tokenizers, while output length is where models differ. In our runs, most models and indicating that CogniLoad puzzles are generally solvable within a 32K context. Overflows trated in specific settings where certain models (notably Gemini 2.5 at high N) produce heavy-tailed outputs while smaller models show occasional outliers that, when inspecting the reasoning traces,
many at N = 250-2e00 rer-reiii well below 32768 tokens in total,
concen-
30

---

boil down to infinite reasoning loops and incoherence issues that repeat the same tokens indefinitely. Consequently, for the majority of models and conditions the observed performance trends at long N reflect reasoning under load rather than system ceilings.
31

---

(1877)
2050
P16801 -4400 [586]
191-1 -0--00555 (
10
0
10
1999
1233
P168411
Omean[Opp (Opl. Pp=Op=p90.
(6330
(433)
338000
02000 10l
111555 0512200
11215111 0670411000
P1207 02993
P12091 03686045000
07
201
P1 P1207[1334]
SOMAIIII EST
(84499944 (2.500
530021
03913 (9975)
(2118]
(223)
Table 7: Prompt and output tokens by model at N=20. Cells show Pmean[Ppl [Ppl
042560
02685
1000
10000000
10100
1101
1010
101100
PA8919922
XX
29180
ps
1
(
PG25/699) 02768
55344
3555]
(444416470)
P61-5 [6855 (2126)2580)
0224313342)
3
-
1
5(685
56685
P6 5(685
P61 (615)
2000 0040
(72M 1
5015
622 747740000
0107- [246]
P3574404 016742311
(750)
1357 (404 0265155U
P357:4044 0033 1 [89] 8591
(297444)06682222 0168222622
100 G4194I
P374 0477
155
P579
625
P37214101
222224411
1572
P5041611
13911111
11111
(355
P357
122000
2355
193
DeepGeek R1 Distill Llame 70B
DeepSeext -81 siist:ll --oen-1. 1.59
peepSeek- R----------- -1l-l1lsl1l---- -------n---- 328 02en-328
SeepSeek R2 Distilll -2wen-73 73
gemini 22 5--5--al- lite
got got-S-miine 5-mini------- 200---222-----00 07
08-07
rrassoning
Phi 4 seasoning glus
gpt- S-nano------ 20----222-------
04 16
noepSeek -0528
EXMONE upep- 328
Phi phi-4-m: 14-----m---- -
Phi 44- seasoning
3wen3- 30B-A38
gemini 2.5 rlash
gemini- -2. 5-pxo
gpt 5-2025-220--20 00-5-00077
03- 08-2020-04- 04-1-44166
04 mini 2025
owen3-1.1 .778
Qwen3-33 3-328
Owen3-88 8B
uo0- 2w0-322 328
Model
32

---

1 9553)
[9611]
- 1e000
DE24
10
6111
Omean OOeal[Opl. (Opl. Pp=Op=po0. -90.
11997
08831
I0 3
16682
SUNE
11
FAAM 19166
DI T
1999 010177
Table 8: Prompt and output tokens by model at N=50. Cells show Pmean[Ppl
10 SOROE
2612
5
191
1000
11921 109)
ONE
111
12531
(19381146))
11
213385
P1
TL
P
1999
1811
04 755TISSS
157744
151
17700883 -1 77442330
03146/681N
P770088-] 02022(4420)
103)
07708844
177101888
P770/8841
Distil Llana 73B
155
11- 528
7B
DeepSeek R1 1
pespssaee 81 Sasttill -wen-
beenSeek R1 sttt111
phi -a----in wonii-rraaonnii ng
phi-4 4-4 resssoning plus
5-Elash lite
gpt- -utt--minn-- 2025-08-07
20--2000----1- 04 16
Deepsiee R1 -1-3332-1
ExaONE- -22---22
gemini 2.5 rlash
gpt 3nt-8-200 -
Qwen3-S32 3--------- -38-33
2omini-2.5-pe 5-pro
20-2020- -16
gemini- 1 2..1--2.5.. 22.5-255.
04 mini 2025
Deepseek
Qwen3-1. 78
Qwena 332
owana-88 888
apt-5-nand
QwQ-328
phi 4
gpt-
Model
33

---

10
10
1327
1157 2032000000
1881
16R77 10000000
Uuzz So
Omean| Pp=Op=po0
(o
11
088 ST
Peeaaa 1333
199 15 WOOOTTI
1991 1008 1
1A91U0 13477)
Table 9: Prompt and output tokens by model at N=100. Cells show Pmean[Ppl
MASSS 0% 11111300
3
1911
19
12
ESS5
P2606 -07 51119199220
(96622
07744
010000
P2606) 2884
$455
03808 5627
6006]
Pl P1446[1683 0271 1 (4066))
1 04153[66166
0220444400
0.999
1O
1999
(909
1960
7881
1117770
Pl [
P1607/11683 1683
P144611681
703
Desppeeee 82 -Diettttl 1ai---ieeen-- 2328--328
-owen---- 5B
ptt---ittll----e 1ol---l-------e --we--1 -78
NeepSeek R1 Dis.111 111 Llama
beepSeek R1 DD wew----eee---1
phi phi- -m ni rraacning
Phi 4 reasening plus
---ffashh-iic lice
2025-08-07
get ------nno--- 2025-08-07
03- 23-2222--4--1 04 16
AeepSaee R3 -05-28
EXMONE -eepp- 32B
Phi-4 reasoning
gemini 2.5 rlash
beepSeek RR
owen3- 308-A38
gemini 2..5--2-------.... -2.5--2---0000
gemini-2 -2.5--225-00 22.5-22.5pp..
apt 2002----2500------
04 mini 2925
0.en3-1.78 78
Owen3 323 323
gpt met-5-mini- 5--mmi--nn-----
0wa-322 328
owan3-8R BB
Model
34

---

[4O1]] Tras0
16EET 1 1L+SHO
TEOSAI LESOOO
10061 CE220
19160
Pa 10000 A8STHE
18801
1
100001110 10 13001 Soppooe
Pazz 400/5 03389
10399
PHO
10- 196691
10
186991
1
FE &
176601
Prigg 10030
5469144
(ppl, P7=0ppp -...
(SCNN I
TRAST 890 1192110000
RESNEE
TEPES 199990 (9002) BRONE SPREE
IMPPL TILAC SOSTEE EESIS SEREON SSO SORTTT
13991 Ia DNO
(
2
/
1099E
(ESti PO1P22
12806 100011100
1019 SESCCE
1018
1snne SM0E11611
1ssne
Omeann
1/
[LAS 11
141191
19551 PORLSSEE
AECIILL 50
(O0RS
10s Mo19999 SARRIAA
Table 10: Prompt and output tokens by model at N=250. Cells show Pmean[Ppl
199991999 SWESO
109010
th
188
180001
164
[1668]
11946
1208 000S00
199 1Or//1
1894
19 SEL0OA SARA
1ir5 Nlow
IT 1239
SEL60 10
550
1011
70B
775
118eE1 ng
glus
Class lite
penanaae -025-08---7
38--00
- in Seeoana - Reassss Put
caminn 2.5 Calashi
00 16
- 323
peeppse eeeeeee R1
o4 mini 2029
owens 3-1.--11100 78
32B
mn mamiii
phi a
Phi 4 3-33
gwa- 328
Sward 888
Model
35


Reviews:
[{"review_id": "gpZCuQ1qrd", "reviewer": "Reviewer_je6v", "summary": "This paper proposes a new benchmark, CogniLoad, that leverages concepts from the Cognitive Load Theory to design a logic grid puzzle reasoning benchmark that systematically evaluates LLMs on their *intrinsic* load (ICL), *extrinsic* load (ECL), and *germane* cognitive load (CGL), with the last one being approximated via task length $N$. The authors evaluate 22 SOTA LLMs and present several trends and insights on the scaling of each of these dimensions of reasoning ability, for different classes of LLMs.", "questions": "* Any thoughts on the $\\rho$ parameter not quite being as clean as anticipated, as increasing it makes the problem both easier and harder in different ways? (see note above)", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* CogniLoad is a systematic benchmark and the first to make proper distinction between the 3 studied kinds of cognitive load LLMs have to deal with when solving hard reasoning problems.\n\n* The work is grounded in a formalism from cognitive science, namely Cognitive Load Theory. Although some aspects (in particular, the connection to germane load) is a bit of a stretch, the notions of intrinsic and extrinsic load align very well with the notions of actual task and distracting information.\n\n* The study is quite detailed, with many LLMs of various kinds evaluated on the proposed benchmark. The authors discuss several trends. While some of these (e.g., increasing the intrinsic difficulty reduces LLM accuracy) are both intuitive and, to some extent, reported before, the present paper does it carefully while keep all (or most) other variables constant. This, thus, provides a cleaner evidence. Additional analyses, like failure modes and differential sensitivity, are useful.\n\n* The paper was easy to read and understand.", "weaknesses": "* The paper can use a clearer description of the *germane* load from the CLT theory -- what it means, how it manifests, etc. -- along with some more discussion of why the task length parameter, $N$, is a good proxy for it.\n\n* (stronger concern) The needle-to-hay parameter, $\\rho$, is not as clean cut as intended, and this might even explain the U-shaped performance the authors observe. Specifically, the $\\rho$ parameter mixes two distinct aspects: higher $\\rho$ means more statements about the PoI (person of interest), which has two competing effects: (a) it means fewer distractor statements, which makes the problem easier to solve; and (b) it means a higher number of state changes for PoI, which makes the problem harder to solve. As an example, as one moves from $\\rho = 5\\%$ to $\\rho = 10\\%$, the number of distractor statements drop by (only) $5\\%$, but on the other hand, the number of state transitions for PoI go up by 2x, i.e., double!! The authors don't discuss this important aspect and its implications. Note that this provides a reasonable explanation of the U shaped curve observed -- at very low values of $\\rho$, the number of state transitions dominate the accuracy (the relative change in distractors isn't much), while at very high values of $\\rho$, the relative change in the number of distractors is the one that's significant. I would ask the authors to look more deeply into this and see how it affects the message of and explanations in the paper.\n\n* While it's good to have a formal puzzle definition in section 2, it is also important to verbally convey what each of the constraints is achieving. Can you add some explanatory text?\n\n* For task length $N$, and especially at relatively higher $\\rho$ values (which together means many state transitions for PoI), there is good reason to expect models to struggle without chain-of-thought reasoning, i.e., \"non-reasoning models\", based on theoretical results showing that transformers (without CoT) are inherently parallel (in complexity class $TC^0$ [1,2]) while state-tracking is a sequential task (in general well-known to be complete for the stronger complexity class $NC^1$). It's also known formally that using CoT allows models to perform state tracking and other sequential tasks [3,4]. This might explain some of the observed behavior, e.g., GPT-5 and o3 being resiliant to increase in $N$. Including some discussion of this connection would be valuable.\n\n[1] The Parallelism Tradeoff: Limitations of Log-Precision Transformers, Merrill et al., 2022\n\n[2] Transformers in Uniform TC0, Chiang, 2024\n\n[3] The Expressive Power of Transformers with Chain of Thought, Merrill et al., 2024\n\n[4] Chain of Thought Empowers Transformers to Solve Inherently Serial Problems, Li et al., 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4REYE7cOEm", "reviewer": "Reviewer_xtvk", "summary": "The paper introduces CogniLoad, a synthetic long‑context reasoning benchmark grounded in Cognitive Load Theory (CLT). It factorizes task stressors into three independently tunable dimensions: (1) intrinsic difficulty d (element interactivity and rule complexity), (2) distractor density ρ (needle‑to‑hay ratio), and (3) task length N (as an operational proxy for sustained “germane‑like” processing). The benchmark generates natural‑language logic‑grid puzzles with sequential state updates, evaluates exact‑match accuracy, and provides an error taxonomy. They have run factorial evaluations for 22 LLMs and fit a per‑model binomial GLM to derive interpretable “capacity thresholds” (ECL50, NT50, ID50). The main empirical findings are that length N is the dominant constraint, that distractors exhibit a characteristic U‑shaped performance curve with worst accuracy at intermediate ρ, and that models differ meaningfully in their sensitivity profiles.", "questions": "1. Please report, in the main paper, at least a compact interaction analysis among d, N, and ρ (one table with key coefficients or ANOVA‑style significance and effect sizes). The appendix suggests clear qualitative patterns; quantifying them would greatly help interpretation.\n2. How sensitive are the main conclusions to the instruction format? If the final answer is allowed to be free‑form (no strict one‑sentence requirement), do “last‑logic” errors diminish while overall accuracy remains similar?\n3. What is the average token length of puzzles at each N and d? Reporting token distributions per condition and per model tokenizer would help separate system limits from reasoning load.\n4. Can you provide size‑matched comparisons with harmonized decoding settings (temperature, top‑p, max tokens) across representative models, to reduce variance induced by vendor defaults?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The design isolates three often‑confounded aspects of long‑context reasoning and does so in a way that scales to arbitrarily long sequences. The generator is cleanly specified with validation constraints and produces semantically coherent, non‑trivial distractors. The regression view and derived thresholds offer compact, model‑specific “cognitive fingerprints” that go beyond aggregate accuracy; this is a useful diagnostic interface for both research and engineering. The error taxonomy and supplementary analyses (e.g., AIC comparison for quadratic ρ, confidence intervals) increase interpretability and make the empirical claims credible. The reproducibility is strong: the code repository and dataset release enable immediate verification.", "weaknesses": "The CLT analogy is motivating but only partially validated. Using sequence length N as a proxy for germane load conflates at least three mechanisms in LLMs - attention budget, recurrent state tracking over updates, and compounding interference from earlier updates. The paper acknowledges this, but the empirical analysis does not probe which of these mechanisms dominates or whether N’s effect interacts mechanistically with d and ρ in predictable ways.\n\nThe evaluation mixes performance and infrastructure limits. The max‑context failures are treated as an error category, but 32K token limits, tokenization differences, prompt overhead, and vendor defaults create avoidable confounds between “reasoning under load” and “serving limits.” Several models show many “long‑context budget overflow” errors at N=250; this is partly a systems artifact rather than a cognitive limitation. The paper would be stronger if it normalized prompt and decoding budgets across models, used a shared short instruction template, and pre‑tested that all N regimes fit for at least a subset of items per model.\n\nEvaluation relies on an exact match with a final one‑sentence answer. The authors mitigate formatting drift via post‑processing and categorize “last‑logic” failures, but the metric remains coarse. Without step‑wise verification, it is hard to separate state‑tracking mistakes from late‑stage overwrites or local misparses. Given that the generator has full ground‑truth state trajectories, a step‑fidelity metric or a probabilistic grading of intermediate updates would add substantial value and make the “germane‑like” interpretation more concrete.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "744ZzQFDnp", "reviewer": "Reviewer_kvXx", "summary": "CogniLoad introduces a synthetic benchmark grounded in Cognitive Load Theory (CLT) to evaluate LLMs' long-context reasoning capabilities. This benchmark independently controls three dimensions: intrinsic difficulty, task length, and distractor density.\n\nKey contributions include: (1) operationalizing CLT for LLM evaluation, (2) revealing task length as the dominant performance constraint, (3) identifying U-shaped sensitivity to distractors, and (4) providing capacity thresholds (e.g., ECL50, ID50) for 22 models.", "questions": "1. The factorial design isolates each dimension, but are there ​interaction effects​ (e.g., high d+ high ρexacerbating errors)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Originality​**\n\nIntroduces the first benchmark grounded in Cognitive Load Theory (CLT) for LLMs, disentangling intrinsic difficulty (d), distractor density (ρ), and task length (N) to diagnose reasoning failures.\n\n​**Quality​**\n\nRigorous synthetic puzzle generation with independent parameter control, validated through large-scale evaluation and GLM-based capacity thresholds (ECL50, ID50).\n\n​**Clarity​**\n\nWell-structured with explicit CLT-LLM mapping. The operationalization of task length (N) as a proxy for germane load (GCL) could be further justified; visualizations (e.g., U-shaped ρ sensitivity) enhance interpretability.\n\n​**Significance​**\n\nSets a new standard for interpretable LLM evaluation, revealing task length (N) as the dominant bottleneck and providing actionable \"cognitive fingerprints\" for model improvement.", "weaknesses": "**Limited Generalizability Beyond Deductive Reasoning**\n\nThe benchmark exclusively tests ​deductive logic puzzles, neglecting inductive/abductive reasoning or real-world tasks (e.g., math, code debugging). This restricts insights into broader reasoning capabilities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OCGJftslbj", "reviewer": "Reviewer_aNzL", "summary": "The paper introduces a benchmark for long-context reasoning problems based on Cognitive Load Theory, which categorizes cognitive demands into three categories: intrinsic, extrinsic, and germane. Using this framework, they create puzzles in deductive reasoning and state tracking. They describe a generation procedure to change the difficulty of the benchmark by changing each of these dimensions separately. They generate 14,000 (4x5x7x100) puzzles to evaluate 22 LLMs across the dimensions. They fit a GLM and show the performance of the LLMs with respect to variations in each of the three dimensions, with task length still dominating.", "questions": "1. What is the distribution of lengths of the puzzles in tokens? What fraction of the context window is consumed by the inputs? This task seems to require tracking a considerable amount of information. How much context would actually be required to just write out the states after each step?\n\n2. What do the findings of the error analysis mean for people trying to improve LLMs? Would training models via RL to perform state tracking tasks using the context window be a valuable thing to do?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "**Originality:** This paper offers a new lens for dividing up the factors which contribute to LLM performance using cognitive loading.\n\n**Quality:** The results in the paper are sound and thorough. I am particularly pleased that the authors used statistical methods to perform their analysis of benchmark results, and have a large sample size. \n\n**Clarity:** The description of the approach is described in detailed mathematical notation, making it highly precise. \n\n**Significance:** Meaningful dimensions for the analysis of LLM results are important to guide research, and I find comparisons to cognitive science to be particularly interesting, as they draw on well-established concepts (though of course with the caveat that LLMs do not have human minds...). The algorithm for procedurally-generated benchmark items is also useful.", "weaknesses": "I find the paper to be well-executed, and have minimal concerns about the quality of the results. The concerns that I do have come from unclear or missing details in the paper, grounding in cognitive science, and from uncertainty about the broader impact of the work. \n\n**Soundness:** As a sense check, I do want to know that the task is possible within the context lengths provided. As with another recent paper (https://arxiv.org/abs/2506.06941) models running over the context window seems like a poor justification for calling a performance a \"reasoning\" failure.\n\n**Clarity:** \n\n1. The mathematical notation of the paper was comprehensive, but not always helpful. I think some pieces were missing, such as the use of k_t and m_t after their initial definition on line 199-200. I would also suggest distinguishing between \"=\" as a logical operator and \"=\" as a definition (probably by using \":=\"). v_{c,t} and u_{c,t} were a bit confusing along this lines, when moving between the definitions and their use validating statements.\n\n2. I am not certain how the 14000 problems were actually finally generated. Was this done with an LLM? Did you create formal logical sets and then translate them into plain language? The latter certainly seems more reliable.\n\n(I will note that although there were references to supplementary material, I only found a zip file with code and data, not further text. In any case, I do think that these details should be in the main body of the paper.)\n\n**Cognitive Science:** I am not a cognitive scientist, but I felt that the grounding in this theory was relatively light. I would appreciate more explanation/justification about why d, N, \\rho are the right way to operationalize these dimensions, beyond what appears in Table 1. In particular, my intuition is that d does not capture \"difficulty\" in the way I would usually conceive of it, since higher values require more working memory, but not more complex deductions. Similarly, I would imagine that instruction following is a subset of ECL, but seems to be described as outside of the three dimensions in the error analysis. One way to make this argument would be by comparison to a human exam that varies along the same cognitive dimensions, to show an analogy between how those variations are made and the variations made in this puzzle.\n\n**Significance:** I am open to being wrong here, but I worry that this paper does not show us things we didn't already know. State tracking is a known weakness of LLMs (https://arxiv.org/abs/2404.08819, https://proceedings.neurips.cc/paper_files/paper/2024/hash/e6231c5f46598cfd09ff1970524e0436-Abstract-Conference.html). It is also a relatively simple task to be testing relative to the more complex reasoning benchmarks described in related works. The variation of performance along context length (most previous works), task size (https://arxiv.org/abs/2506.06941) and number of needles (https://arxiv.org/pdf/2404.06654) have also been shown previously. Being able to track these three dimensions separately is probably new, but I would like to see a clearer discussion of why this matters. \n\nI also find the error analysis to be somewhat unhelpful. Finding that the models often run out of context is described as as failure to use tokens efficiently, but efficient token use is not something that the benchmark describes itself as measuring. Why shouldn't the task be adjusted to avoid these issues in order to ensure a pure assessment of the target reasoning abilities?\n\nIncidentally, I think addressing this can be tied to the previous comment about grounding in cognitive science. To the extent that the dimensions are well-justified, papers like (https://arxiv.org/html/2503.06378v2) have shown useful predictive results from cognitive science dimensions. \n\n**Overall**: I have netted out to a weak accept because I believe the work is methodologically sound, and want to be charitable about the significance, but I am very open to having my opinion swayed, particularly in how to weight the different elements of the papers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)