CogniLoad: A Synthetic Natural Language Reasoning Benchmark With Tunable Length, Intrinsic Difficulty, and Distractor Density

## Abstract

Abstract Current benchmarks for long-context reasoning in Large Language Models (LLMs) often blur critical factors like intrinsic task complexity, distractor interference, and task length. To enable more precise failure analysis, we introduce CogniLoad , a novel synthetic benchmark grounded in Cognitive Load Theory (CLT). CogniLoad generates natural-language logic puzzles with independently tunable parameters that reflect CLT’s core dimensions: intrinsic difficulty ( d d ) controls intrinsic load; distractor-to-signal ratio ( ρ \rho ) regulates extraneous load; and task length ( N N ) serves as an operational proxy for conditions demanding germane load. Evaluating 22 SotA reasoning LLMs, CogniLoad reveals distinct performance sensitivities, identifying task length as a dominant constraint and uncovering varied tolerances to intrinsic complexity and U-shaped responses to distractor ratios. By offering systematic, factorial control over these cognitive load dimensions, CogniLoad provides a reproducible, scalable, and diagnostically rich tool for dissecting LLM reasoning limitations and guiding future model development.

## 1 Introduction

Cognitive Load Theory (CLT) (Sweller, 1988) characterizes three types of cognitive load on human working memory when solving problems (Sweller, 1988; Paas et al., 2003; Lieder and Griffiths, 2020): intrinsic (ICL), extraneous (ECL), and germane (GCL). ICL stems from the inherent complexity and element interactivity of the task (Halford et al., 1998). ECL is induced by suboptimal task presentation requiring the processing of elements that are not task-relevant (Chandler and Sweller, 1991). GCL concerns effective remaining resources allocated to engaging with the intrinsic task demands for mental schema construction (Ericsson and Kintsch, 1995; Sweller, 2010).

Large language models (LLMs) demand analogous computational resources when solving reasoning tasks. The essential element interactivity of a reasoning chain mirrors ICL; distractor elements reflect ECL; and sustained engagement with intrinsically relevant information over a long reasoning process acts as an operational proxy for germane-like processing - the constructive effort to maintain a coherent problem representation.

To the best of our knowledge, no study has based the evaluation of problem-solving capacities of LLMs in CLT by distinguishing these three load types, and existing benchmarks often confound them: LongBench (Bai et al., 2024a) and L-Eval (An et al., 2024) vary context length but not necessarily the intrinsic reasoning depth; LogicBench (Parmar et al., 2024) probes ICL with minimal demands on ECL or context-induced load; BABILong (Kuratov et al., 2024) mixes multi-step reasoning with fixed distractor ratios, obscuring precise failure attribution.

We introduce CogniLoad, a controllable synthetic benchmark for long-context reasoning, inspired by CLT, that operationalizes these load types through tunable parameters in randomized natural-language logic puzzles:
(i) Intrinsic Load via Intrinsic Difficulty $d$ controls the number of interacting entities, attributes, and logical clauses, directly manipulating ICL by varying essential element interactivity and reasoning depth.
(ii) Extraneous Load via Distractor Density $\rho$l dictates distractor density; lower $\rho$ increases irrelevant elements, manipulating ECL.
(iii) Germane Load Proxy via Task Length $N$ serves as an operational proxy for demanding germane-like cognitive work.

Our key contributions are summarized as follows:

- 1.
We ground the evaluation of LLMs in CLT, precisely defining benchmark parameters that control ICL, ECL, and an operational proxy for the conditions conducive to GCL.
- 2.
We introduce *CogniLoad*, the first benchmark designed to independently control these three dimensions of cognitive load, while scaling to arbitrarily long contexts.
- 3.
We provide an algorithm for the automatic randomized generation and evaluation of puzzle instances, enabling large-scale and reproducible comparison of LLM capabilities.
- 4.
We report empirical results on 22 state-of-the-art (SotA) reasoning LLMs (see [Figure˜1](https://arxiv.org/html/2509.18458v2#S1.F1)), revealing distinct failure regimes across the $(d,N,\rho)$ dimensions and highlighting specific targets for improving LLM design.

Together, these contributions translate CLT into a precise diagnostic framework for understanding and advancing long-context reasoning in LLMs.

Figure: Figure 1: The average accuracy of models across the evaluated parameter space for $d\in\{1,3,5,7,10\}$ (left panel), $N\in\{20,50,100,250\}$ (center panel), and $\rho\in\{5,...,95\}$ (right panel). Each plot selects one dimension for the X-axis and averages the accuracy of all evaluated puzzles for the other two dimensions relative to it.
Refer to caption: x1.png

### 1.1 Related work

Long-context Benchmarks (Working Memory Capacity).
A line of work starting with Long-Range Arena (LRA) (Tay et al., 2020) and followed by several recent benchmarks probe LLM performance on long sequences, often framed as testing “memory load” or context utilization. Earlier studies such as SCROLLS (Shaham et al., 2022), BookSum (Kryściński et al., 2021), and QMSum (Zhong et al., 2021) scale document length without manipulating intrinsic difficulty. LongBench (Bai et al., 2024a, b) and L-Eval (An et al., 2024) aggregate multi-task corpora up to 200k tokens, while BABILong (Kuratov et al., 2024), LongReason (Ling et al., 2025), RULER (Hsieh et al., 2024), ZeroSCROLLS (Shaham et al., 2023), and Michelangelo (Vodrahalli et al., 2024) increase context while the inherent difficulty of individual sub-tasks (ICL) may vary unsystematically and distractor density (ECL) is often not a controlled variable. Consequently, performance degradation could be due to sheer length overwhelming processing capacity, or an inability to sustain germane-like cognitive work over extended relevant information, but the precise cause of failure is not clear.

Logical-reasoning Benchmarks (Intrinsic Load).
A complementary line of benchmarks focuses on ICL by presenting tasks with high inherent complexity but often within minimal context lengths or distractors. Notable classical suites include ReClor (Yu et al., 2020), LogiQA (Liu et al., 2020), and BIG-Bench-Hard (BBH) (Suzgun et al., 2022). AutoLogic (Zhu et al., 2025) is a benchmark that explicitly focuses on scaling ICL through controllable complexity. LogicBench (Parmar et al., 2024), CLUTRR (Sinha et al., 2019), and ZebraLogic (Lin et al., 2025) also exemplify this by formulating symbolic logic puzzles that demand processing many interacting elements (e.g., multi-step deductions, handling negation, and constraint satisfaction). Similarly, mathematical reasoning datasets, e.g., GSM8K (Cobbe et al., 2021) and abstract rule induction tasks, e.g., ARC-AGI (Chollet et al., 2024) primarily escalate ICL by increasing the complexity of essential rules and their interdependencies.

Needle in a Haystack Benchmarks (Extraneous Load).
Needle in a haystack (NIAH) designs (Gkamradt, 2023) specifically target ECL by embedding relevant facts (“needles”) within large volumes of distractor text (“hay”). Variants such as Sequential NIAH (Yu et al., 2025) and Nolima (Modarressi et al., 2025) investigate the impact of such distractors, which constitute non-essential elements requiring processing for filtering, thereby imposing ECL. While these benchmarks effectively isolate the impact of distractors on information retrieval, the “needle” tasks themselves typically involve low ICL (e.g., simple fact lookup).

Need for Multi-dimensional Evaluation.
CLT highlights the interplay of ICL, ECL, and germane processing under finite working memory (Paas et al., 2003). Existing LLM reasoning benchmarks, however, typically manipulate only one dimension without systematic, independent control over the others. Even benchmarks like MIR-Bench (Yan et al., 2025) which combine high ICL with extensive input, do not offer the factorial control needed to disentangle these loads, hindering precise diagnostics. Similar to our work, GSM-$\infty$ (Zhou et al., 2025) allows manipulating noise and difficulty. However, these parameters are not adjusted independently of task length.

Contribution of CogniLoad. CogniLoad addresses this critical gap by providing a framework for independently controlling parameters that influence: (i) ICL via intrinsic puzzle difficulty ($d$), (ii) ECL via distractor density ($\rho$), and (iii) the demands for sustained, germane-like processing via task length ($N$), all within a single synthetically generated natural language puzzle. This factorial design enables a precise diagnosis of LLM failure modes, specifically the inability to handle increased intrinsic complexity, susceptibility to extraneous distractors, or incapacity to maintain coherent reasoning over an extended number of sequences. By explicitly grounding these dimensions in CLT, CogniLoad offers the first benchmark to diagnostically map LLM capability surfaces across these distinct cognitive demands, thereby complementing and extending the insights from evaluations that focus on a single factor.

## 2 Benchmark Design: CogniLoad Logic Puzzles

### 2.1 Puzzle Definition and Construction

CogniLoad is a family of natural-language logic-grid puzzles explicitly crafted to probe sequential reasoning capabilities of LLMs. The design goals are threefold: each puzzle (i) necessitates sequential multi-step deduction where order fundamentally matters; (ii) embeds a controllable number of relevant “needle” facts within the context of a controllable number of “hay” distractor statements; and (iii) provides parameters that control distinct dimensions of cognitive load. This section formalizes the task and describes the puzzle generation process, the control parameters, and key design choices.

Each puzzle in CogniLoad (see  [Figure˜2](https://arxiv.org/html/2509.18458v2#S2.F2)) consists of a set of people with independent and mutable attributes. A series of statements, applied in strictly sequential order, updates these attributes according to conditions specified in each statement. The puzzle generation is parameterized by three key parameters: intrinsic difficulty $d$, total number of statements $N$, and needle-to-hay ratio $\rho$.

Figure: Figure 2: Example CogniLoad puzzle with intrinsic difficulty $d=3$, statements $N=20$, and needle-to-hay ratio $\rho=50\%$. Only a subset of the initial state and update statements is shown.

#### 2.1.1 Basic Puzzle Construction

A puzzle is formally characterized by the following components:

- •
People: A set $P=\{p_{1},p_{2},\ldots,p_{n}\}$ of persons in the puzzle, and $n=\max(d,2)$.
- •
Person of Interest (PoI): A randomly selected person $p^{*}\in P$ about whom the final question is asked.
- •
Attribute Categories: A set $A=\{c_{1},c_{2},\ldots,c_{d}\}$ of attributes randomly selected from a predefined taxonomy of 12 categories. Each category takes values in a Value Domain with a given finite cardinality, smaller or equal to 10.
- •
Value Domains: For each category $c\in A$, a value domain $V_{c}=\{v_{c,1},v_{c,2},\ldots,v_{c,\ell_{c}}\}$ where $\ell_{c}=d+1$ for $d>1$ or $\ell_{c}=3$ when $d=1$. See Table [1](https://arxiv.org/html/2509.18458v2#S2.T1) for the ontology.
- •
State Function: $S_{t}(p,c)$ represents the value of attribute $c$ for person $p$ at step $t$. Each person has values for the $d$ attribute of the selected attribute categories $A$, thus the state value represents a vector of dimension $d$.

**Table 1: Overview of the attribute ontology. The full ontology contains 12 categories of varying domain sizes and is detailed completely in LABEL:supp-app:ontology.**
| Category Name (Code) | Domain Size | Examples of Values |
| --- | --- | --- |
| location | 50+ | kitchen, balcony, zoo, museum, park… |
| clothes_socks | 10 | blue, red, yellow, green, purple… |
| clothes_gloves | 10 | (same as clothes_socks) |
| hair | 10 | (same as clothes_socks) |
| recent_listen | 13 | rock, jazz, disco, classical, funk… |
| recent_eat | 10 | pizza, pasta, burrito, sushi, taco… |
| … | … | … |

#### 2.1.2 Puzzle Initialization

A puzzle starts with initialization statements ($t=0$) that assign unique attribute values to each person: $\forall p\in P,\forall c\in A:S_{0}(p,c)\in V_{c}$ such that $\forall p_{i},p_{j}\in P,i\neq j$, $\exists c\in A:S_{0}(p_{i},c)\neq S_{0}(p_{j},c)$.

#### 2.1.3 Statement Generation Process

For each step $t$ from 1 to $N$, a statement is generated that changes the state of a person. If it updates the PoI, the statement is called a *needle*. An update for a non-PoI is called a *hay*.

1. Statement Type Selection: Given $N$ and $\rho$, let $n_{\text{needle}}^{t}$ and $n_{\text{hay}}^{t}$ be the remaining numbers of needles and hays to satisfy the desired proportion $\rho$ in the complete puzzle. The probability of selecting a needle statement is then $\mathbb{P}(T_{t}=\text{needle})={n_{\text{needle}}^{t}}/{(N-t)}$. The total number of needle statements in the puzzle is calculated as $n_{\text{needle}}^{0}=\max(1,\min(N,\text{round}(N\cdot\rho/100)))$.

2. Reference Person Selection: Given the selected statement type $T_{t}$, the algorithm selects the reference person $r_{t}$: if $T_{t}=\text{needle}\implies r_{t}=p^{*}$ and if $T_{t}=\text{hay}\implies r_{t}\sim\text{Uniform}(P\setminus\{p^{*}\})$.

3. Statement Structure: For each statement, CogniLoad samples a number of conditions $k_{t}\sim\text{Uniform}\{1,\ldots,d\}$, and a number of state updates $m_{t}\sim\text{Uniform}\{1,\ldots,d\}$ and uniformly sample attribute categories $C_{t}\subseteq A,\;|C_{t}|=k_{t}$ and state updates $U_{t}\subseteq A,\;|U_{t}|=m_{t}$.

4. Condition and Update Value Specification: For each category $c\in C_{t}$, the condition value is set by the reference person’s current state: $v_{c,t}=S_{t-1}(r_{t},c)$. For needles, these conditions target the PoI, while for hays the conditions can match multiple people. For update values, if $T_{t}=\text{needle}\implies u_{c,t}\sim\text{Uniform}(V_{c})$ and if $T_{t}=\text{hay}\implies u_{c,t}\sim\text{Uniform}(V_{c}\setminus\{S_{t-1}(p^{*},c)\})$.

5. Logical Form: The statement at step $t$ has the logical form:

$$ $\displaystyle\forall p\in P:\Big(\bigwedge_{c\in C_{t}}S_{t-1}(p,c)=v_{c,t}\Big)\Rightarrow\Big(\bigwedge_{c\in U_{t}}S_{t}(p,c)=u_{c,t}\Big).$ $$

Attributes not mentioned in the update set remain unchanged $\forall p\in P,\forall c\in A\setminus U_{t}:S_{t}(p,c)=S_{t-1}(p,c)$. This is not specified in the prompt but implicitly assumed by the LLMs.

#### 2.1.4 Validation Constraints

A sequence of validations verifies that the generated statement does not result in a state that prevents the generation of further needles and hays. If all validations pass, the statement is appended to the puzzle; otherwise a new statement is generated.

For hay statements ($r_{t}\neq p^{*}$): After the update, the state of affected non-PoIs must not become identical to PoI $\forall p\in P\setminus\{p^{*}\}$ such that $\forall c\in C_{t}:S_{t-1}(p,c)=v_{c,t}$, $\exists c\in A:S_{t}(p,c)\neq S_{t}(p^{*},c)$ and the update must not affect the PoI $\exists c\in C_{t}:S_{t-1}(p^{*},c)\neq v_{c,t}$.

For needle statements ($r_{t}=p^{*}$): The update must not affect all non-PoI people $\exists p\in P\setminus\{p^{*}\}:\exists c\in C_{t}:S_{t-1}(p,c)\neq v_{c,t}$ and after the update not all non-PoIs have identical states as the PoI $\exists p\in P\setminus\{p^{*}\}:\exists c\in A:S_{t}(p,c)\neq S_{t}(p^{*},c)$.

To prevent the distractors from becoming too trivial to track at lower difficulties, we require that a hay statement does not result in all non-PoIs become identical so the set $P\setminus\{p^{*}\}$ must contain at least two persons with distinct attribute values. CogniLoad construction ensures that each hay statement $T_{t}=\text{hay}$ affects at least one non-PoI $\exists p\in P\setminus\{p^{*}\}:\forall c\in C_{t}:S_{t-1}(p,c)=v_{c,t}$.

#### 2.1.5 Final Question Generation

After all $N$ statements have been generated, the puzzle concludes with a question about a random attribute of the PoI, sampled as a random category $c_{q}\sim\text{Uniform}(A)$. The correct answer to the puzzle is $S_{N}(p^{*},c_{q})$ obtained from the final state of the PoI.

#### 2.1.6 Evaluation metrics

We evaluate each puzzle by exact-matching of the queried attribute value in the output of model $M$ with accommodating minor phrasing and common lexical variants. See Appendix LABEL:supp-app:eval for the specifics of the evaluation pipeline and an overview of the granular failure types that contextualize model specific error modes. The accuracy of model $M$ across the evaluation set is calculated as
$\text{acc}(M)=\frac{1}{|Z|}\sum_{z\in Z}\mathbf{1}\bigl[\text{answer}_{M}(z)=S_{N}(p^{*},c_{q})\bigr]$ where $S_{N}(p^{*},c_{q})$ represents the final state value of the queried attribute $c_{q}$ for the PoI $p^{*}$ after all $N$ statements have been processed.

### 2.2 Tunable Parameters

To systematically probe long-context reasoning, CogniLoad employs three independent parameters. These parameters are designed to operationalize distinct cognitive load dimensions as defined by CLT (Paas et al., 2003), allowing the creation of puzzles with varying characteristics. Together, they define the load profile of a puzzle instance.

##### Intrinsic Difficulty ( d ) (d)

for $d\in\{1,3,5,7,10\}$ controls multiple facets of puzzle complexity (see Table [2](https://arxiv.org/html/2509.18458v2#S2.T2)), directly manipulating ICL which according to CLT hinges on element interactivity (Halford et al., 1998). Increasing $d$ increases ICL via: (i) combinatorial growth in state space ($\approx(d+1)^{d}$), (ii) increased interactivity between persons, attributes, and values, and (iii) increased rule complexity (up to $d$ conditions/updates per statement).

##### Task Length ( N ) (N)

for $N\in\{20,50,100,250\}$ sets the total number of sequential state-update statements. While directly determining sequence length, $N$ serves as an operational proxy for conditions demanding GCL. Increasing $N$, particularly with a large $d$ and $\rho$, compels deeper reasoning through more essential interacting elements (Sweller, 2010). Additionally, increasing $N$ necessitates the maintenance of a coherent (stateful) problem representation over a longer term with the construction of an efficient schema for it (Ericsson and Kintsch, 1995).

**Table 2: Key parameters controlling the puzzle generation.**
| Symbol | Name | Definition | Cognitive Load Affected |
| --- | --- | --- | --- |
| $d$ | Intrinsic Difficulty | Controls cardinality of people set $|P|=\max(d,2)$, attribute categories $|A|=d$, for each category $c\in A$ the cardinality of value domains $|V_{c}|=\max(d+1,3)$, and the distribution of conditions and updates per statement: $k,m\sim\text{Uniform}\{1,...,d\}$. | ICL: Element interactivity, state space/rule complexity. |
| $N$ | Task Length | Total number of sequential state transitions in the puzzle. | GCL Proxy / Task Length: Demands sustained engagement with core elements. |
| $\rho$ | Needle-to-hay Ratio | Percentage of statements directly influencing the PoI (needles) versus distractor statements (hay) | ECL: Distractor density challenges filtering, selective attention, and imposing load from processing non-essential elements. |

##### Needle-to-hay Ratio ( ρ ) (\rho)

for $\rho\in\{5,...,95\}$ sets the percentage of PoI-relevant (“needle”) versus distractor (“hay”) statements, directly manipulating ECL. ECL arises from processing non-essential elements (Chandler and Sweller, 1991). Decreasing $\rho$ increases ECL via increased distractor density which challenges filtering. Increasing $\rho$ controls ECL by focusing resources on relevant information. Critically, CogniLoad’s “hay” statements are syntactically similar to “needles” and involve valid state updates for non-PoIs, imposing a more challenging ECL than easy to distinguish distractor text.

## 3 Results

We have evaluated the performance of 13 open weights LLMs on 100 random CogniLoad puzzles per $(d,N,\rho)$ configuration resulting in 14’000 puzzle instances per LLM in total. In addition, we have evaluated the proprietary Gemini-2.5 and gpt-5(^1^11The gpt-5 family models were evaluated using the “medium” reasoning effort setting.) models and DeepSeek-R1-0528 on 10 CogniLoad puzzles per configuration (i.e., 1’400 puzzle instances). The maximum context length (input + output) was set to 32K tokens and LLMs run with their preset default decoding settings and system prompts.

[Figure˜1](https://arxiv.org/html/2509.18458v2#S1.F1) shows mean accuracy across models as each load dimension varies with trends corroborated by our regression analysis ([Section˜3.1](https://arxiv.org/html/2509.18458v2#S3.SS1)).

Intrinsic Difficulty ($d$) Performance declines monotonically with $d$ for most models, although a few mid-tier models show small bumps at $d=3$ (e.g., QwQ-32B: 0.62$\rightarrow$0.69; DS-Qwen-32B: 0.63$\rightarrow$0.66). Top models degrade only slightly from $d=1$ to $d=3$ (o3: 0.96$\rightarrow$0.93; gpt-5: 1.00$\rightarrow$0.97), while smaller or distilled models drop by 0.10-0.25 in the same range. By $d=5$, 12 of 22 models fall below 50% accuracy. Beyond $d\geq 7$ the marginal decline flattens for the majority of models: the strongest models maintain their performance even at $d=10$ (gpt-5: 0.82; o3: 0.80), and the weakest ones approach 0.10-0.15.

Task Length ($N$) This parameter remains the dominant stressor. Most models exhibit their steepest decline between $N=20$ and $N=50$ (e.g., DS-Llama-70B: 0.89$\rightarrow$0.66; Qwen3-8B: 0.71$\rightarrow$0.40), while the best performing ones show relative resilience (gpt-5: 1.00$\rightarrow$0.98; o3: 0.99$\rightarrow$0.98). Accuracy declines with longer sequences as at $N=100$, several models roughly halve their $N=20$ accuracy (e.g., DS-Llama-70B: 0.48; Qwen3-32B: 0.38) and at $N=250$ only two models show above 50% accuracy (i.e., gpt-5 at 0.76 and o3 at 0.68) while the majority ones perform 0.20-0.30 accuracy.

Extraneous Load / Needle-to-hay Ratio ($\rho$) A characteristic U-shaped response is typical with performance usually reaches as low as $\rho\in[25,50]$% and recovers as $\rho$ increases. Recovered performance equals or exceeds that of small-$\rho$ baseline in several cases (DS-Llama-70B: 0.61$\rightarrow$0.64; gpt-5-mini: 0.84$\rightarrow$0.86; gemini-2.5-flash-lite: 0.38$\rightarrow$0.53). The strongest models show smooth variations (gpt-5: 0.97$\rightarrow$0.89$\rightarrow$0.91; o3: 0.93$\rightarrow$0.89$\rightarrow$0.88), indicating marginal sensitivity to distraction, while some models recover only partially or not at all (Phi-4-reasoning-plus: 0.59$\rightarrow$0.45; EXAONE-Deep-32B: 0.47$\rightarrow$0.32).

### 3.1 Load-sensitivity Regression

To quantify model-specific sensitivities of the accuracy to load dimensions and derive interpretable capacity thresholds for each model, we employ a regression-based approach that allows us to isolate the impact of each type of cognitive load (see [Table˜3](https://arxiv.org/html/2509.18458v2#S3.T3)).

**Table 3: Per-model quadratic‐$\rho$ GLM estimates with Wald $z$ statistic for p-values alongside derived 50% load‐capacity thresholds (see [Section˜3.1.3](https://arxiv.org/html/2509.18458v2#S3.SS1.SSS3)). The value $--$ for NT50 indicates that no real root exists in $[0,1]$. “DS” abbreviates “DeepSeek” in the model names. ^∗∗∗$p{<}0.001$, ^∗∗$p{<}0.01$, ^∗$p{<}0.05$**
| Model | $\beta_{0}$ | $\beta_{d}$ | $\beta_{N}$ | $\beta_{\rho}$ | $\beta_{\rho^{2}}$ | ECL50 | NT50 | ID50 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| gemini-2.5-pro | $22.51^{***}$ | $-0.41^{***}$ | $-9.15^{***}$ | $-1.67$ | $1.76$ | $153.3$ | $--$ | $12.74$ |
| gemini-2.5-flash | $18.14^{***}$ | $-0.44^{***}$ | $-7.56^{***}$ | $-1.79$ | $2.16$ | $111.5$ | $--$ | $8.56$ |
| gemini-2.5-flash-lite | $3.19^{***}$ | $-0.30^{***}$ | $-1.22^{***}$ | $-2.88^{**}$ | $3.82^{***}$ | $8.8$ | $0.93$ | $1.53$ |
| gpt-5-2025-08-07 | $17.34^{***}$ | $-0.39^{***}$ | $-5.11^{***}$ | $-7.04^{***}$ | $5.62^{***}$ | $382.8$ | $--$ | $14.78$ |
| gpt-5-mini-2025-08-07 | $11.09^{***}$ | $-0.22^{***}$ | $-3.96^{***}$ | $-4.87^{***}$ | $5.10^{***}$ | $164.1$ | $--$ | $11.72$ |
| gpt-5-nano-2025-08-07 | $6.50^{***}$ | $-0.31^{***}$ | $-2.43^{***}$ | $-4.30^{***}$ | $4.12^{***}$ | $35.7$ | $0.94$ | $2.87$ |
| o3-2025-04-16 | $12.83^{***}$ | $-0.22^{***}$ | $-4.26^{***}$ | $-2.72$ | $2.07$ | $356.9$ | $--$ | $19.07$ |
| o4-mini-2025-04-16 | $13.00^{***}$ | $-0.23^{***}$ | $-4.89^{***}$ | $-5.99^{***}$ | $6.37^{***}$ | $132.1$ | $--$ | $10.86$ |
| DS-R1-0528 | $13.70^{***}$ | $-0.39^{***}$ | $-5.28^{***}$ | $-4.13^{***}$ | $4.19^{***}$ | $104.6$ | $--$ | $7.51$ |
| DS-Llama-70B | $8.36^{***}$ | $-0.30^{***}$ | $-3.28^{***}$ | $-3.50^{***}$ | $3.92^{***}$ | $69.8$ | $0.53$ | $5.14$ |
| DS-Qwen-32B | $5.15^{***}$ | $-0.19^{***}$ | $-2.12^{***}$ | $-2.07^{***}$ | $2.29^{***}$ | $54.3$ | $0.78$ | $3.95$ |
| DS-Qwen-7B | $1.74^{***}$ | $-0.23^{***}$ | $-0.96^{***}$ | $-0.45$ | $0.58^{*}$ | $2.9$ | $--$ | $-0.53$ |
| DS-Qwen-1.5B | $-0.35^{**}$ | $-0.20^{***}$ | $-0.33^{***}$ | $0.47$ | $-0.14$ | $0.0$ | $--$ | $-3.95$ |
| Phi-4-reasoning-plus | $9.52^{***}$ | $-0.45^{***}$ | $-3.58^{***}$ | $-4.21^{***}$ | $3.41^{***}$ | $45.7$ | $0.16$ | $3.68$ |
| Phi-4-reasoning | $9.11^{***}$ | $-0.39^{***}$ | $-3.35^{***}$ | $-4.62^{***}$ | $3.99^{***}$ | $52.3$ | $0.92$ | $4.08$ |
| Phi-4-mini-reasoning | $1.70^{***}$ | $-0.24^{***}$ | $-0.81^{***}$ | $-1.40^{***}$ | $1.45^{***}$ | $1.3$ | $--$ | $-0.55$ |
| EXAONE-Deep-32B | $4.09^{***}$ | $-0.26^{***}$ | $-1.55^{***}$ | $-3.16^{***}$ | $2.45^{***}$ | $14.1$ | $--$ | $1.0$ |
| QwQ-32B | $5.68^{***}$ | $-0.21^{***}$ | $-2.08^{***}$ | $-3.70^{***}$ | $3.07^{***}$ | $48.0$ | $0.95$ | $3.56$ |
| Qwen3-32B | $7.22^{***}$ | $-0.29^{***}$ | $-2.75^{***}$ | $-3.21^{***}$ | $2.75^{***}$ | $53.9$ | $0.94$ | $4.1$ |
| Qwen3-30B-A3B | $5.83^{***}$ | $-0.30^{***}$ | $-2.25^{***}$ | $-2.96^{***}$ | $2.87^{***}$ | $36.7$ | $0.99$ | $3.05$ |
| Qwen3-8B | $5.40^{***}$ | $-0.26^{***}$ | $-2.19^{***}$ | $-2.87^{***}$ | $2.66^{***}$ | $30.8$ | $--$ | $2.2$ |
| Qwen3-1.7B | $0.62^{***}$ | $-0.17^{***}$ | $-0.46^{***}$ | $-1.53^{***}$ | $1.24^{***}$ | $0.0$ | $--$ | $-4.07$ |

#### 3.1.1 Regression Model Specification

We model the performance of LLMs using a binomial generalized linear model (GLM) with a logit link function:

$$ $\Pr\!\bigl(Y{=}1\bigr)=\sigma\!\bigl(\beta_{0}+\beta_{d}\,d+\beta_{N}\,\log_{10}N+\beta_{\rho}\,\rho+\beta_{\rho^{2}}\,\rho^{2}\bigr),$ $$

where the binary outcome $Y$ represents exact-match accuracy ($Y=1$, when the model solves the puzzle correctly), $\sigma(\cdot)$ is the inverse logit function, and the coefficients $\beta_{d}$, $\beta_{N}$ and $\beta_{\rho}$ quantify sensitivity to intrinsic difficulty (ICL), task length (GCL), and distractor ratios (ECL), respectively. The inclusion of a quadratic term for $\rho$ is motivated by the characteristic U-shape observed in the third panel of [Figure˜1](https://arxiv.org/html/2509.18458v2#S1.F1) and based on an improved Akaike Information Criterion (AIC) value for 18 out of the 22 fitted models when included (see Appendix LABEL:supp-app:aic). Since $N$ ranges up to 250, we apply $\log_{10}$ to keep it at a similar scale as the other parameters of the regression.

#### 3.1.2 Significance of Main Effects

In all models, $\beta_{d}$ and $\beta_{N}$ are significant and highly negative, confirming performance degradation with increased ICL and GCL. The quadratic term for $\rho$ is also significant (except for two models) confirming the U-shaped response for most models: models typically perform worst at intermediate $\rho$ values and recover as $\rho$ approaches either extreme. Five models exhibit statistically insignificant coefficients for $\rho$ terms, reflecting poor baseline performance for the smallest models and indifference to distraction for strong models (i.e., o3, gemini-2.5-pro, gemini-2.5-flash).

#### 3.1.3 Capacity Points at 50% Accuracy

The GLM coefficients ([Table˜3](https://arxiv.org/html/2509.18458v2#S3.T3)) allow us to derive interpretable capacity thresholds. These represent the point at which a model’s accuracy is predicted to drop to 50% when varying a single load parameter, while holding other load parameters at their estimated mean values:

ECL50 (Effective Context Length): Maximum number of statements a model can process while maintaining 50% accuracy. Large ECL50 values indicate superior context handling.

NT50 (Needle-to-hay Threshold): Minimum proportion of relevant information required to maintain 50% accuracy. Crucially, small values indicate greater robustness to distractors. If the estimated NT50 is missing, then the model accuracy is not expected to cross the 50% threshold for any value $0\leq\rho\leq 1$, under mean conditions for $d$ and $N$.

ID50 (Intrinsic Difficulty): It is the maximum intrinsic complexity (number of interacting entities/attributes) that a model can handle while maintaining 50% accuracy. Negative values indicate failure to reach 50% accuracy even at the lowest difficulty setting under mean conditions for $N$ and $\rho$.

Mathematically, these thresholds are derived by setting the logit in the GLM equation to zero (for $\Pr(y=1)=0.5$) and solving for the parameter of interest, e.g.:

$$ $\small\mathrm{ECL}_{50}=10^{-({\beta_{0}+\beta_{d}\bar{d}+\beta_{\rho}\bar{\rho}+\beta_{\rho^{2}}\bar{\rho}^{2}})/{\beta_{N}}};\quad\mathrm{ID}_{50}=-(\beta_{0}+\beta_{N}\overline{\log_{10}N}+\beta_{\rho}\bar{\rho}+\beta_{\rho^{2}}\bar{\rho}^{2})/{\beta_{d}}.$ $$

For $\mathrm{NT}_{50}$, we solve the quadratic equation $\beta_{0}+\beta_{d}\bar{d}+\beta_{N}\overline{\log_{10}N}+\beta_{\rho}\rho+\beta_{\rho^{2}}\rho^{2}=0$ for $\rho$.

#### 3.1.4 Model Capacity

The regression analysis and estimated capacity thresholds (Table [3](https://arxiv.org/html/2509.18458v2#S3.T3)) reveal clear variations among models that can be grouped into three classes:

Frontier/High-capacity Models: gpt-5 and o3 lead by a wide margin (ECL${50}$ > 300), followed by gemini-2.5-pro, gpt-5-mini, o4-mini, and DS-R1-0528. The high baseline performance of gemini-2.5-pro ($\beta_{0}$ = 22.5) together with the large $\beta_{N}$ of -9.15 is consistent with the uniquely large amount long context errors as $N$ increases (as illustrated in the Appendix LABEL:supp-app:err_dist).

Mid-capacity Models: DS-Llama-70B , Qwen3-32B, DS-Qwen-32B, QwQ-32B, Phi-4-reasoning, Phi-4-reasoning-plus, Qwen3-30B-A3B, gpt-5-nano-2025-08-07, and Qwen3-8B form a broad middle tier with good performance at moderate $N$ and $d$ values.

Low-capacity Models: DS-Qwen-7B, Phi-4-mini-reasoning, DS-Qwen-1.5B, and Qwen3-1.7B exhibit minimal effective context handling capacity failing to reach 50% accuracy even under mean context/distractor conditions, deteriorating rapidly under slightly increasing load.

#### 3.1.5 Differential Sensitivity to Load Dimensions

The estimated coefficients further reveal distinct sensitivity profiles:

Sensitivity to context length ($\beta_{N}$): Universally negative and highly significant, larger models often show greater relative degradation compared to their higher baselines. Yet large ECL50 values for frontier models arise from the combined effect of $\beta_{0}$, $\beta_{N}$ indicating comparisons are best made via ECL50 and not $\beta_{N}$ in isolation.

Sensitivity to intrinsic difficulty ($\beta_{d}$):
Negative across models with a narrow range, it suggests a more uniform effect. Despite some steep $\beta_{d}$ values, high baselines (e.g., gemini-2.5-flash) yield large ID50 values unlike smaller models with similar $\beta_{d}$ (e.g., Phi-4-reasoning-plus).

Sensitivity to information relevance ($\beta_{\rho}$ and $\beta_{\rho^{2}}$): Confirms the U-shaped response, but NT50 values reveal nuanced distractor robustness variations masked by aggregate scores (e.g., DS-Llama-70B vs. Qwen3-32B). For frontier models, the absence of NT50 indicates achieving above 50% accuracy while for weak models the same absence indicates remaining below 50% accuracy.

### 3.2 Failure modes across models, length, and difficulty

We analyze error categories from the evaluation pipeline (see Appendix LABEL:supp-app:eval) to identify failure modes and provide the complete distributions and per-model breakdowns in the Appendix LABEL:supp-app:err_dist.

State-tracking mistakes dominate under load. Across models, the most common non-context failure is wrong final attribution in the last valid PoI sentence (valid-logic), consistent with mis-tracking sequential updates rather than formatting issues. For example, at $N$=250, Qwen3-32B has 2’541 valid-logic cases, DS-Llama-70B 2’465, and QwQ-32B 2’092. These logic errors also increase monotonically with $d$ for nearly all models.

Long-context budget overflows are a prominent, model-specific failure at extreme $N$. The max-context errors grow sharply with $N$ for some models: gemini-2.5-flash (280 errors in 350 samples at $N=250$) or gemini-2.5-pro (268/350). OpenAI models also make these errors at $N=250$ but at much lower levels (gpt-5: 32/350; o3: 24/348). The high error counts for the Gemini models indicate relatively poor token efficiency when reasoning.

Instruction-following drift emerges under higher $N$ and $d$, mainly in smaller models. While poi-logic stays near zero for most models, last-logic increases notably for compact models (e.g., Phi-4-mini-reasoning: 400 last-logic at $N=250$); DS-Qwen-7B: 116), indicating that under load, models often fail to answer in the instructed format.

"Other" failures rise with sequence length in small and mid-tier models. At $N=250$, DS-Qwen-1.5B has 605 “other” cases (often claiming the puzzle is unsolvable) and DS-Qwen-7B 461 indicating a shift from precise (but wrong) answers to non-answers as load grows.

## 4 Discussion

CogniLoad, by operationalizing CLT, enables a multi-dimensional evaluation of LLM reasoning, revealing nuanced failure patterns obscured by single-dimensional benchmarks. Our empirical results ([Section˜3](https://arxiv.org/html/2509.18458v2#S3)) offer several key insights: task length ($N$) emerges as a dominant determinant, suggesting challenges in sustained, germane-like processing for long, intrinsically demanding tasks; models exhibit distinct sensitivities to intrinsic difficulty ($d$) versus extraneous load ($\rho$), with the latter surprisingly showing U-shaped performance curves, indicating particular difficulties with intermediate distractor densities; and estimated capacity thresholds provide concise “cognitive fingerprints” for diagnostic LLM evaluation. The limitations of our study are summarized as follows:

Nuances of the CLT-LLM Analogy While CLT provides a powerful analogous framework, it is crucial to acknowledge that “cognitive load” in LLMs manifests as computational constraints (e.g., attention saturation, representational bottlenecks) rather than biological working memory limitations. Our operationalization of $N$ as a proxy for conditions demanding GCL, for example, is an abstraction. Future research should aim to bridge CLT concepts with direct, mechanistic measures of the underlying computational processes in LLMs to refine this analogy.

Scope of Reasoning and Generalizability CogniLoad focuses on sequential and pure deductive reasoning without requiring domain knowledge. While this reasoning type is fundamental to various subject areas (e.g., code, math), it is distinct from alternative reasoning paradigms like inductive, abductive, or analogical reasoning. Extending the CLT-grounded multi-dimensional evaluation to other reasoning types and evaluating it in other languages is a promising next step.

Beyond Accuracy and Main Effects The current evaluation relies on exact-match accuracy. Future iterations could incorporate richer metrics (e.g., step-wise reasoning fidelity, solution coherence, uncertainty of solutions) and systematically investigate interactions among $d,N,\rho$, which CogniLoad’s factorial design supports. Reinforcement learning on verifiable rewards (Guo et al., 2025) presents a promising application of CogniLoad in LLM training, as its generated metadata enables precise verification of reasoning steps despite limited data of this kind.

Despite these considerations, by decomposing the “task difficulty” into principled, controllable dimensions inspired by CLT, CogniLoad provides a more insightful perspective than single-dimensional benchmarks. It allows a more differentiated understanding of LLM reasoning capabilities and limitations, paving the way for more targeted development of robust and generalizable AI systems.

## 5 Conclusion

We introduced CogniLoad, a novel synthetic benchmark grounded in CLT for multi-dimensional evaluation of LLM long-context reasoning. By independently controlling parameters for intrinsic cognitive load ($d$), extraneous cognitive load ($\rho$), and task length ($N$), CogniLoad offers unprecedented diagnostic precision. Our evaluations revealed task length as a dominant performance constraint and uncovered unique “cognitive fingerprints” of LLM sensitivities to different load types, providing actionable insights beyond single-dimensional benchmarks. CogniLoad offers a reproducible, scalable, and theoretically-grounded tool to systematically dissect LLM reasoning limitations and guide the development of more capable and robust AI systems. While human and artificial cognition are mechanistically distinct, applying frameworks such as CLT to AI evaluation can provide valuable perspectives for understanding and characterizing their operational differences and capabilities.

## 6 LLM Disclosure

LLMs were used only in the early stages of writing the paper to refine phrasing and correct grammar. During coding they were used to update the chart formatting code, and to translate the manual implementation of the generation algorithm in Elixir to python for reproducibility.

## 7 Reproducibility

The code for generating CogniLoad puzzles according to the algorithm described in this paper is provided at: [https://anonymous.4open.science/r/cogniload-292B/](https://anonymous.4open.science/r/cogniload-292B/). The dataset of the puzzles on which the LLMs were evaluated for the results presented in this paper is provided on HuggingFace:
[https://huggingface.co/datasets/cogniloadteam/cogniload](https://huggingface.co/datasets/cogniloadteam/cogniload)

## 8 Author Contributions

All authors contributed in editing the manuscript towards the final version. D.K conceived the idea, designed and implemented the generation algorithm, prepared the dataset, scored the LLMs, analysed the empirical results, designed and implemented the evaluation pipeline, and wrote the first draft of the manuscript. All authors refined the benchmark design. B.R. compiled the first draft of the literature overview and acquired the computational resources for the evaluation.

## 9 Acknowledgements and Support

This work was supported by the Research Council of Norway through its Centres of Excellence scheme, Integreat - Norwegian Centre for knowledge-driven machine learning, project number 332645.

Ali Ramezani-Kebrya was supported by the Research Council of Norway through FRIPRO Grant under project number 356103, its Centres of Excellence scheme, Integreat - Norwegian Centre for knowledge-driven machine learning under project number 332645, and its Centre for Research-based Innovation funding scheme (Visual Intelligence under grant no. 309439).

We acknowledge NRIS Norway for awarding this project access to the LUMI supercomputer, owned by the EuroHPC Joint Undertaking, hosted by CSC (Finland) and the LUMI consortium through Sigma2, project number nn12027k.

## References

- An et al. [2024]
C. An, S. Gong, M. Zhong, X. Zhao, M. Li, J. Zhang, L. Kong, and X. Qiu.
L-eval: Instituting standardized evaluation for long context language models.
In *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 14388–14411, 2024.
- Bai et al. [2024a]
Y. Bai, X. Lv, J. Zhang, H. Lyu, J. Tang, Z. Huang, Z. Du, X. Liu, A. Zeng, L. Hou, et al.
Longbench: A bilingual, multitask benchmark for long context understanding.
In *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 3119–3137, 2024a.
- Bai et al. [2024b]
Y. Bai, S. Tu, J. Zhang, H. Peng, X. Wang, X. Lv, S. Cao, J. Xu, L. Hou, Y. Dong, et al.
Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks.
*arXiv preprint arXiv:2412.15204*, 2024b.
- Chandler and Sweller [1991]
P. Chandler and J. Sweller.
Cognitive load theory and the format of instruction.
*Cognition and instruction*, 8(4):293–332, 1991.
- Chollet et al. [2024]
F. Chollet, M. Knoop, G. Kamradt, and B. Landers.
Arc prize 2024: Technical report.
*arXiv preprint arXiv:2412.04604*, 2024.
- Cobbe et al. [2021]
K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al.
Training verifiers to solve math word problems.
*arXiv preprint arXiv:2110.14168*, 2021.
- Ericsson and Kintsch [1995]
K. A. Ericsson and W. Kintsch.
Long-term working memory.
*Psychological review*, 102(2):211, 1995.
- Gkamradt [2023]
Gkamradt.
Needle in a haystack - pressure testing llms, 2023.
URL [https://github.com/gkamradt/LLMTest_NeedleInAHaystack](https://github.com/gkamradt/LLMTest_NeedleInAHaystack).
- Guo et al. [2025]
D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al.
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.
*arXiv preprint arXiv:2501.12948*, 2025.
- Halford et al. [1998]
G. S. Halford, W. H. Wilson, and S. Phillips.
Processing capacity defined by relational complexity: Implications for comparative, developmental, and cognitive psychology.
*Behavioral and brain sciences*, 21(6):803–831, 1998.
- Hsieh et al. [2024]
C.-P. Hsieh, S. Sun, S. Kriman, S. Acharya, D. Rekesh, F. Jia, Y. Zhang, and B. Ginsburg.
Ruler: What’s the real context size of your long-context language models?
*arXiv preprint arXiv:2404.06654*, 2024.
- Kryściński et al. [2021]
W. Kryściński, N. Rajani, D. Agarwal, C. Xiong, and D. Radev.
Booksum: A collection of datasets for long-form narrative summarization.
*arXiv preprint arXiv:2105.08209*, 2021.
- Kuratov et al. [2024]
Y. Kuratov, A. Bulatov, P. Anokhin, I. Rodkin, D. Sorokin, A. Sorokin, and M. Burtsev.
Babilong: Testing the limits of llms with long context reasoning-in-a-haystack.
*Advances in Neural Information Processing Systems*, 37:106519–106554, 2024.
- Lieder and Griffiths [2020]
F. Lieder and T. L. Griffiths.
Resource-rational analysis: Understanding human cognition as the optimal use of limited computational resources.
*Behavioral and brain sciences*, 43:e1, 2020.
- Lin et al. [2025]
B. Y. Lin, R. L. Bras, K. Richardson, A. Sabharwal, R. Poovendran, P. Clark, and Y. Choi.
Zebralogic: On the scaling limits of llms for logical reasoning.
*arXiv preprint arXiv:2502.01100*, 2025.
- Ling et al. [2025]
Z. Ling, K. Liu, K. Yan, Y. Yang, W. Lin, T.-H. Fan, L. Shen, Z. Du, and J. Chen.
Longreason: A synthetic long-context reasoning benchmark via context expansion.
*arXiv preprint arXiv:2501.15089*, 2025.
- Liu et al. [2020]
J. Liu, L. Cui, H. Liu, D. Huang, Y. Wang, and Y. Zhang.
Logiqa: A challenge dataset for machine reading comprehension with logical reasoning.
*arXiv preprint arXiv:2007.08124*, 2020.
- Modarressi et al. [2025]
A. Modarressi, H. Deilamsalehy, F. Dernoncourt, T. Bui, R. A. Rossi, S. Yoon, and H. Schütze.
Nolima: Long-context evaluation beyond literal matching.
*arXiv preprint arXiv:2502.05167*, 2025.
- Paas et al. [2003]
F. Paas, A. Renkl, and J. Sweller.
Cognitive load theory and instructional design: Recent developments.
*Educational psychologist*, 38(1):1–4, 2003.
- Parmar et al. [2024]
M. Parmar, N. Patel, N. Varshney, M. Nakamura, M. Luo, S. Mashetty, A. Mitra, and C. Baral.
Logicbench: Towards systematic evaluation of logical reasoning ability of large language models.
*arXiv preprint arXiv:2404.15522*, 2024.
- Shaham et al. [2022]
U. Shaham, E. Segal, M. Ivgi, A. Efrat, O. Yoran, A. Haviv, A. Gupta, W. Xiong, M. Geva, J. Berant, et al.
Scrolls: Standardized comparison over long language sequences.
*arXiv preprint arXiv:2201.03533*, 2022.
- Shaham et al. [2023]
U. Shaham, M. Ivgi, A. Efrat, J. Berant, and O. Levy.
Zeroscrolls: A zero-shot benchmark for long text understanding.
*arXiv preprint arXiv:2305.14196*, 2023.
- Sinha et al. [2019]
K. Sinha, S. Sodhani, J. Dong, J. Pineau, and W. L. Hamilton.
Clutrr: A diagnostic benchmark for inductive reasoning from text.
*arXiv preprint arXiv:1908.06177*, 2019.
- Suzgun et al. [2022]
M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, et al.
Challenging big-bench tasks and whether chain-of-thought can solve them.
*arXiv preprint arXiv:2210.09261*, 2022.
- Sweller [1988]
J. Sweller.
Cognitive load during problem solving: Effects on learning.
*Cognitive science*, 12(2):257–285, 1988.
- Sweller [2010]
J. Sweller.
Element interactivity and intrinsic, extraneous, and germane cognitive load.
*Educational psychology review*, 22:123–138, 2010.
- Tay et al. [2020]
Y. Tay, M. Dehghani, S. Abnar, Y. Shen, D. Bahri, P. Pham, J. Rao, L. Yang, S. Ruder, and D. Metzler.
Long range arena: A benchmark for efficient transformers.
*arXiv preprint arXiv:2011.04006*, 2020.
- Vodrahalli et al. [2024]
K. Vodrahalli, S. Ontanon, N. Tripuraneni, K. Xu, S. Jain, R. Shivanna, J. Hui, N. Dikkala, M. Kazemi, B. Fatemi, et al.
Michelangelo: Long context evaluations beyond haystacks via latent structure queries.
*arXiv preprint arXiv:2409.12640*, 2024.
- Yan et al. [2025]
K. Yan, Z. Ling, K. Liu, Y. Yang, T.-H. Fan, L. Shen, Z. Du, and J. Chen.
Mir-bench: Benchmarking llm’s long-context intelligence via many-shot in-context inductive reasoning.
*arXiv preprint arXiv:2502.09933*, 2025.
- Yu et al. [2020]
W. Yu, Z. Jiang, Y. Dong, and J. Feng.
Reclor: A reading comprehension dataset requiring logical reasoning.
*arXiv preprint arXiv:2002.04326*, 2020.
- Yu et al. [2025]
Y. Yu, Q.-W. Zhang, L. Qiao, D. Yin, F. Li, J. Wang, Z. Chen, S. Zheng, X. Liang, and X. Sun.
Sequential-niah: A needle-in-a-haystack benchmark for extracting sequential needles from long contexts.
*arXiv preprint arXiv:2504.04713*, 2025.
- Zhong et al. [2021]
M. Zhong, D. Yin, T. Yu, A. Zaidi, M. Mutuma, R. Jha, A. H. Awadallah, A. Celikyilmaz, Y. Liu, X. Qiu, et al.
Qmsum: A new benchmark for query-based multi-domain meeting summarization.
*arXiv preprint arXiv:2104.05938*, 2021.
- Zhou et al. [2025]
Y. Zhou, H. Liu, Z. Chen, Y. Tian, and B. Chen.
GSM-infinite: How do your LLMs behave over infinitely increasing context length and reasoning complexity?, 2025.
URL [https://arxiv.org/abs/2502.05252](https://arxiv.org/abs/2502.05252).
- Zhu et al. [2025]
Q. Zhu, F. Huang, R. Peng, K. Lu, B. Yu, Q. Cheng, X. Qiu, X. Huang, and J. Lin.
Autologi: Automated generation of logic puzzles for evaluating reasoning abilities of large language models.
*arXiv preprint arXiv:2502.16906*, 2025.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: * Any thoughts on the $\rho$ parameter not quite being as clean as anticipated, as increasing it makes the problem both easier and harder in different ways? (see note above)
soundness: 2
strengths: * CogniLoad is a systematic benchmark and the first to make proper distinction between the 3 studied kinds of cognitive load LLMs have to deal with when solving hard reasoning problems.

* The work is grounded in a formalism from cognitive science, namely Cognitive Load Theory. Although some aspects (in particular, the connection to germane load) is a bit of a stretch, the notions of intrinsic and extrinsic load align very well with the notions of actual task and distracting information.

* The study is quite detailed, with many LLMs of various kinds evaluated on the proposed benchmark. The authors discuss several trends. While some of these (e.g., increasing the intrinsic difficulty reduces LLM accuracy) are both intuitive and, to some extent, reported before, the present paper does it carefully while keep all (or most) other variables constant. This, thus, provides a cleaner evidence. Additional analyses, like failure modes and differential sensitivity, are useful.

* The paper was easy to read and understand.
summary: This paper proposes a new benchmark, CogniLoad, that leverages concepts from the Cognitive Load Theory to design a logic grid puzzle reasoning benchmark that systematically evaluates LLMs on their *intrinsic* load (ICL), *extrinsic* load (ECL), and *germane* cognitive load (CGL), with the last one being approximated via task length $N$. The authors evaluate 22 SOTA LLMs and present several trends and insights on the scaling of each of these dimensions of reasoning ability, for different classes of LLMs.
weaknesses: * The paper can use a clearer description of the *germane* load from the CLT theory -- what it means, how it manifests, etc. -- along with some more discussion of why the task length parameter, $N$, is a good proxy for it.

* (stronger concern) The needle-to-hay parameter, $\rho$, is not as clean cut as intended, and this might even explain the U-shaped performance the authors observe. Specifically, the $\rho$ parameter mixes two distinct aspects: higher $\rho$ means more statements about the PoI (person of interest), which has two competing effects: (a) it means fewer distractor statements, which makes the problem easier to solve; and (b) it means a higher number of state changes for PoI, which makes the problem harder to solve. As an example, as one moves from $\rho = 5\%$ to $\rho = 10\%$, the number of distractor statements drop by (only) $5\%$, but on the other hand, the number of state transitions for PoI go up by 2x, i.e., double!! The authors don't discuss this important aspect and its implications. Note that this provides a reasonable explanation of the U shaped curve observed -- at very low values of $\rho$, the number of state transitions dominate the accuracy (the relative change in distractors isn't much), while at very high values of $\rho$, the relative change in the number of distractors is the one that's significant. I would ask the authors to look more deeply into this and see how it affects the message of and explanations in the paper.

* While it's good to have a formal puzzle definition in section 2, it is also important to verbally convey what each of the constraints is achieving. Can you add some explanatory text?

* For task length $N$, and especially at relatively higher $\rho$ values (which together means many state transitions for PoI), there is good reason to expect models to struggle without chain-of-thought reasoning, i.e., "non-reasoning models", based on theoretical results showing that transformers (without CoT) are inherently parallel (in complexity class $TC^0$ [1,2]) while state-tracking is a sequential task (in general well-known to be complete for the stronger complexity class $NC^1$). It's also known formally that using CoT allows models to perform state tracking and other sequential tasks [3,4]. This might explain some of the observed behavior, e.g., GPT-5 and o3 being resiliant to increase in $N$. Including some discussion of this connection would be valuable.

[1] The Parallelism Tradeoff: Limitations of Log-Precision Transformers, Merrill et al., 2022

[2] Transformers in Uniform TC0, Chiang, 2024

[3] The Expressive Power of Transformers with Chain of Thought, Merrill et al., 2024

[4] Chain of Thought Empowers Transformers to Solve Inherently Serial Problems, Li et al., 2024

---

confidence: 2
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. Please report, in the main paper, at least a compact interaction analysis among d, N, and ρ (one table with key coefficients or ANOVA‑style significance and effect sizes). The appendix suggests clear qualitative patterns; quantifying them would greatly help interpretation.
2. How sensitive are the main conclusions to the instruction format? If the final answer is allowed to be free‑form (no strict one‑sentence requirement), do “last‑logic” errors diminish while overall accuracy remains similar?
3. What is the average token length of puzzles at each N and d? Reporting token distributions per condition and per model tokenizer would help separate system limits from reasoning load.
4. Can you provide size‑matched comparisons with harmonized decoding settings (temperature, top‑p, max tokens) across representative models, to reduce variance induced by vendor defaults?
soundness: 2
strengths: The design isolates three often‑confounded aspects of long‑context reasoning and does so in a way that scales to arbitrarily long sequences. The generator is cleanly specified with validation constraints and produces semantically coherent, non‑trivial distractors. The regression view and derived thresholds offer compact, model‑specific “cognitive fingerprints” that go beyond aggregate accuracy; this is a useful diagnostic interface for both research and engineering. The error taxonomy and supplementary analyses (e.g., AIC comparison for quadratic ρ, confidence intervals) increase interpretability and make the empirical claims credible. The reproducibility is strong: the code repository and dataset release enable immediate verification.
summary: The paper introduces CogniLoad, a synthetic long‑context reasoning benchmark grounded in Cognitive Load Theory (CLT). It factorizes task stressors into three independently tunable dimensions: (1) intrinsic difficulty d (element interactivity and rule complexity), (2) distractor density ρ (needle‑to‑hay ratio), and (3) task length N (as an operational proxy for sustained “germane‑like” processing). The benchmark generates natural‑language logic‑grid puzzles with sequential state updates, evaluates exact‑match accuracy, and provides an error taxonomy. They have run factorial evaluations for 22 LLMs and fit a per‑model binomial GLM to derive interpretable “capacity thresholds” (ECL50, NT50, ID50). The main empirical findings are that length N is the dominant constraint, that distractors exhibit a characteristic U‑shaped performance curve with worst accuracy at intermediate ρ, and that models differ meaningfully in their sensitivity profiles.
weaknesses: The CLT analogy is motivating but only partially validated. Using sequence length N as a proxy for germane load conflates at least three mechanisms in LLMs - attention budget, recurrent state tracking over updates, and compounding interference from earlier updates. The paper acknowledges this, but the empirical analysis does not probe which of these mechanisms dominates or whether N’s effect interacts mechanistically with d and ρ in predictable ways.

The evaluation mixes performance and infrastructure limits. The max‑context failures are treated as an error category, but 32K token limits, tokenization differences, prompt overhead, and vendor defaults create avoidable confounds between “reasoning under load” and “serving limits.” Several models show many “long‑context budget overflow” errors at N=250; this is partly a systems artifact rather than a cognitive limitation. The paper would be stronger if it normalized prompt and decoding budgets across models, used a shared short instruction template, and pre‑tested that all N regimes fit for at least a subset of items per model.

Evaluation relies on an exact match with a final one‑sentence answer. The authors mitigate formatting drift via post‑processing and categorize “last‑logic” failures, but the metric remains coarse. Without step‑wise verification, it is hard to separate state‑tracking mistakes from late‑stage overwrites or local misparses. Given that the generator has full ground‑truth state trajectories, a step‑fidelity metric or a probabilistic grading of intermediate updates would add substantial value and make the “germane‑like” interpretation more concrete.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. The factorial design isolates each dimension, but are there ​interaction effects​ (e.g., high d+ high ρexacerbating errors)?
soundness: 3
strengths: **Originality​**

Introduces the first benchmark grounded in Cognitive Load Theory (CLT) for LLMs, disentangling intrinsic difficulty (d), distractor density (ρ), and task length (N) to diagnose reasoning failures.

​**Quality​**

Rigorous synthetic puzzle generation with independent parameter control, validated through large-scale evaluation and GLM-based capacity thresholds (ECL50, ID50).

​**Clarity​**

Well-structured with explicit CLT-LLM mapping. The operationalization of task length (N) as a proxy for germane load (GCL) could be further justified; visualizations (e.g., U-shaped ρ sensitivity) enhance interpretability.

​**Significance​**

Sets a new standard for interpretable LLM evaluation, revealing task length (N) as the dominant bottleneck and providing actionable "cognitive fingerprints" for model improvement.
summary: CogniLoad introduces a synthetic benchmark grounded in Cognitive Load Theory (CLT) to evaluate LLMs' long-context reasoning capabilities. This benchmark independently controls three dimensions: intrinsic difficulty, task length, and distractor density.

Key contributions include: (1) operationalizing CLT for LLM evaluation, (2) revealing task length as the dominant performance constraint, (3) identifying U-shaped sensitivity to distractors, and (4) providing capacity thresholds (e.g., ECL50, ID50) for 22 models.
weaknesses: **Limited Generalizability Beyond Deductive Reasoning**

The benchmark exclusively tests ​deductive logic puzzles, neglecting inductive/abductive reasoning or real-world tasks (e.g., math, code debugging). This restricts insights into broader reasoning capabilities.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. What is the distribution of lengths of the puzzles in tokens? What fraction of the context window is consumed by the inputs? This task seems to require tracking a considerable amount of information. How much context would actually be required to just write out the states after each step?

2. What do the findings of the error analysis mean for people trying to improve LLMs? Would training models via RL to perform state tracking tasks using the context window be a valuable thing to do?
soundness: 4
strengths: **Originality:** This paper offers a new lens for dividing up the factors which contribute to LLM performance using cognitive loading.

**Quality:** The results in the paper are sound and thorough. I am particularly pleased that the authors used statistical methods to perform their analysis of benchmark results, and have a large sample size. 

**Clarity:** The description of the approach is described in detailed mathematical notation, making it highly precise. 

**Significance:** Meaningful dimensions for the analysis of LLM results are important to guide research, and I find comparisons to cognitive science to be particularly interesting, as they draw on well-established concepts (though of course with the caveat that LLMs do not have human minds...). The algorithm for procedurally-generated benchmark items is also useful.
summary: The paper introduces a benchmark for long-context reasoning problems based on Cognitive Load Theory, which categorizes cognitive demands into three categories: intrinsic, extrinsic, and germane. Using this framework, they create puzzles in deductive reasoning and state tracking. They describe a generation procedure to change the difficulty of the benchmark by changing each of these dimensions separately. They generate 14,000 (4x5x7x100) puzzles to evaluate 22 LLMs across the dimensions. They fit a GLM and show the performance of the LLMs with respect to variations in each of the three dimensions, with task length still dominating.
weaknesses: I find the paper to be well-executed, and have minimal concerns about the quality of the results. The concerns that I do have come from unclear or missing details in the paper, grounding in cognitive science, and from uncertainty about the broader impact of the work. 

**Soundness:** As a sense check, I do want to know that the task is possible within the context lengths provided. As with another recent paper (https://arxiv.org/abs/2506.06941) models running over the context window seems like a poor justification for calling a performance a "reasoning" failure.

**Clarity:** 

1. The mathematical notation of the paper was comprehensive, but not always helpful. I think some pieces were missing, such as the use of k_t and m_t after their initial definition on line 199-200. I would also suggest distinguishing between "=" as a logical operator and "=" as a definition (probably by using ":="). v_{c,t} and u_{c,t} were a bit confusing along this lines, when moving between the definitions and their use validating statements.

2. I am not certain how the 14000 problems were actually finally generated. Was this done with an LLM? Did you create formal logical sets and then translate them into plain language? The latter certainly seems more reliable.

(I will note that although there were references to supplementary material, I only found a zip file with code and data, not further text. In any case, I do think that these details should be in the main body of the paper.)

**Cognitive Science:** I am not a cognitive scientist, but I felt that the grounding in this theory was relatively light. I would appreciate more explanation/justification about why d, N, \rho are the right way to operationalize these dimensions, beyond what appears in Table 1. In particular, my intuition is that d does not capture "difficulty" in the way I would usually conceive of it, since higher values require more working memory, but not more complex deductions. Similarly, I would imagine that instruction following is a subset of ECL, but seems to be described as outside of the three dimensions in the error analysis. One way to make this argument would be by comparison to a human exam that varies along the same cognitive dimensions, to show an analogy between how those variations are made and the variations made in this puzzle.

**Significance:** I am open to being wrong here, but I worry that this paper does not show us things we didn't already know. State tracking is a known weakness of LLMs (https://arxiv.org/abs/2404.08819, https://proceedings.neurips.cc/paper_files/paper/2024/hash/e6231c5f46598cfd09ff1970524e0436-Abstract-Conference.html). It is also a relatively simple task to be testing relative to the more complex reasoning benchmarks described in related works. The variation of performance along context length (most previous works), task size (https://arxiv.org/abs/2506.06941) and number of needles (https://arxiv.org/pdf/2404.06654) have also been shown previously. Being able to track these three dimensions separately is probably new, but I would like to see a clearer discussion of why this matters. 

I also find the error analysis to be somewhat unhelpful. Finding that the models often run out of context is described as as failure to use tokens efficiently, but efficient token use is not something that the benchmark describes itself as measuring. Why shouldn't the task be adjusted to avoid these issues in order to ensure a pure assessment of the target reasoning abilities?

Incidentally, I think addressing this can be tied to the previous comment about grounding in cognitive science. To the extent that the dimensions are well-justified, papers like (https://arxiv.org/html/2503.06378v2) have shown useful predictive results from cognitive science dimensions. 

**Overall**: I have netted out to a weak accept because I believe the work is methodologically sound, and want to be charitable about the significance, but I am very open to having my opinion swayed, particularly in how to weight the different elements of the papers.