Comprehension Without Competence: Architectural Limits of LLMs in Symbolic Computation and Reasoning
Authors: Zheng Zhang
Venue: Accepted by TMLR ()

Abstract: Large Language Models (LLMs) display striking surface fluency yet systematically fail at tasks requiring symbolic reasoning, arithmetic accuracy, and logical consistency. This paper offers a structural diagnosis of such failures, revealing a persistent gap between \textit{comprehension} and \textit{competence}. Through controlled experiments and architectural analysis, we demonstrate that LLMs often articulate correct principles without reliably applying them—a failure rooted not in knowledge access, but in computational execution. We term this phenomenon the computational \textit{split-brain syndrome}, where instruction and action pathways are geometrically and functionally dissociated. This core limitation recurs across domains, from mathematical operations to relational inferences, and explains why model behavior remains brittle even under idealized prompting. We argue that LLMs function as powerful pattern completion engines, but lack the architectural scaffolding for principled, compositional reasoning. Our findings delineate the boundary of current LLM capabilities and motivate future models with metacognitive control, principle lifting, and structurally grounded execution. This diagnosis also clarifies why mechanistic interpretability findings may reflect training-specific pattern coordination rather than universal computational principles, and why the geometric separation between instruction and execution pathways suggests limitations in neural introspection and mechanistic analysis.

Published in Transactions on Machine Learning Research (10/2025)
Comprehension Without Competence: Architectural Limits of LLMs in Symbolic Computation and Reasoning* *
Zheng Zhang
Reviewed on OpenReview: https: // openreview. net/ per/formm? id= G25HMiLLLq
Abstract
Large Language Models (LLMSs) display striking surface fluency yet systematically fail at tasks requiring symbolic reasoning, arithmetic accuracy, and logical consistency. This pa- per offers structural diagnosis of such failures, revealing persistent gap between com- prehension and competence. Through controlled experiments and architectural analysis, we demonstrate that LLMs often articulate correct principles without reliably applying them- failure rooted not. in knowledge access, but in computational execution. We term this phe- nomenon the computational aplit-braih syndrome, where instruction and action pathways are geometrically and functionally dissociated. This core limitation recurs across domains, from mathematical operations relational inferences, and explains why model behavior remains brittle even under idealized prompting. We argue that LLMs function powerful pattern completion engines, but lack the architectural scaffolding for principled, compositional rea- soning. Our findings delincate the boundary of current LLM capabilities and motivate future models with metacognitive control, principle lifting, and structurally grounded execution. This diagnosis also clarifies why mechanistic interpretability findings may reflect training- specific pattern coordination rather than universal computational principles, and why the geometric separation between instruction and execution pathways suggests limitations in neural introspection and mechanistic analysis.
1 Introduction
When asked whether 9.9 is greater than 9.11, Claude Sonnet 4 confidently states that 9.11 is larger, explaining that "since 90 greater than 11 in the hundredths place, 9.11 the larger number" while simultaneously calculating that 9.11 9.9 0.21 (incorrect). GPT-4o claims that "9.11 greater than 9.9" in mathematical contexts, invoking software versioning to justify the confusion. Yet when asked to explain how to compare decimal numbers, both models provide flawless algorithmic descriptions: "Write the numbers one above the other, aligning the decimal points... Compare digit by digit from left to right." Claude Sonnet 4 even works through the exact same comparison correctly as an instructional example, concluding that "9.90 > 9.11." Both models articulate decimal comparison procedures with textbook precision while failing to exccute them reliably.
This disconnect reveals a fundamental limitation in Large Language Models: they exhibit comprehension without competence systematic dissociation where modcls perfectly explain principles they camnot reliably execute. We term this phenomenon compulational splil-brain syndrome]] drawing analogy to neuro- logical conditions where different brain systems cannot coordinate effectively. Like patients who can verbally describe actions they cannot perform, LLMs develop geometrically separated pathways for "knowing about" procedures versus "executing" them.
"Code repository:
Video presentation: https://
youtu. be/2AuaLOGahyy
This work completed personal research project while employed at Amazon Web Services. The views expressed
those of the author and do not necessarily reflect those of Amazon.
'Te term also appears distributed systems to describe partitioned networks that cannot coordinate Brewer 2000
use neuroscience analogy better captures the functional dissociation between capabilities.

---

Published in Transactions on Machine Learning Research (10/2025)
Here, comprehension and competence are understood behaviorally whene an LLM flawlessly explains decimal comparison algorithms, observers reasonably judge it as "comprehending" the procedure, just we assess human understanding through explanation ability. Yet the model cannot reliably execute what it explains. Both capabilitios emerge from pattern completion but as goometrically separated, uncoordinated pathways explanation without execulion. This represents an inversion of Dennett's framnwork where competence typically precedes comprehension in biological systems (Denneti) 2017).
This split-brain syndrome manifests most visibly as computational hallucination- models must exe- cute multi-step algorithms they cannot actually perform, they generate plausible-soudding answers through pattern completion rather than computation. Unlike hallucinations that contradict facts or context, com- putational ballucination emerges directly from architectural impossibility; the model compares 9.11 with 9.9 not through arithmetic but by maatern-matching what such calculations typically look like. Every task requiring symbolic manipulation- from arithmetic to logical inference to data operations bectteess--eommss source of systematic hallucination, with models confidently producing answers that seem reasonable but lack com- putational grounding.
This is not limitation of scale, training data, or prompting techniques. We argue that computational split- brain syndrome stems from structural constraints in how LLMs represent symbols, learn procedures, and execute reasoning steps. This dissociation challenges fundamental assumptions about intelligence derived from human cognition, where explanatory fluency typically correlates with execution competence. LLMs systerattically violate this expectation, revealing that artificial and biological intelligence may operate under fundamentally different principles. Unlike symbolic systems that bind tokens abstract roles and apply rules over those bindings, LLMs operate pattern completion engines optimized for next-token prediction.
Main Claims of This Work. Our analysis establishes this disconnect through three interconnected claims:
Claim 1 (Representation): LLM token embeddings encode context-weighted averages that sys- tematically resist automatic domain binding, preventing stable symbolic circuits across computa- tional domains.
Claim 2 (Computation): Feed-forward networks face architectural impossibility of implementing exact symbolic operations through weight configuration alone, forcing them to resort to residual pattern fitting rather than implementing generalizable symbolic procedures.
Claim 3 (Iustruction-Eeecattion Disconnect): Nexi-token prediction objectives decouple algo- rithmic descriptions from executable behavior.
These three constraints must be understood logether to explain the computational split-brain syndrome:
Claims & 2 (Execution failures): When encountering execution instances (e.g., '9.11 > 9.9?",, contextual averaging prevents clean mathematical binding (Claim 1). Even if binding were perfect, transformers face architectural bottleneck: attention identifies operations and operands but cannot generate new values, forcing FFNs to produce results yet FFNs cannot implement exact operations, only pattern approximations (Claim 2). Together, these establish that reliable execution is impossible.
Claim (The disconnect): Next-token prediction provides no mechanism for well-learned instructions to guide execution. Both capabilities emerge as separate pattern-matching pathways with no automatic binding between them. The model thus can explain perfectly yet fail at execution the split-brain syndrome.
Importantly, demonstrate that these constraints manifest consistently across two critical domains: sym- bolic computation (arithmetic operations) and relational reasoning (logical inference and variable binding). In both domains, models exhibit the same split-brain syndrome fluent explanation coupled with unreli- able execution-reveeee a domain-specific failure.
this as limitation of the architecdure-training paradigm combination rather than
We now face a fundamental design trade-off By reducing all language processing to a single continuous opti- mization task (next-token prediction) over unified token embeddings, transformers achieve their remarkable
2

---

Published in Transactions on Machine Learning Research (10/2025)
generality and fluency. However, this same unification prevents the type-based reasoning and domain- specific binding that symbolic computation requires. The contextual averaging that enables rich semantic understanding also contaminates mathematical symbols; the single objective that scales so effectively treats instruction and execution as indistinguishable patterns. Understanding this trade-off clarifies why the com- pulational split-brain syndrome persists across model families and scales: it emerges from the very design choices that make LLMs powerful.
Contributions. Our analysis establishes computational split-brain syndrome as a unifying framework for understanding when and why L.Mss fail at systematic reasoning. We provide empirical evidence for each ar- chitectural constraint through embedding analysis, layer-by-layer computation tracking, and systematic eval- uation across arithmetic and logical tasks. We then examine three compensatory strategies srrategiss-sell-salllle -elf--caffolldiggg tool delegation, and hybrid architectures showing how each leverages LLMs' pattern completion strengths while working around their execution limitations, yet all require metacognitive capabilities that current architectures fundamentally lack.
Our findings suggest that mechanistic interpretability studies identifying "arithmetic circuits" or "adder neurons" may be observing sophisticated forms of pattern matching coordination that are formed path- dependently during learning, rather than genuine computational subroutines. More critically, the geometric dissociation between instruction and exccution pathways raiscs concerns about both model self-cxplanations and interpretability research: models may articulate reasoning procedures through neural roules distinct from those used for actual computation, potentially misleading both self-monit and researcher analysis of neural activations. Section 7 extends this analysis to modern interpretability methods, examining whether Sparse Autoencoders and feature-level decomposition escape these limitations or merely surface more stable statistical patterns.
These constraints appear unavoidable within the current paradigm: contextual averaging emerges inevitably from next-token prediction over diverse corpora, while predicting next tokens on internet-scale data forces FFNs to work with attention layers into hierarchical pattern assembly rather than principled computation, and cnstruction-execttiin gests that computational split-brain syndrome will persist across model families and scales unless addressed through fundamental architectural innovations rather than incremental improvements.
disconnect results from treating all text as equivalent prediction targets. This sug-
Roadmap. Section 2 reviews foundational work on transformer computation, symbolic reasoning, and re- cent interpretability findings that inform our analysis. In Section 3 we investigate LLM failures in symbolic computation, focusing on arithmetic tasks that reveal unstable embeddings and residual fitting behavior. Section $ extends this analysis to relational reasoning, demonstrating that the same architectural bottle- necks underlie L.M failures in multi-step inference, variable binding, and logical consistency. Section 5 examines compensatory approaches self-scaffolding, tool delegation, and hybrid architectures revealing how all three strategies converge on the same reliable metacognitive limitations that current architectures lack. Section G analyzes LLMs as hierarchical pattern completion engines, distinguishing between general intelligence (which LLMs achieve through sophisticated pattern matching) and gencralizable intelligence (which requires systematic rule discovery and principled reasoning), and examining the performance cliffs that emerge when tasks transition from pattern recognition to genuine algorithmic discovery. Section 7 explores how our framework applies to mechanistic interpretability research, examining whether current methods like Sparse Autoencoders identify genuine computational mechanisms statistical patterns, and proposing tests to distinguish between these possibilities. Section S offers a brief reflection on the research journey that led to these insights.
Limitations. This analysis focuses pretrained transformer-basee LLMs trained on next-token prediction objectives over natural language corpora, without access to external tools or explicit reasoning scaffolds. We analyze these "pure" systems to understand why compensatory strategies have become essential revealing not just what fails, but why tool use, scaffolding, and hybrid architectures represent necessary workarounds rather than optional enhancements for fundamental architectural constraints. Our empirical evaluation con- ters on specific model families (LLaMA2, Claude, GPT-4) and may not generalize to fundamentally different architectures training paradigms. The geometric separation experiments rely on t-SNE projections which
3

---

Published in Transactions on Machine Learning Research (10/2025)
may not capture all relevant representational structure. Our theoretical analysis of FFN computational limits focuses on ReLU networks and may not apply to other activation functions. While recent architectural varia- tions like mixturee-f-expeets or reasoning- augmented models may improve performance, they operate within the same fundamental transformer framework and thus likely face similar constraints- empirical validation of this hypothesis remains future work.
2 Related Work
This work builds on several lines of inquiry across theory, empirical analysis, architecture, and cognitive science. We organize our review around the emergence, diagnosis, and mitigation of systematic reasoning failures in LLMs, providing a unified framework for understanding when and why current architectures fall short of reliable symbolic computation.
2.1 Foundations and Theoretical Limits
Expressivity and Computational Boundaries. Recent theoretical frameworks have clarified the fun- damental boundaries of transformer expressivity. Peng et al. (202)) use communication complexity to prove that transformer layers cannot compose functions (c.g.. identifying grandparents in gencalogies) when do- mains are sufficiently large. Strobl et al. (2024) provide a comprehensive survey documenting how trans- formers function as recognizers of formal languages, revealing that while scaffolding strategies can improve performance within bounded expressivity classes, they remain fundamentally constrained by architectural limitations rather than achieving true symbolic reasoning capabilities. The Simons Institute workshop (Si-) mons Institute for the Theory of Computing, 2024) established that transformers act highly parallel pattern matching engines with limited capacity for long-range sequential computation, constraining their theoretical expressivity in structured symbolic tasks.
However, Li et al. (2024b) prove that scaffolding strategies, techniques in which models generate intermediate steps to guide reasoning, can theoretically expand computational power: While constant-depth transformers can only solve problems TC", adding T intermediate steps enables them to solve any problem solvable by Boolean circuits of size T. Chain-of- ofouthtttt prompting (Wei et al. 2022) exemplifies this approach: instead of directly computing '23 x 17 = ?", models generate '23 10 = 230, 23 x 7 161, 230 + 161 = 391', converting parallel constraints into serial computation. These analyses can be seen as the theoretical foundation for the use of self-soaffokding as principled workaround (see Section 5). Recent large reasoning models such ol and DeepSeek-RI (OpenAl 2024; DeepSeek-AN et al. 2025) show improved performance by essentially scaffolding their thought processes at inference time. Yet, they remain fundamentally compensatory rather than curative- -they by bypass core architectural bottlenecks rather than resolving them.
Training Dynamics and Path Dependence. Understanding how reasoning capabilities emerge during training reveals critical insights into LLM limitations. Power et al. (2022) demonstrate that late-phase gen- eralization involves a fundamental reorganization of internal representations, showing how training dynamics create path-dependent artifacts that appear as reasoning capabilities but reflect statistical regularities rather than systematic algorithms. Tiiges et al. (2024) extend this analysis to moderate scale LLMs with decoder only (up to 2.8B parameters), finding that while computational circuits emerge consistently across scale, their specific implementations vary significantly during training. We interpret these findings as suggesting that apparent "reasoning circuits" may reflect training-spocific pattern coordination rather than universal computational principles, although this inference requires further validation through controlled experiments with different training permutations.
Recent mechanistic analysis of factual learning provides direct evidence for these training-dependent phc- nomena. Zucchet et al. (2025) demonstrate that the acquisition of knowledge in transformers follows three distinct phases: general statistics learning, a plateau phase where attention circuits develop, followed by ac- quisition of individual-specfic knowledge. Critically, they show that adding new factual knowledge rapidly corrupts existing memories stored in feedforward layers, confirming that apparent knowledge storage reflects fragile statistical coordination rather than stable representation systems.
4

---

Published in Transactions on Machine Learning Research (10/2025)
2.2 Empirical Failures Across Domains
Symbolic Computation Breakdowns. Systematic evaluation reveals consistent failures in symbolic computation despite surface fluency. Srivastava et al. (2024) quantify sharp performance drops when prob- lem structures deviate from training distributions. Mirzadeh et al. (2024) demonstrate through the GSM- Symbolic benchmark that LLMs exhibit noticeable variance when only numerical values change, with per- formance drops up 65% when irrelevant clauses are added. This fragility confirms that current LLMs cannot perform genuine logical reasoning and instead replicate reasoning patterns from training data.
Dziri et al. (2023) provide comprehensive analysis showing that transformers solve compositional tasks through "lincarized subgraph matching"- memoriiin computation patterns rather than learning systematic algorithms. Their frequency analysis reveals that models succeed primarily when test computation subgraphs appeared frequently in training data, failing dramatically on cuttof-diitribbttion examples. Nikankkn et al.) (2024) confirm this patter-dependent behavior, showing that models rely on a "bag of houristics" for arith- metic rather than implementing systematic algorithms Wu al. (2025) further demonst that domain- specific fine-tuning can increase answer accuracy while degrading the logical coherence of chain-of-thought explanations, reinforcing the comprehension execution split.
Multiple recent studies provide converging evidence for the execution limitation across different domains. Yang et al. (2024) show that LLMs consistently outperform chain-of-thought when generating Prolog pro- grams for external execution rather than attempting direct calculation. The Illusion of Thinking commen- tary (Opus and Lawsen 2025) demonstrates similar improvements when LLMs generate Lua code for Tower of Hanoi problems rather than attempting direct move enumeration. Wolff and Hulsebos (2025) reveal significant deficits when LLMs attempt tabular reasoning tasks requiring multi-step algorithms such as puting averages or finding maximum values operations that would be trivial for database systems. Cheng et al. 2024) provide systematic evidence that LLMs excel at pattern recognition and code generation while struggling with deductive reasoning, with their successful framework architecturally separating symbolic code generation from symbolic code execution. This convergent pattern across arithmetic reasoning, logic puzzles, tabular data operations, and systematic inference demonstrates that LLMs excel at extracting predicates and generating symbolic representations while requiring external systems for reliable computation, precisely the instruction-exeention disconnect we formalize.
Relational and Logical Reasoning Failures. The same architectural constraints manifest in relational reasoning. [erglund et al. (2023) identify the "reversal curse," " where models fail to infer bidirectional relations from symmetric facts duc to asymmetric training exposure. [Nezhrriaa et al. 2024 show similar failures in multi-step family reasoning problems that should be trivial for systematic logical processing.
A controlled biography-corpun study by Allen-Zhu et al. (Allen-Zhu and Li 2024 reinforces the same point: a GPT-2 model trained exclusively on person relation tuples and matching QA pairs achieves near-perfect in in-distribution accuracy, yet its OOD QA accuracy on unseen entities collapses below 10% unless the pre- training data are aggressively rewritten, permuted, or translated. This highlights that relational exposure alone is insufficient for lifted reasoning; surface-form augmentation is required to abstract away from indi- vidual instances.
et et al. 2024a) provide systematic evaluation across inductive logic programming tasks, finding that LLMs despite being 100,000 times larger than specialized logical reasoning models perform dramatically worse on tasks requiring variable binding and systematic rule application. This performance gap persists even when given explicit logical structure through truth-value matrices, definitively ruling out training data scarcity as explanation and confirming architoctural limitations.
2.3 Mechanistic Explanations
Recent mechanistic interpretability rescarch has developed sophisticated tools for tracing computational pathways within transformer models. Attribution graph studies by Lindsey et al. Lindsey et al. (022b)) demonstrate methods for mapping how models transform inputs to outputs through intermediate representa-
5

---

Published in Transactions on Machine Learning Research (10/2025)
tional steps, while other circuit analyses reveal consistency patterns across training and scale Tigges et al, 2024).
Representational Pathologies. The inttruction-executie disconnect operates at the representational level through systematic failures in symbolic binding. McLeish et al. (2024) demonstrate that altering imput geometry alone can dramatically improve simple numerical reasoning, confirming that embedding ippresentations constitute significant bottleneck for symbolic manipulation. This supports our analysis that contextual averaging prevents stable domain binding required for symbolic manipulation.
Training-Orde Dependencies and Pattern Storage. Mechanistic interpretability reveals that ap- parent "reasoning circuits" often reflect distributed heuristics rather than principled computation. Nikankin et al. (024)) show that arithmetic behaviour emerges from "bag of heuristics" spread across layers instead of dedicated algorithmic modules. Tigges et al. (2024) further demonstrate that, even when a high-level algorithm scems stable, the specific attention heads instantiating it drift throughout training, undorscoring strong path dependence. Ye et al. (2025) train a 124M- -aMameeret GPT-2 exclusively on a procedurally generated grade-school math curriculum and uncover a linearly decodable dependency--ruphh representation that tracks which intermediate quantities depend on which others. We interpret these coherent circuits as artifacts of an unnaturally homogeneous training path; they dissolve once heterogeneous text is introduced, reinforcing our claim that stable algorithmic modules are fragile in realistic open-domain settings.
Reasoning Faithfulness and Post-Hoc Construction. Comprehensive analysis of reasoning faith- fulness reveals fundamental limitations in how LLMs construct explanations. Plaat et al. (2024) survey extensive evidence that LLM reasoning may be "post-hoc" and "constructed after conclusion has been found," with larger models showing less faithful reasoning. Chain-of-thougle contimues to work "even with invalid steps in the reasoning chain," suggesting pattern matching rather than logical exccution. This evi- dence strongly supports our geometric separalion hypothesis models access instructional pathways distinct from computational pathways when generating explanations.
Sparse Autoencoders and Feature-Level Analysis. Recent advances in mechanistic interpretabil- ity have shifted from ncuron-level analysis to feature-level dccomposition through Sparse Autocncoders (SAEs) (Bricken et al. 2023 Templeton et al.] 2024). These methods extract interpretable features that appear more stable across models than individual neurons (Wang et al., 2025), with techniques like Linear Computation Graphs tracing complete computational pathways through these features (He et al.) 2024)) While SAEs promise to overcome the polysemanticity and path-dependency limitations of neuron-level anal- ysis, we examine in Section 7 whether these features represent genuine computational mechanisms or merely more stable statistical patterns. Our analysis suggests that even sophisticated feature decomposition cannot escape the fundamental architectural constraints that create the computational split-brain syndrome.
2.4 Compensatory Strategies and Architectural Remedies
Prompt-Level and Self-Scaffolding Interventions. The effectiveness of compensatory strategies has theoretical grounding in computational complexity. et al. (2024b) prove that chain-of-thought fundamen- tally changes the computational expressiveness of transformers: while constant-depth transformers without CoT can only solve problems in TC", adding steps of CoT enables them to solve any problem solvable by Boolcan circuits of size T. This provides the thcoretical foundation for why self-scaffoldig and chain-of- thought prompting partially succeed they effectively transform parallel computation constraints into serial computation opportunities, allowing transformers to simulate P/poly with polynomially many intermedi- ate steps. This theoretical result directly explains the empirical success of self-scaffolding as one of the compensatory strategies.
However, Peng et al. (2024) demonstrate that chain-of tfouthtttthtttttoo requires an exponentially growing number of intermodiate steps for complex function composition tasks, suggesting that self-scaffolding becomes incffective when problems demand impractically long reasoning chains. This theoretical prediction aligns with empirical failures we discuss in Section

---

Published in Transactions on Machine Learning Research (10/2025)
Beyond standard chain-of-thought (Wei et al. 2022), rescarchers have proposed metacognitive variants that embed self-evaluation into the reasoning process. AbstRaT, (Gao et al. 2025 trains models to first generate an abstract symbolic representation of problem via reinforcement learning before attempting execution, substantially improving robustness under distribution shift. Self-correction frameworks such as SPOC (Zhao
ett al. 2025 train models interleave solution generation with explicit verification sleps, enabling dynamii revision when the internal verifier detects errors. This exemplifics the self-scafoolding pattern analyzed
Section 5
Tool Integration and Hybrid Exccution.
Tool-angmented approaches explicitly address the exocution
gap. Toolformer (Schick al., 2023 enables automated tool calling, while ReAct (Yao al. 2022) combines reasoning with external action. Yang et al. (2024) provide-mmirical empirical validation that this separation LLMs for
predicate extraction, external systems for computation consistently outperforms end-to-end LLM execution across arithmetic tasks.
Architectural Modifications. To address fundamental limitations, researchers have explored hybrid ar- uritectures incorporaling explicit symbolic modules. OccamLLM (Dggan et. al. 2024) and IGC (Dietz and Klakow 2024) embed arithmetic reasoning units, while Logic-LM (Pan et al. 2023 NLM (Dong et al, 2019), and Diffcrentiable Logic Machines (Zimmer et al.) 2023) implement logical operations through struc- tured tensors. These systems represent promising directions for true architectural solutions rather than compensatory workarounds.
2.5 Positioning and Contributions
Concurrent work by Lin et al. (Lin et al. 2025 observes similar disconnect between LLMs' ability to generale symbolic lext and execute reliably, framing this safety and aligmment. concern arising from "semiotic disconnects." While we share this empirical observation, understanding how these limitations emerge requires identifying the architectural factors that create them. Our work addresses this in three crucial ways.
First, while previous studies document what failures occur across domains, we provide principled mech- anistic explanation for why they occur through three interconnected architectural constraints: contextual averaging, computational impossibility, and Instruction-exeentton disconnect. Second, rather than treating arithmetic and relational reasoning as separate problems, we demonstrate that identical underlying limi- tations create the computational split-brain syndrome across all symbolic domains. Third, our geometric analysis reveals that this disconnect operates at the representational level, with direct implications for inter- pretability research showing why mechanistic studies often surface training-dependent. artifacts rather than universal computational principles.
This unified framework not only explains current limitations but predicts why certain compensatory strate- gies succeed while others fail, providing foundation for developing more reliable reasoning systems that leverage LLMs' pattern completion strengths while addressing their exccution limitations through principled architectural modifications.
3 Structural Limits Symbolic Computation
The computational split-brain syndrome manifests most elearly in symbolic computation, where LLMs can perfectly explain algorithms they cannot reliably execute. This systematic dissociation drawing analogy to neurological conditions where different brain systems cannot coordinate effectively- emerges geometric separation in representational space: models develop distinct pathways for "knowing about" procedures versus "executing" them.
This section examines the architectural constraints that prevent reliable symbolic reasoning through a causal chain of three interdependent factors. Importantly, no single constraint alone would be cont--ouat averaging could be overcome with better training techniques, architectural limitations might be addressed through scaling, and instruction-exeention disconnect could potentially be bridged through clever prompt-
7

---

Published in Transactions on Machine Learning Research (10/2025)
ing. However, these three constraints reinforce each other systematically: contextual averaging prevents automatic domain binding (Claim 1), while architectural limitations make direct symbolic computation impossible through weight configuration alone, forcing models toward pattern storage (Claim 2), and next- token prediction treats algorithmic descriptions and exccution traces as equivalent pattern-completionn tasks, preventing instructional guidance from bridging this gap (Claim 3). Together, these interdependent factors reveal why LLMs function as sophisticated pattern completion engines rather than symbolic reasoners, with implications extending to relational reasoning we explore in Section 4
3.1 Claim 1: Contextual Averaging Prevents Automatic Domain Binding
Token embeddings in LLMs encode wengexd-weighted averages that resist automatic domain binding, making stable symbolic circuits extremely difficult to form. Unlike traditional programs that explicitly bind variables to types and domains, LLMs derive token meanings from statistical patterns across all training contexts.
The Failure of Mathematical Binding. The root of this problem lies in how LLMs Iearn token repre- sentations. During training, the model encounters "9.11" in countless different contexts historical articles about September 11th, software version discussions, financial data, and mathematical expressions. The training objective forces the model to find a single embedding that works reasonably well across all these contexts.
Formally, this creates fundamental tension. The training objective optimizes embeddings through next- token prediction:
[(0) =
Σ log (wre wee)
(1)
e" ("(w) = arg min
Σ
Eun'naa ( | log po (wo e,, E(c w())
(2)
eEC(we
Equation 0 is the familiar next-token prediction loss, where denotes the model parameters, my is the token at position t, Wet represents all tokens before position t, and p= is the conditional probability distribution learned by the model. Equation shows how individual token embeddings are optimized: e (w) is the optimal embedding for token w, C(w) denotes all contexts containing token w, e is the candidate embedding being optimized, E(c) w) represents embeddings of all tokens in context except w, and w' is the next token sampled from the data distribution Pdatal-le. w). This optimization produces context-averaged embeddings that trade off among multiple usage modes, similar to distributional semantics models such as Word2Vec (Mikolov et al. 2013).
In formal reasoning, we bind symbols to precise domains (a = 9.9, a E R), stripping away non-matthemalical associations. LLMs typically struggle to perform this domain binding reliably. When encountering "9.9", they tend to preserve contextual associations rather than mapping to clean mathematical representation.
To demonstrate this contextual contamination, presented LLaMA2-7B-chan with prompts like "9.11 is a" and "9.9 is a". The completions reveal dramatically different semantic associations:
For "9.11 is a", the model produces: "day", "date", "remembrance", "tragedy"
For "9.9 is a", the model produces: "number", "decimal", "perfect", "high"
This divergence shows "9.11" retaining historical event associations while "9.9" behaves as a scalar quantity. Quantitative analysis using negative ligelikelssoods across different templates (DATE, VERSION, MEA- SURE) confirms those contextual biases. The missing operation is domain binding ostablishing "9.11" element in R governed by mathematical laws rather than historical associations.
Failure of Embedding Geometry. Symbolically stable circuits require representation spaces that pre- consistent structural relationships. While this doesn't demand strictly Fuclidean geometry, it requires
8

---

Published in Transactions on Machine Learning Research (10/2025)
Numeric embeddings in LLaMA2-TB-chat preserve symbolic geometry
Cosine distances: digits
(b) Contextusl decimals (5.11220)
Digits wordes "5" "live"
005
8:03
0:22
0:48
0:51
6:53 8%
1.70
1.0
0:43
0:46
0.51
8.00
0.55
0.51
0.54
C5
0C3
0.5
0.49
0.43
0:00
0:45
0.52
0:55
0.57
0.30
6:77
0.55
0054
0.50
0.53
6.00
0:40
0.50
0C4
0.3
0.51
0.55
0.52
0.32
0.45
040
0.00
0.50
-0.0
UC2
0.75
0.54
0.53
0.05
0.55
0.53
0.45
D00
0.55
01
OC1
0:4
8
0070
84
RC3
Decimal values
Figure 1: Numeric embeddings in -LaMAA2-B-chas fail to preserve symbolic geometry: (a) irregular cosine
distances between digit tokens; (b) bifurcation at "10.0" due to tokenization boundaries; (c) large distances between equivalent representations like "5" and "five".
systematic, predictable structure. As Equation 2 predicts, contextualll--veeagee embeddings resist the do- main binding required for stable symbolic circuits.
The embedding analysis reveals the extent of this geometric chaos (Figure What should be the mathe- matically ordered space of numbers instead resembles scrambled puzzle, with qualitatively similar failures observed across Qwen and Mistral models:
Panel (a): Cosine distances between digit tokens "1 "10" show irregular, asymmetric patterns. Notable violations of numerical ordering include "10" being closer to "1" (distance 0.20) than to "g" (distance 0.55), and "6" being closer to "10" (distance 0.59) than to adjacent digits like "7" (distance 0.40).
Panel (b): Decimal embeddings exhibit clear clustering by first digit rather than numerical prox- imity. The sharp visual boundaries separate 9.x values (top=left dark block), 10.x values (middle diagonal band), and 11.x-12.0 values (bottom-right region), revealing tokenization-driven rather than mathemmatically-rrinigg
organization.
Panel (c): Symbolic equivalents show systematic misalignment in embedding space. Ideally, we should observe single diagonal of low distances where digit-word pairs ("1"/"one", "2" """/""""", etc.) align. Instead, the heatmap reveals fragmented diagonal patterns with significant irregularities. While some digit-word pairs show reasonable proximity (c.g., "5" is relatively close to "five"), the embedding space exhibits systematic failures: a group of higher-numbered words like "ten", "eleven", and "twelve" show poor alignment with any digit tokens, and paradoxically, "5" appears closer "fifteen" " than many other digits are to their correct word equivalents. This demonstrates that the embedding geometry fails to consistently preserve mathematical equivalence relationships.
These geometric failures directly impede the formation of stable arithmetic circuits that could operate reliably across different input formats numerical ranges. For LLMs to form stable, input-agnostic circuits for any algorithm, inputs must preserve the properties that algorithm expects, Consider sorting-it fails if vector representations have random perturbations that violate ordering rela- tionships. Without isometric properties in embedding space (preserving numerical order and relationships). models cannot form circuits that work consistently across all inputs. McLLish et al. (2024) demonstrate that altering input geometry alone can dramatically improve simple numerical reasoning, which might appear contradict our contextual averaging argument. However, their approach involves training models exclusively arithmetic problems, which inadvertently avoids the fundamental contextual averaging problem faced by
when cast in high dimensions.

---

Published in Transactions on Machine Learning Research (10/2025)
x() = xx(111 Ah AMPPN
A
attn Attention (a (1)1-11
(3)
PFN
(1)
atta = (1-1) + A. attn
(1)
A 10)
FFN FFN,
(5)
PFN
1 + +AMp
PEN
(8)
(6)
= (111) + A A
(7)
Ah attn
ATTN
Final output after L layers:
x(1-1)
( embeddings
A !I) sataa
A
! PPNN
Figure 2: Computational flow through transformer layers. When processing arithmetic operations (e.g., "43 x 78 ?"), attention aggregates the operation context (operator and operands) while FFNs must generate the result. Due to causal masking, attention at the "=" position only compute weighted averages of the embeddings it sees, creating representation that cannot contain the novel direction for the result token.
real-world LLMs. Once embeddings are contaminated through multi-domain training where tokens like "9.11" accumulate conflicting associations across historical, versioning, and mathematical contexts this ge- ometric solution breaks down. The McLeish results thus support rather than refute our analysis: they show embedding geometry matters precisely because real-world training makes it unsolvable.
But suppose could somehow solve the representation problem imagine LLMs had perfect, mathenaateelll-ssruuuuurddd embeddings where "9.9" and "9.11" occupied precisely the right positions in embedding space. Would this enable reliable symbolic computation? Unfortunately, a deeper architectural constraint ensures the answer is no. Even with idcal symbolic reprosontations, the computational mecha- nisms of transformer networks face fundamental limitations that force them toward pattern storage rather than algorithmic execution.
3.2 Claim 2: Feed-Forward Networks Resort to Pattern Storage
Even if we solved the reprosentation problem-imagine LLMs had perfect, mathematieally-sttuuuure beddings-we would transformers be able to execute symbolic computation? The answer reveals a profound gap between capacity and capability. Feed-forward networks possess, through the Universal Approxima- tion Theorem, the capacity to approximate any contimuous function. Yot in practico, they systematically fail at symbolic computation, resorting instead to pattern storage. This paradox lies at the heart of the computational split-brain syndrome.
The Architectural Division of Labor. To understand this paradox, we must first. examine how com- putation flows through the transformer architecture and identify where symbolic computation must occur.
Figure 2 illustrates the computational pipeline of transformer layers Each transformer layer implements two successive residual connections: attention reads from the residual stream and adds its output back, creating intermediate state that FFN then reads, transforms, and adds to produce the next layer's representation.
Consider concrete example: '43 x 78 = ??. Due to causal masking at the '=' position, attention only sees {43, x, 78, and computes weighted averages of these embeddings. This creates a rich representation encoding 'multiply 43 by 78' 78' iee-idnttfyiing the operation and operands- but crucially, since attention can
We omit layer normalization and dropout from analysis these operations being clement-wise rescaling and mask- ing-do not alter the fundamental computational limitations identify. LayerNorm normalizes representations but cannot. enable symbolic computation.
10

---

Published in Transactions on Machine Learning Research (10/2025)
only compute weighted combinations of its imput embeddings, this representation cannot contain the novel embedding direction needed for '3354', which lies outside the space spanned by the input tokens.
Feed-forward networks then receive this attention-enriched representation and must somehow produce the correct result. Since attention cannot generate novel values, FFNs bear the entire computational burden: transforming the encoding of 'multiply 43 by 78' into the embedding direction for 3354'.
Theoretical Capacity via Universal Approximation. In principle, FFNs have sufficient capacity for this task. Each FFN implements an expand-compresd architecture Vaswani et al. 2017
FFN(z) = W2 20--ReLLU(112 b1) +
(8)
By the Universal Approximation Theorem HHrnik 1991 such networks approximate any continuous function f : Rd + I to arbitrary precision compact sets. Theoretically, this includes multiplication: given sufficient width and appropriate weights, an FFN could approximate xy within any bounded domain.
This raises a critical question: FFNs have the capacity to approximate multiplication, why do they fail so catastrophically in practice?
The Training Reality: Role Specialization. gradient descent trains them to do. We now present our central theoretical result:
The answer lies not in what FFNs can do, but in what
Theorem (Role Specialization under Next-Token Prediction)
Under gradient descent optimization for next-token prediction loss L log po (wrlwee)) the three parameter sets necessarily specialize into distinct computational roles:
OF (embeddings) - Context-averaged representations
A (attention) + Operation context encoding
OF (FFNs) Memorized pattern-to-result mapping
Proof Sketch: The cross-entropy loss requires x(4)_ Cresult to be maximal among all vocabulary to- kens. Since attention constrained to weighted averages of inputs, it cannot generate an embedding direction for the result. token that lies outside the space spanned by the input embeddings. FFNs must therefore learn the discrete mapping from attention's 's operation encoding result embeddings. While UAT guarantees this mapping can be approximated on the training set, gradient descent op- timizes for memorizing specific gatter-to--token mappings rather than discovering algorithmic rules. See Appendix B for detailed analysis.
Why Pattern Storage Wins. To understand why memorization emerges inevitably, consider the opti- mization dynamics for arithmetic like $43 x 78 3354'. The gradient descent must minimize:
log pl (3354 43 78 78=))
This requires the final representation (LL) to have maximum dot product with the embedding of token '3354' compared to all other vocabulary tokens. The gradients flow through three components with distinct constraints:
Embeddings lack the isometric properties required for arithmetic tokens cannot pre- serve ordering distance relationships due to contextual averaging (Section 3.1
Attention can only compute weighted averages of existing embeddings, unable to generate novel token directions outside their span.
11

---

Published in Transactions on Machine Learning Research (10/2025)
MULTIPLICATION Convergence to Final Layer
ADDITION
Convergence to Final Layer
improvsment: 0.40d
10
Improvement: 0.407
0.5
0.6
Similarity to Einal
Casine Similarity Finel
0.1
04
Early Middle
Larty
MULT (::5)
0.-
ADD (2:5E)
10
25
Layer
Layer
Figure 3: Experimental validation of hierarchical pattern assembly in arithmetic computation. Layer-by- layer convergence analysis for multiplication (left) and addition (right) shows progressive refinement from initial similarity of ~0.07 to late-layer similarity of ~0.48. Error bars represent standard error across 10 examples per operation. The consistent improvement patterns (0.404 for multiplication, 0.407 for addition) provide direct neural evidence for residual pattern fitting rather than algorithmic computation. Shaded regions indicate early (blue), middle (orange), and late (green) processing phases.
FFNs must bridge the gap: mapping attention's encoding to the specific embedding direction that maximizes probability for '3354".
Since FFNs cannot implement true multiplication (piecewise linear functions cannot compute x x ex- actly- see Appendix A gradient descent converges to the only viable solution: memorizing a lookup table of pattern + result mappings. The UAT capacity that could theoretically approximate multiplication is instead allocated to storing these discrete mappings.
Experimental Validation of Hierarchical Pattern Assembly. To test our theoretical framework, we analyzed layer-by-layer hidden state trajectories in LLaMA-2-7BB during arithmetic computationFor prompts like "23 96 =" and "63.7 + 3.5 =", we measured how each layer's representation at the prediction position converges toward the final layer's output representation.
Figure 3 demonstrates strong empirical support for theoretical framework. Both multiplication and ad- dition exhibit nearly identical progressive refinement patterns: representations begin with low similarity to the final output (~0.07) and gradually converge through three distinct phases. This layer-by-layer analysis confirms that since each individual FFN faces the same architectural impossibility of exact. computation, the only viable learning strategy becomes hierarchical pattern decomposition, where multiple FFNs coor- dinate their residual contributions through learned pattern sequences rather than implementing systematic algorithms.
This pattern storage bchavior aligns with recent findings by Dziri al. 2023 who demonstrate that transformers memorize compulation patterns from training rather than learning generalizable algorithms, with models succeeding primarily when test patterns appeared frequently in training data.
The Compnaatiio-Commtttttiii Gap. The fundamental distinction between what FFNs can do in theory versus practice reveals why pattern completion differs from algorithmic approximation:
12

---

Published in Transactions on Machine Learning Research (10/2025)
UAT Approximation:
FFNs can learn f K I2 on compact training set K, successfully memorixing patterns like '23x + 391'
Symbolic Computation: True algorithms exccute for any valid input, maintaining systematic behavior on novel combinations.
The Key Difference: Robustness to distribution shift algorithmic implementation (even imper- lect) generalizes systematically, while paltern completion catast fails outside memorized examples.
For novel inputs outside the training support K, FFNs have no algorithm to execule only nearby memorized patterns to interpolate between. The discrete nature of token prediction exacerbates this: small errors in x.44) cause complete failure (wrong token), unlike continuous approximation where errors degrade gracefully, If students memorized multiplication tables then tried 3-digit problems, they would fail completely- true algorithmic understanding enables generalization.
Many researchers, including ourselves initially, searched for generalizable arithmetic circuitry in LLMs. At inference time, such circuitry does not exist, as confirmed by studics showing LLMs rely on "bags of heuris- ties" rather than systematic algorithms (Nikankin et. al., 2024 T Tigges et al. 2024). Models perform pattern completion whether explaining algorithms or attempting execution they just complete different memorized patterns.
Complexity-Theoree Confirmation. This empirical limitation aligns with theoretical impossibility results. Li et al. (2024b) prove that constant-depth transformers can only compute functions TC while multiplication requires serial carry propagation- fundamentally outside TC Even with universal approxi- mation capacity, architectural constraints prevent true symbolic computation in single forward pass. The TC result establishes that no clever weight configuration can enable exact multiplication, while our analysis shows that training dynamics allocate the available approximation capacily toward pattern memorization rather than algorithmic discovery.
This analysis of pattern storage versus algorithmic computation raises a puzzling question: LLMs cannot execute symbolic operations reliably, why do they excel at. explaining algorithmic procedures? When asked "How do you multiply two numbers?", models provide textbook perffett explanations of the algorithm. This suggosts a potential solution: perhaps these articulated procedures could implicitly guide exccution, with the model's comprehensive knowledge of algorithms compensating for its execution limitations. Unfortunately, this intuitive hope fails due to a third architectural constraint that prevents instructional knowledge from rescuing symbolic computation.
3.3 Claim 3: Next- Token Prediction Decouples Instruction from Execution
The next-token objective treats algorithmic descriptions and execution examples as equivalent prediction tasks, preventing the binding of procedural knowledge to computational implementation. Even if archi- tectural mechanisms existed to enable instructional guidance, the geometric separation of instruction and exccution pathways ensures that procodural knowledge cannot rescue symbolic computation.
The False Promise of Instructional Binding. Training data contains examples that might appear to support instruction-cxeeatiae binding. Consider instructional sequences like:
"To mulliply 23 17: First mulliply 23 x = 161, then mulliply 23 x 10 230, finally add 161 + 230- 391."
Such examples seem to demonstrate algorithmic steps paired with execution traces, potentially enabling models to bind descriptions to implementations. However, next-token prediction processes these as sequential pattern completions without distinguishing instructional content from computational steps.
The expectation that exposure to such instruction-cxeeutaae pairs would automatically enable binding rep- resents fundamental illusion about neural learning. Training models to recite algorithmic procedures does
13

---

Published in Transactions on Machine Learning Research (10/2025)
t-SNE of Instruction, Execution, and Word-Form Embeddings
add instruction
sub instruction
60
div instructiion
compare. instruction
execution
40
sub, execution
mult_exetationn
compare. execution
add, execution, word sub www.eeeeeoo.tte word mult execution, word
20
ewwcucttonnneeee word compare_ caarrreeeecttonwwwee
c:NE Dimension
-20
-40
-60
.80 60 40 -20
20
40
Figure 4: t-SNE projection of instruction, symbolic execution, and word-form execution embeddings from LaaMA2-7B-haa across 50 problems each in five arithmetic operations. Instructions (circles), symbolic expressions (squares), and word-form results (triangles) occupy geometrically distinct regions, despite de- scribing the same operation. This illustrates that LLaMA2-7B-chat stores instructional knowledge and executional know-how in separate regions of its latent space.
not automatically configure weights to perform those operations or enable inference-time binding between instruction and exccution. The next-token objective does not provide mechanism for such binding: it treats "how to multiply" and "what is 23x17" independent text completion tasks.
Indiscriminate Processing Across Context Types. The training corpus contains arithmetic in diverse contexts: instructional examples with step-by-step explanations, homework solutions, calculations embedded within articles, and standalone problem-answer pairs. Next-token prediction treats all of these insstruct ional descriptions, execution traces, and standalone calculations as equivalent token sequences to predict, without distinguishing their functional roles.
This creates two separate learning pathways with distinct representational signatures: models learn to recite algorithms by pattern-matching instructional texts, while learning execute operations by retricving stored pallern fragments from training examples. Critically, these compelencies become geometrically separated in the model's latent space coore-iistructionaa knowledge and execution capabilities occupy different representa- tional regions.
Experimental Validation of Geometric Separation. our hypothesis correct, we should able directly observe this scparation the model's representational space. When model encounters "To multiply 56 and 76, first break one number into parts..." " versus "56 76 4256," " are these processed by the same neural pathways completely different ones?
14

---

Published in Transactions on Machine Learning Research (10/2025)
To test this, we constructed dataset of 250 arithmetic problems spanning five operations, each expressed in three distinct forms:
an instructional sentence that explains the procedure (e.g., "To multiply 56 and 76, first break one number into parts..").
a symbolic execution form (e.g., "56 76 4256"), and
word-form result (e.g., "A worker makes 56 parts per day for 76 days, totaling 4256 parts").
We embedded each sentence using I.aMA2-7BB-call and projected the resulting vectors with t-SNE. As shown in Figure 4 the model organizes these representations into clearly separated geometric clusters: in- struction embeddinggs are well-separated from execution forms, and word-form results cluster more closely with symbolic executions than with instructions. This visualizes the structural dissociation between verbal- ized procedures and learned execution templates the model's latent space.
We computed the mean embedding (centroid) for each (operation, role) cluster based on 50 examples per category. We then calculated pairwise cosine distances between these centroids and visualized the result as heatmap (Figure 5). The cluster labels are grouped first by role (instruction, execution, worded execution) and then by operation. The plot reveals a clear block structure: instructional forms form one cluster with tight internal cohesion but are distant from both executional forms. In contrast, execution and worded execution are notably closer to one another, reflecting their shared emphasis on output form rather than procedural structure.
Inference- Time Dissociation. At inference time, different prompts steer the model toward geometrically distinet subspaces in its representational space. When prompted for algorithmic explanation (e.g. "How do you multiply two numbers?"). the model gravitates toward the instructional subepace, retrieving pedagogical patterns from its training distribution. When prompted for calculation (c.g., "What 56 76?"), the model seeks out the execution subspace, accessing stored arithmetic patterns.
These operate as independent systems developed through separate pattern-matchin processes during train- ing, explaining why modols can fluently articulate multiplication procodures while failing to execute them reliably. The geometric separation we observed in Figure manifests functionally as this prompt-dependent subspace selection, where the model's response pathway is determined by which representational region the prompt activates rather than any principled binding between instruction and execution.
With all three architectural constraints now impossibility of direct symbolic operations, and geometric separation of instruction from execution we can see how they interact to create the computational split-brain syndrome. No single constraint alone would be fatal, but together they systematically prevent LLMs from bridging the gap between comprehension and competence.
symbolic representations, computational
3.4 Real-World Manifestation: Structured Data Operations
The architectural constraints identify manifest concretely in real-world applications where organizations atlempt to use LLMs for data analysis lasks. Consider seemingly simple query: finding the maximum sales value across branches from CSV table embedded in a prompt. This operation, trivial for any database system, requires:
Sequential iteration through records
Maintaining comparison state across rows
Tracking both the maximum value and its associated entity
Consistent numerical comparison operations
15

---

Published in Transactions on Machine Learning Research (10/2025)
Inter-Cluster Cosine Distance Heatmap (Grouped by Role)
add Instruction
0.7
compare, comparc_instruttion Instruction
div instruction
0.6
Instruction
sub insstrtuuutinn
add_execudee execution
compare execution
Diztanco
004
div_esecttion
execution
sub evecution
edd execution word
0.2
compare www.uteionnnnoeeeee
diy ewoeecutioon_wod
mult execution word
execcutoo_worr
0.0
add instaccion
compare Instraction
eiu. instruction
mult instruction
Instruction
add eerecutiio
compare evecurion
muit execution
ovecuttion
compare execution word
mult exeecution wordd
sab, exerution, worr
add, exerutionn
Figure 5: Cosine distances between cluster centroids for each operation-role pair, grouped by role: in- structional texts (top block), execution strings (middle), and worded executions (bottom). Each centroid represents the average embedding of 50 samples per category, computed using LLaMA2-7B metric separation is observed between instructional and exccution-related forms, with execution and worded execution clusters being closer to each other than either is instruction. This provides quantitative support for the hypothesis that instruction following and execution are learned distinct representational pathways within the model's embedding space.
Clear geo-
Recent empirical work by Wolff and Hulsebos (2025
provides direct evidence of these failures. Their eval- uation of LLMs analytical aggregations including operations fundamental as computing averages, "significant deficits in tabular reasoning performance." These multi-
sums, and maximum step algorithms exemplify preciscly the symbolic operations our three architectural constraints predict to be impossible through pattern completion alone.
The failure systematic: MAX requires maintaining comparison state across iterations, AVG demands ac- cumulating sums while counting elements, and even COUNT nccessitates consistent increment operations. Each violates our identified constraints: contextual averaging (Claim 1) prevents stable numerical compari- son, FFNs cannot implement exact. iterative operations (Claim 2), and the instruction-exeeution disconnect
16

---

Published in Transactions on Machine Learning Research (10/2025)
(Claim 3) means that even when models can describe these algorithms perfectly, they cannot execute them reliably.
This has immediate practical implications. This has immediate practical implications. The inevitable failures are not due to inadequate training on tabular data or poor prompt engineering they reflect fundamental architectural impossibilities that practitioners do not expect. As Wolff and Hulsebos (2025) demonstrate, performance degrades further with realistic data characteristics like missing values and duplicates, confirming these are architectural rather than training limitations.
3.5 The Broader Pattern: From Architecture to Cognition
With all three escape routes blocked unstable symbolic representations, architectural impossibility of direct computation, and geometric separation of instructional knowledge from exccution-LILM. inevitably resort to sophisticated paltern storage and retrieval systems. This explains the computational split-brain syndrome: models become excellent tutors who can articulate multiplication procedures (through instructional pattern- matching from Claim 3) while executing them through brittle hierarchical pattern retrieval rather than principled computation.
Contrast with Symbolic Systems. Traditional symbolic computation systems avoid these limitations through explicit design choices that transformer architectures lack:
Type Systems: Variables are bound to specific domains (e.g., a E R) with operations defined over those types, preventing contextual contamination.
Algorithmic Implementation: Operations like multiplication are implemented as explicit proce- dures rather than learned approximations.
Compositional Binding: Complex expressions maintain consistent variable bindings through syn- tactic structure.
Systematic vs. Pattern-Based Approximation. Even when traditional systems use approximation, as in floating-point arithmetic, they maintain systematic behavior with guaranteed error bounds (IEEE 754). Each operation follows precise rules with predictable maximum errors. LLMs, by contrast, exhibit unsystematic pattern-based failures: correctly computing most arithmetic while failing unpredictably on specific cases, with no principled error bounds. For queries involving arithmetic operations, LLMs will therefore always encounter caat-o-ddisributtoon problems left to their own devices. This distinction highlights the fundamental difference between engineered approximation algorithms and emergent pattern retrieval.
Path- Bath-Dependent Fragmentation and Interpretability Challenges. The residual fitting process is highly sensitive to training dynamics across diverse domains, suggesting why mechanistic interpretability findings often lack generalizability. Since arithmetic represents just one of many competing objectives, the order and context of examples during training significantly impact which pattern clusters emerge and where they are encoded. Training data ordering, scheduling decisions, and model architecture jointly determine which statistical regularities crystallize into seemingly coherent "circuits."
Recent detailed mechanistic analysis confirms this training-dependenccy hypothesis. Nikankin et al. (2024) provide valuable evidence that arithmetic computation relies on distributed "bags of heuristics" spread across multiple layers, rather than implementing dedicated algorithmic circuits for basic operations (+, x, +). Their neuron-level analysis reveals that coherent computational circuits exist for these fundamental operations. Importantly, they find that models within the same training lincage (Llama3-8B vs Llama3-70B) share more similar heuristic patterns than models from different. training backgrounds (Pythia-6.9B) GPT-J), yet even within this shared lineage, the models differ in the degree and soplistication of these patterns. This draendngyy gradient- where shared training yiclds similar but not identical mechanisms conffrms that apparent "arithmetic mechanisms" reflect training-specific statistical regularities rather than universal computational principles.
17

---

Published in Transactions on Machine Learning Research (10/2025)
Reversing Dennett: Comprehension Without Competence. This pattern inverts what philosopher Daniel Dennett observed in natural systems, where competence typically precedes comprehension (Dennett, 2017). Simple organisms demonstrate complex behaviors before developing explicit understanding, LLMs exhibit the reverse: sophisticated explanatory capabilities coupled with unreliable execution comprehension without competence.
This reversal reveals LLMs as fundamentally different from both biological intelligence and traditional compu- tational systems. They excel at linguistic pattern completion while generally lacking the grounded symbolic manipulation capabilities that such fluent descriptions would suggest.
These representational, computational, and learning constraints reveal why LLMs function as sophisticated pallern completion engines rather than symbolic reasoners. The computations split-brain syndrome emerges as a fundamental property of transformer architectures optimized for pattern completion rather than symbolic manipulation.
Testable Predictions. Our framework makes specific predictions that can be empirically validated:
1. Models trained exclusively on arithmetic should show different embedding geometry than general- purpose models, with more isometric numerical relationships
Pattern assembly signaturcs (gradual layer-by-layer convergence) should appear across all trans- former model families
Instruction-cxocutine scparation should persist across architectures, including instruction-tuned and reasoning models
Novel format generalization should consistently fail while familiar format variations succeed (the key discriminator between pattern completion and algorithmic approximation)
These predictions provide roadmap for future quantitative validation of our claims.
4 From Arithmetic to Relational Reasoning
If our architectural analysis is correct, the computational split-brain syndrome should not be limited to arith- metic. The same three constraints mmsttrainss-contestuull averaging, architectural impossibility exact computation, and instruction-eexection disconnect- should create identical failure patterns wherever systematic symbolic manipulation required.
Relational reasoning provides the perfect test case. Like arithmetic, it demands automatic binding from natural language to symbolic structure, followed by systematic execution of transformations. Just as *9.11" must be bound as numerical quantity rather than a historical date, "Alice" must be bound to appropriate relational roles before applying logical transformations like Væ, [Parent(x, y) + Child(y, x)).
Table 1: Two-Phase Computational Requirements. Both arithmetic and relational reasoning require auto- matic binding from natural language followed by systematic execution of transformations. Phase Input Binding
Arithmetic Operations
Relational Reasoning
Automatically infer and bind: "9.11" + numerical value, "9.9" numerical value
Automatically infer and bind: "Alice" + PERSON, family relationships from syn- tax
Computational Exe-
Execute bound operations: 9.9 > 9.11
Execute inference rules: Parent Y) Child(Y,
cution
Consistency
Maintain across multi-step procedures:
Maintain variable bindings multi-step inference: Parent(A, B)A Parent(B, C*) - Grandparent(AA C)
across
a> a>bab>c-ad
Table Iilustrates these parallel computational demands across domains. Both require the same fundamental operations: automatic symbolic binding from natural language, reliable rule execution, and consistency
18

---

Published in Transactions on Machine Learning Research (10/2025)
maintenance actoss multi-step reasoning chains. The same three architectural constraints predict systematic failures in relational reasoning: contextual averaging prevents clean symbolic binding, instruction-exxectioon disconnect separates rule articulation from application, and pattern storage leads to memorized templates rather than systematic variable manipulation.
This theoretical extension predicts LLMs will exhibit identical split-brain syndrome across domains fluent explanation of logical principles coupled with unreliable execution. The evidence strongly supports this prediction.
4.1 The Reversal Curse and Alice Problem
Consider two cases where LLMs fail at drawing basic conclusions that appear deceptively simple. The Reversal Curse (Berglund et al. 2023 tests models their ability to perform bidirectional inference: models trained only on statements like "Tom Cruisc's mother is Mary Lee Pfeiffer" are tested on the reverse direction "Who is Mary Lee Pfeiffer's son?" symmetric transformation humans perform unconsciously. The Alice problem (Nezhurina et al., 2024 presents a different challenge: models given " "Alice has sisters and 1 brother" must answer "How many sisters does Alice's brother have?" This requires rebinding Alice from her own perspective to her brother's perspective while maintaining compositional consistency across lamily relationships.
Empirical Failures The magnitude of these failures is striking. Even after fine-tuning on fictional state- ments like "Uriah Hawthorne is the composer of Abyssal Melodies," models cannol answer "Who composed Abyssal Melodies?" revealing the same automatic binding failures we identified in arithmetic domains. Ex- periments show that models fine-tuned on 1,000 fictional "A 3" statements achieved near-perfect accuracy on forward retrieval but only 7% accuracy on reverse queries (Berglund et al., 2023). This failure persists in real-world scenarios: when tested on celebrity facts, GPT-4 correctly answers forward questions like "Who is Tom Cruise's mother?" 79% of the time, compared to only 33% for reverse questions like "Who is Mary Lee Pfeiffer's son?" (Berglund et al. 2023 The model's log-probability for correct answers in the reverse direction shows no improvement over random baseline, indicating systematic rather than accidental failure.
The Alice problem (Nezhurina et al.
2024 presents equally dramatic breakdowns. Given the scomingly simple prompt. "Alice has 4 sisters and brother" and asked "How many sisters does Alice's brother have?", state-of-the-art models including GPT-4, GPT-4o, and Claude Opus show "strong collapse of reasoning"
across most tested models (Nezhurina et al., 2024 Despite the straightforward answer (5: Alice plus her 4 sisters), comprehensive testing across multiple prompt variations and at least 30 trials per model revealed frequent failures and extreme performance fluctuations on trivial problem variations. When confronted with more complex family structures (AIW+), performance collapsed to "close to 0" even for the most capable models (Nezhurina et al.] 2024). Notably, models cxhibit strong overconfidence in wrong solutions while providing "comlabulation-like like" explanations that sound plausible but demonstrate fundamental failures in compositional reasoning and variable binding.
Computational Split-Brain in Relational Reasoning These failures exemplify computational split- brain syndrome in relational reasoning, directly predicted by our architectural analysis in Section B Models demonstrably know both generic rules (*If X is Y's mother and Y is male, then Y is X's son") and specific facts ("Tom Cruise's mother is Mary Lee Pfeiffer"), yet systenatically fail to execute the algorithmic steps required for inference. This reflects the same instruction-execction disconnect identified in Section 3.3 where procedural knowledge and computational implementation occupy gcometrically separated representational spaces.
Consider the computational requirements for the reversal curse: given "Tom Cruise's mother is Mary Lee Pfeiffer" and asked "Who is Mary Lee Pfciffer's son?", models must automatically infer that the tokens represent PERSON entities, bind the implicit MOTHER OF relationship from surface syntax, recognize the query seeks the inverse relationship, apply the symmetric transformation Mother(x, y) + Son(y, x) in reverse, bind variables x =Mary Loc, y =Tom, and conclude Tom is Mary Loe's son. Similarly, the Alice problem requires binding family structure from "has sisters and brother," " then rebinding Alice from SISTER role to SIBLING role to execute the perspective-shift computation.
19

---

Published in Transactions on Machine Learning Research (10/2025)
Both tasks demand the symbolic manipulation capabilities that our architectural analysis shows cur- rent LLMs lack: contextual averaging prevents the clean symbolic binding required for role assignment, cnstruction-xecutiin disconnect separates relational rule articulation from application, and pattern storage leads to memorized relationship templates rather than systematic variable manipulation.
Instead, models resort to directional pattern memorization. The reversal curse occurs because "A is B" patterns vastly outnumber "B is A" patterns in training corpora, causing asymmetric learning of funda- mentally symmetric relationships. The same architectural constraints that prevent stable arithmetic circuits also undermine the variable binding and systematic transformation required for relational reasoning, creat- ing the identical split-brain syndrome across domains- explanation of logical principles coupled with unreliable execution.
We note that the reversal curse cannot be addressed through training data frequency balancing simply ensuring equal exposure to "A is B" and "B A" patterns. The issue architoctural: models learn separate representational pathways for each synlactic form rather than understanding the symmetric logical relation- ship. Even with perfect frequency balance, the system would still develop distinct pattern- pattern-matching rules instead of unified bidirectional binding. Solutions like LoCo-LM (Calanzone et al.) 2025) succeed precisely because they bypass this limitation through explicit logical constraint enforcement in the training objec- tive, essentially forcing systematic truth table coverage rather than relying on natural pattern completion tendencies.
4.2 Rule Applications, Logical Inconsistencies and Operator Failures
In contrast to the implicit complexity of the Reversal Curse and Alice problem, logical inconsistency lasks present LLMs with explicit formal structure that should simplify reasoning. These problems provide clear premises, stated rules, and well-defined logical relationships LLMs continue to fail systematically. This failure despite explicit scaffolding reveals that the architectural constraints operate independently of how logical structure is presented.
Large language models exhibit a range of logical inconsistencies, as identified LoCo-LM (Calanzone et al. 2025 and summarized by Cheng al. (2025):
Negation Inconsistency: The model affirms both "X is an organism" and "X not an organism."
Implication Failure: Given "All birds are animals" and " "An albatross is a bird," " the model fails to infer "An albatross is an animal."
Reverse Implication Failure: From "If made of metal, then conducts electricity" and "X does not conduct electricity," " the model fails infer "X not made of metal."
Deductive Chain Breakdown: Given "Nails are made of iron," " "Iron is a metal," and "Metals conduct electricity," the model fails to infer "Nails conduct electricity."
These failures persist even when facts and rules are fully spelled out (e.g., "All birds can fly" and "Tweety is bird" - "Tweety fly")-a setup that, unlike the Reversal Curse Alice problem, explicitly states the binding relationships that should enable deduction if the architecture supported it.
Vashishtha et al. (2025) demonstrates success on causal reasoning tasks that test rule applications without variable binding. Pattern matching works when models are trained from scratch on clean synthotic data, with a 67M-parameter transformer outperforming billion-parameter LLMs on causal axioms. This confirms that direct rule application falls within transformers' capabilities when training conditions isolate logical operations from the contextual contamination of multi-domain pretraining.
Systematic Evaluation of Relational Reasoning The comprchensive study by Li al. (2024a) pro- vides decisive evidence of LLMs' architectural limitations in relational reasoning. They systematically eval- uated state-of4the-art LLMSs on inductive logic programming (TIP)) tasks across both natural language and
20

---

Published in Transactions on Machine Learning Research (10/2025)
truth-value matrix formats, comparing performance against Differentiable Logic Machines (DLM) Zimmer et al.
2023
specialized neural program induction models designed explicitly for logical reasoning.
The results are striking: despite being 100,000 times larger than DLM models, LLMs performed dramatically worse across all logical reasoning tasks. For family tree reasoning tasks requiring multi-step inference:
Simple relations (HasFather): LLMs achieved 47-100% accuracy vs. DLM's perfect 100%
Complex relations (IsUncle): LLMs dropped to 0-49% accuracy vs. DLM's 85%
Hierarchical relations (IsMGUncle): LLMs achieved only 0-48% vs. DLM's 55%
Critically, these failures occurred across both natural language prompting and truth-value matrix repre- sentations. When given explicit logical structure through truth-value matrices- the same format used by specialized logical reasoning systems-LLMS still failed systematically. This definitively rules out training data scarcity or prompt engineering as explanations.
Hallucination in Logical Reasoning The study revealed systematic hallucination patterns that illu- minate the underlying architectural problems. LLMs generated contradictory reasoning processes, such claiming "P8 is the father of P3 and also the mother of P3" when given explicit family relationships. These aren't random errors but systematic pattern completion mistakes where models correctly identify surface facts but fail at the transformations required for compositional reasoning.
4.3 The Pattern Across Relational Reasoning
Our analysis reveals convergent pattern across complexity levels: LLMs fail systematically regardless of how relational reasoning lasks are presented. Problems that appear simple (Reversal Curse, Alice problem) actually require sophisticated computational transformations that LLMs cannot execute. Problems that appear complex but provide explicit logical structure (truth-valuc matrices, clear logical relationships) should be easier to solve, yet. LLMs still fail systematically. Table 2 summarizes the magnitude of these systematic
failures (data. from (Berglund et al.) 2023 Nezhurina et al. 2024 Li et al. 2024a)), revealing consistently poor performance despite the apparent simplicity of many tasks.
Table 2: Systematic Failures in Relational Reasoning
Task Domain
LLM Performance
Comparison
Reversal Curse (forward)
79%
Reversal Curse (reverse)
7%
Random baseline
Alice Problem (complex)
0%
Family Relations (IsUncle)
0-49%
DLM: 85%
Family Relations (IsMGUncle)
0-48%
DLM: 55%
This convergence is theoretically significant. If architectural limitations were merely about training data scarcity or prompt engineering, we would expect LLMs to succeed when given explicit logical structure. Instead, failure persists regardless of presentation format, confirming that the constraints operate at the ar- chitectural level-thhe the same three limitations that prevent stable arithmetic circuits also undermine relational reasoning across all contexts.
Lifted Reasoning the Missing Capability The consistent failure across both implicit and explicit reasoning tasks points to fundamental missing capability: lifted reasoning. What these tasks demand- and what LLMs fundamentally lack-is the ability to apply general rules over arbitrary entities rather than memorizing instance-specific patterns. Lifted reasoning underlies human-like generalization in logic, mathematics, and programming. Unlike pattern completion, which operates over specific instances, lifted reasoning requires variable abstraction (binding entities to abstract roles that can be systematically manipu- lated), rule generalization (applying logical transformations that work across arbitrary entity instantiations), and compositional consistency (maintaining variable bindings across multi-step inference chains).
21

---

Published in Transactions on Machine Learning Research (10/2025)
Current transformer architectures lack the representational and computational mechanisms necessary for these operations. Models specifically designed for these capabilities, like Neural Logic Machines (NLM) (Dong et al. 2019) and Differentiable Logic Machines (DLM) (Zimmer et al. 2023), achieve lifted reasoning through explicit tensor representations for predicates and specialized circuits for logical operations. These architec- tures explicitly address the binding problem by representing relationships as lensors and implementing differentiable operations that maintain variable consistency- architocturrl innovations absent from standard transformers.
The performance gap persists even when LLMs are given identical inputs to specialized models: 100,000 times larger than DLM models, LLMs performed dramatically worse across all logical reasoning tasks (Li et al. 2024a), demonstrating that architectural constraints, not data availability, prevent reliable logical reasoning. This provides definitive evidence that the failures reflect fundamental limitations in how transformers process symbolic relationships, confirming our architectural analysis from Section g
despite being
This lifted reasoning gap explains why LLMs exhibit sophisticated pattern completion capabilities while systematically failing at the variable binding and rule application required for reliable relational reasoning. The computational split-brain syndrome manifests regardless of how logical structure is presented, confirming that the constraints operate at the architectural rather than training or prompting level. Understanding this fundamental limitation clarifies both what current LLMs can achieve as pattern completion engines and what alternative approaches might be needed to transcend these constraints.
5 Compensatory Strategies and Their Execution Gap
The computational split-brain syndrome reveals a fundamental paradox: LLMs can articulate principles they cannot reliably execute. However, this very comprehension capability suggests compensatory strategies. If models have memorized algorithmic procedures well, they can leverage these memorized patterns to unroll step-by-step solutions (self-ccafoolding. specialized external systems (Lool delegation). Finally, rather than working around execution limitations, we can address them architocturally by integrating dedicated symbolic modules (hybrid architectures); es- sentially an internal tool-calling for critical operations.
they can recognize problem types reliably, they can delegate to
Each approach leverages LLMs' strengths while addressing execution limitations differently. Yet we demonstrate, all three strategies converge on the same fundamental requirement: coordinating these ap- proaches demands reliable metacoguitive capabilities that current architect ures lack (see Section 5.4)
5.1 Self-Scaffolding: Leveraging Comprehension for Step-by-Step Execution
Self-scaffolding exploits L.MSs' demonstrated ability to articulate memorized algorithmic procedures by hav- ing them generate explicit step-by-step decompositions, then attempt to exccute each step themselves. This approach converts implicit reasoning into structured text patterns that leverage models' pattern completion strengths.
Examples Across Domains. A self-scaffolding approach to the problems of multi-step arithmetic, rever- sal curse, and Alice problem would generate:
"To multiply 742 x 89: First, multiply 742 6678. Then multiply 742 80 593600 Finally, add 6678 59360 66038."
"Since Tom Cruise's mother Mary Lee Pfciffer, and the relationship 'mother of sym- metric to 'son of', Mary Lee Pfeiffer's son must be Tom Cruise."
"Alice has 4 sisters and brother. From her brother's perspective, he has Alice plus her 4 sisters, making 5 sisters total."
The Illusion of Thinking study (Shojace et al. 2025 is a recent example, demonstrating this pattern across planning puzzles, where Large Reasoning Models (IRMss) including Claude 3.7 Sonnet, DeepSeek-R1, and
22

---

gap Published in Transactions on Machine Learning Research (10/2025)
OpenAl's o3-mini generate detailed algorithmic traces for problems like Tower of Hanoi while attempting execution.
5.2 Tool Delegation: Bypassing Model Execution Entirely
Tool delegation represents fundamentally different strategy: rather than improving the model's execution capabilities, it bypasses them entirely by delegating computational tasks to specialized external systems. This approach has been systematized in frameworks like ReAct (aao et al. 2022), Toolformer (Schick et al, 2023 and various agent architectures that enable LLMs to interact with external APIs and computational tools.
Direct Delegation Examples. For arithmetic, instead of attempting step-by-step execution, the model simply recognizes the problem type and delegates:
"This is multiplication problem" + calculator (ibuulator.mlliily((77.,, 89).
For database queries: "Find maximum sales" + SQL: SELECT MAX FROM data.
For logical reasoning: "Check satisfiability" + SMT SMT_sslver.o wtl.ssveee.ccccccoosstteeceeeeee choccc(constrraintts))
The necessity of such delegation becomes clear when consider the structured data operations analyzed in Section 3.4 Operations such as finding maximum values computing averages require iterative state tracking that LLMs cannot perform reliably, yet these same models excel at recognizing when to invoke SQL or pandas functions that implement these algorithms correctly. This pattern extends beyond data operations to other domains requiring algorithmic execution.
The comment on the Illusion of Thinking paper (Opus and Lawsen 2025) provides particularly striking example of tool delegation's effectiveness. When Tower of Hanoi problems were reformulated as: "Solve Tower of Hanoi with 15 disks. Output Lua function that prints the solution when called," models achieved high accuracy across tested systems (Claude-3.--Sonnee, Claude Opus 4, OpenAT o3, Google Gemini 2.5), completing in under 5,000 tokens. This dramatic improvement occurred because models generated algorith- mic code rather than attempting to cnumerate moves themselves. Similarly, Cheng et al. (2024) demonstrato that separating symbolic code generation from symbolic code execution improves deductive reasoning.
5.3 Hybrid Architectures: Specialized Modules for Symbolic Operations
Hybrid architectures represent a more fundamental response to the computational split-brain syndrome: ad- dressing the limitations architecturally rather than through workarounds. These approaches integrate spe- cialized computational modules within LLM systems handle symbolic operations that current transformer architectures cannot reliably implement. Essentially, the idea is mend the internal to the model.
Domain-Specific Integration. Several promising approaches have emerged across different symbolic do- mains. For arithmetic computation, OccamL.LM (Dugan et al. 2024) and Internal General Computations (IGC) (Dictz and Klakow 2024) incorporate specialized circuits that directly implement arithmetic op- erations, achieving perfect accuracy while avoiding paltern-based approxinations. For logical reasoning, Logic-LM (Pan et al.) 2023) integrates LLMs with symbolic solvers through translation-exxectioon pipelines that maintain logical consistency.
Particularly encouraging are approaches like Differentiable Logic Machines (DLM) Zimmer el al. 2023 which represent predicates as tensors and implement logic operations as differentiable functions. Unlike approaches that use neuro-symbolic losses improve consistency in existing architectures, DLMs provide fundamental architectural support for rule learning and symbolic generalization, directly addressing the induction barriers we identified.
23

---

Published in Transactions on Machine Learning Research (10/2025)
Functional Specialization Architectures. Such integration, if done well, would combine pattern- matching flexibility with principled symbolic operations. Rather than emulating symbolic computation through pattern completion, such systems would implement dedicated mechanisms for variable binding, rule application, and systematic inference.
Current Mixture-of-Fxperts (MoF) architectures (Shazeer et al. 2017 optimize primarily for computational efficiency routing tokens to different parameter sets increase model capacity rather than functional spe- cialization based on cognitive capabilities. A heuroscienco-inspired approach could allocate specific experts to handle symbolic operations, relational reasoning, pattern completion, and other specialized functions. Just as the human brain has dedicated regions for exccutive reasoning (prefrontal cortex) and memory (temporal lobe) (Kandel et al., 2021), functionally partitioned MoE could develop experts that excel at exactly the feed-forward networks struggle
symbolic binding and manipulation operations that current to implement.
5.4 Coordination and The Metacognitive Bottleneck
Complementary Strengths and Shared Limitations. The three compensatory strategies form a com- plementary toolkit. for addressing the computational split-brain syndrome, each leveraging I.LMs' pattern completion strengths while circumventing execution limitations in different ways. Self-scaffolding requires external components or architectural modifications, working entirely within existing capabilities by con- verting implicit reasoning into explicit textual decompositions. It excels in peddium-complexity scenarios where individual steps fall within reliable pattern storage, as demonstrated by Large Reasoning Models successfully solving problems like the Alice puzzle where base LLMs fail (Mitchel] 2025). Tool delegation converts architectural impossibilities into architectural strengths, achieving perfect reliability by offloading computation to dodicated symbolic processors. When appropriate tools exist, this approach works reliably across complexity levels for well-defned domains while minimizing computational overhead.
However, each strategy faces critical limitations that reveal deeper architectural constraint. Self-scaffolding incurs massive computational overhead (multiplying two 10-digit numbers requires approximately 102 FLOPs on CPU but 20 FLOPs in transformer inference), suffers from planning failures when decomposition relies on unreliable pattern completion, and encounters execution unreliability when individual steps require symbolic manipulation beyond pattern storage capabilities (Shojace et al. 2025). Tool delegation faces problem recognition requirements (identifying when and which tools to use), tool availability constraints (limited to well- dell-defi domains with existing specialized systems), and composition challenges when complex problems require coordination between multiple tools combinations of delegation and direct reasoning.
While hybrid architectures offer the most principled solution by mending the comprehension--omppeeer gap internally through dedicated symbolic processing capabilities, they face fundamentally similar limitations to external tool delegation. Just as external tool calling requires recognizing when and which tools to use, hybrid architectures must coordinate between symbolic and pattern completion modules essentially internal tool routing decisions. This introduces integration complexity, neural adaptation requirements that may constrain symbolic expressiveness, and the same metacognitive challenge of determining when symbolic processing versus pattern completion is appropriate. Moreover, the internal nature of this coordination makes it potentially harder to monitor, debug, or override compared to external tool delegation.
The Metacognitive Dependency. Most critically, all three approaches converge on the same fundamen- tal requirement: they demand reliable metacognitive assessment coordinate when and how to apply each strategy. Self-scaffolding requires knowing when decomposition will help versus hurt, tool delegation requires recognizing appropriate problem types and available tools, and hybrid architectures must coordinate between symbolic and pattern completion modules. This shared dependency exposes recursive problem: the com- pensatory strategies themselves require the very introspective abilities that the computational split-brain syndrome prevents.
However, compensatory strategics require careful coordination and expose deeper introspection require- ments. Consider an idcalized metacognitive control loop (Algorithm D as an illustrative framework: while this process could theoretically be executed through pattern completion L.MSs can perform individual steps
24

---

Published in Transactions on Machine Learning Research (10/2025)
Algorithm 1 Idealized Metacoguitive Control Loop
P+ 4- imput problem
d ASSESS SYSESS_DIFFICULYYY
(requircs introspection)
3: if d S then
4: return PATTERN "NTTERN_COMPLEE(("
5: clse
6: +' DECOMPOSE_ OR ) {tool call planner)
7: return SOIVE(P))
end if
like tool calling and decomposition using their existing capabilities (e.g., GPT-4's tool-calling demonstra- tions) it reveals fundamental challenges in metacognitive assessment. The difficulty assessment in line 2 and decomposition choice in line would require metacoguitive capabilities that face inherent limitations:
Complexity assessment (line 2): "Is this problem computationally hard for my specific architecture?" (But how can the model assess difficulty without attempting the task?)
Capability introspection (line 2): "Can I execute this operation reliably through my pattern storage?"
Tool appropriateness (line 6): "Do have the right external system for this problem type?"
Decomposition quality (line 6): "Will breaking this into sub-steps actually help?" (The decom- position process itself can be brittle, suggested by reasoning model studies in (Shojace et. al. 2025
Step reliability (line 6): "Can trust each sub-computation in my scaffolding?"
The deeper issue metacognitive: while commercial LLMs have developed sophisticated capabilities for recognizing when to delegate computational tasks to external tools, they lack reliable self-awareness of their internal computational limitations when such delegation is unavailable. The same instruction-exeeutie disconnect that prevents reliable symbolic computation also creates fundamental challenges for model self- assessment of their internal pattern completion capabilities. Recent work has identified these metacogni- tive limitations across LLMs, with systematic overconfidence and poor calibration, particularly on out-of- distribution lasks (Geng al. 2024 Kadavath et al., 2022). This metacognitive deficit manifests critically high-stakes applications: studies of medical reasoning show that L.Mss "lack essential metacognition for reliable medical reasoning," consistently failing recognize knowledge limitations (Griot al. 2025).
To test the importance of metacognition, designed tightly controlled experiment using n-digit n-digit multiplication tasks with place-value decomposition and simulated column-wise addition. We tested Claude Sonnet 4, GPT-4o, and Gemini 2.5 Flash across three conditions, with results sunmarized in Table
Zero-shot direct calculation: Models were asked to calculate multiplication problems directly with explicit constraints against scaffolding, decomposition, or external tools ("Calculate this multi- plication using only your internal reasoning, without using any tools or code"). All models achieved 0% accuracy on 10-digit problems, demonstrating universal computational split-brain syndrome when coistrained to pure transformer computation. At 5-digit complexity, Claude achieved 10.5% accuracy while GPT-4o and Gemini remained 0%.
Self-generated decomposition: We tested whether models could independently generate ap- propriate decomposition prompts for multiplication problems. When asked to create step-by-step decomposition instructions, all models succeeded on 5-digit tasks. For 10-digit problems, GPT-40 produced prompts with 65% accuracy in decomposition steps and 94.2% overall quality, while Claude Sonnet 4 and Gemini 2.5 Flash maintained perfect decomposition accuracy.
Golden decomposition exccution: We tested execution using golden decomposition prompts that explicitly guide step-by-step computation using place-value breakdown. Models were constrained to
25

---

Published in Transactions on Machine Learning Research (10/2025)
use only internal capabilities without external tools self-correction. mance across two difficulty levels.
Table summarizes perfor-
Table 3: Performance on multiplication problems with golden decomposition prompts. Sum errors refers to
failures in final addition despite correct intermediate steps. Complexity
Metric
GPT-4o
Gemini 2.5
Claude Sonnet
Overall Accuracy
5% (1/20)
95% (19/20)
100% (20/20)
5-digit
Step-wise
95-100%
95-100%
100%
Sum Errors
19/20
1/20
0/20
Overall Accuracy
0% (0/21)
0% (0/20)
0% (0/20)
10-digit
Step-wise
76-100%
95- 100%00
95-100%
Sum Errors
18/21
20/20
20/20
With 5-digit numbers, GPT-4o achieved 95-100% accuracy on individual multiplication steps yet failed all but one problem due to arithmetic errors in the final summation. Gemini 2.5 Flash performed near-perferr cerr-eefectly (95% overall accuracy), while Claude Sonnet 4 achieved perfect performance. As complexity increased 10-digit numbers, all models degraded: GPT-4o's step-wise accuracy collapsed to 76 100% with systematic errors, while Claude and Gemini maintained high step accuracy (95-100%) but failed all final answers due to summation errors.
Critically, when given natural prompts without explicit constraints, these models automatically delegate multiplication problems to external computational tools, achioving perfect accuracy through sophisticated tool-routing capabilities. Our experiments reveal what happens when this learned compensatory strategy is disabled, exposing the underlying computational split-brain syndrome that automatic tool delegation normally masks.
These results reveal fundamental limitation: even with perfect algorithmic decomposition, neither individ- ual computational steps nor aggregation operations remain reliable as per-step complexity increases. Models face a computational trilemma: either (1) execute multi-step procedures internally and risk systemic break- downs, (2) rely entirely on external tools, or (3) decompose steps further-whit introduces new failure points even under perfect prompting.
Real-world applications involve countless multi-step algorithms of unknown complexity where external com- putational verification is unavailable. This necessitates reliable metacognitive assessment of internal capa- bilities: models must recognize their computational limitations and coordinate appropriate compensatory strategies without external guidance. While commercial models have learned sophisticated tool delegation for well-defined computational tasks, they lack the self-awareness needed to assess the reliability of their internal pattern completion processes.
This raises a fundamental question about the nature of metacognition itself. Large Reasoning Models and approaches like Reflexion (Shinn et al., 2023) appear to learn metacognitive skills: knowing when to reflect, when to distrust their initial judgment, when to backtrack. During training, external verifiers provide ground truth signals about when reflection is needed, allowing models to learn patterns about failure recognition. At inference time, this "metacognition" manifests as pattern association: recognizing cues that previously correlated with the need for reflection. But is this genuine self-awareness or sophisticatod pattern matching? The fundamental challenge remains that any metacognitive assessment should doubt this answer," "This reasoning seems faky"-muss itself emerge from pattern recognition over the same representational pathways we have shown to be unreliable for systematic reasoning.
Perhaps this mirrors human cognition itself sense of self-awareness may similarly emerge from pattern recognition over internal states rather than direct introspective access to our computational processes. This suggests that the distinction between "genuine" and "pattern-matched" ingful than initially apparent: if both humans and LLMs rely on learned associations to assess their own reasoning, the key question becomes not the authenticity of self-awareness, but its reliability and scope.
metacoguition may be less mean-
Summary Understanding these metacognitive requirements clarifies fundamental limitation of current approaches to the computational split-brain syndrome. While the three compensatory strategies form
26

---

Published in Transactions on Machine Learning Research (10/2025)
powerful toolkit- each leveraging LLMs' pattern completion strengths in different ways they all converge on the same recursive problem: coordinating these sophisticated workarounds requires the very introspective abilities that the split-brain syndrome prevents.
Even with perfect algorithmic decomposition, as our experiments demonstrate, execution gaps persist at the individual step level. This suggests that the instruction-execution disconnect operates at a more fundamental architectural level than compensatory strategies can address. The metacognitive bottleneck reveals why current LLM capabilities, however sophisticated, remain fundamentally constrained: they excel as pattern completion engines but lack the self-awareness needed to reliably coordinate between comprehension and competence. This clarifies both what current architectures can achieve and what limitations persist even with the most sophisticated compensatory mechanisms.
6 Pattern Completion: Strengths and Limitations
Having examined LLMs' systematic limitations in symbolic computation (Section 3 relational reasoning (Section 1 and compensatory strategies (Section 5), two critical questions emerge: How do we explain the remarkable power of LLMs as they are today? And what are the consequences of their architectural limitations? Our analysis points to pattern completion as both the source of their strengths and the root of their symbolic weaknesses.
General intelligence = sophisticated pottern completion across diverse domains, with robust knowledge retrieval and flezible task adaptation. Generalizable intelligence = systematic rule discovery and principled reasoning that can be deployed to novel lasks and drives scienlifie progress.
Definition 1.
We demonstrate that LLMs excel at the former but encounter fundamental barriers at the latter. The following analysis traces where today's LLMs succeed on general intelligence but fail on generalizable intelligence.
Multi-level Pattern Completion. LLMs achieve impressive capabilities through a hicrarchy of pattern completion mechanisms. At the most. basic level, next-token prediction enables fluent text. generation, while higher up, the same mechanism supports chain-of-thought reasoning pattern-matching operations over common reasoning structures.
Most recently, test-time computation allows these models to upgrade from LLMs to LRMs, functioning as prof-proggamming metacomputers that dynamically adjust reasoning strategies based on task complexity, Problem decomposition, exploration, Lrial-and--erificaation, and backtracking can all be viewed as higher-level patterns for task execution.
This pattern-matching approach helps to explain why LLMs excel at tasks that can be solved through pure memorization (like relrieving facts) or tasks that appear to require symbolic manipulation but can actually be solved through pattern matching (like solving standardized math problems in familiar formats).
The Performance Cliff: From Pattern Recognition to Rule Discovery. Because LLMs confiate similarity search with logical inference, recent empirical evidence from the ARC-AGT benchmark progression crystallizes the distinction between general and generalizable intelligence. ARC-AGI-I evaluates pattern recognition and single-rule application in visual reasoning tasks, requiring models to identify and apply transformations from minimal examples (Chollet, 2019). A forthcoming benchmark extension (ARC-AGI-2) advances this evaluation by specifically targeting rule discovery and systematic application: tasks require in- ferring multiple interacting rules from limited observations, applying them compositionally across sequential steps, and adapting rule application based on contextual cues F
The results reveal a fundamental performance cliff: while OpenAl's o3 achioved 87.5% on ARC-AGI-1 through sophisticated paltern completion, the same architecture scored 3% on ARC-AGI-2 CCollet et al. 2025 ARC Prize Foundation 2025). Performance plummets once pattern completion loses its near-neighhor anchors from training, despite identical instruction formats. This cliff occurs precisely where tasks transition
3 BARC- AGT-2 details from pre-release technical reports available at https://githhb. http:://ggttab.cc//ttttllel////e http:tttab..cc/fcaaaee////----
27

---

Published in Transactions on Machine Learning Research (10/2025)
from pattern recognition to genuine rule discovery and systematic reasoning exposing the architectural limitations we have identified throughout this paper.
This performance cliff exemplifies our architectural analysis: LRMs achieve improvements through sophis- ticated self-scaffolding and test-time computation (compensatory strategies), but the underlying computa- tional split-brain syndrome persists- they remain pattern completion engines that fail when tasks require genuine rule discovery and systematic reasoning beyond their memorized patterns.
Inductive Reasoning and the Limits of Generalizable Intelligence. Inductive reasoning spans spectrum of complexity, from simple function learning to discovering fundamental principles that govern entire domains. Consider the challenge of deriving an algorithmic solution from scratch. Recent work by Cheng et al. (2024) demonstrates that LLMs can excel at the simplest level learning mathematical functions like base conversion from input-output examples achieving near-perfect accuracy. At a higher level, FunSearch Ktomera-Paredes et al. 2023) shows that LLMs can discover new algorithmic construc- tions for combinatorial problems through evolutionary code search, finding novel solutions to established mathematical challenges like the cap set problem.
However, discovery of novel algorithms such solving the Tower of Hanoi problem from scratch requires a much stronger set of capabilities. The optimal algorithm that works for any number of disks n requires recognizing that moving n disks involves first moving n - 1 disks, then the largest. disk, then the n - disks again. This recursive insight, itself a meta-pattern, even more challenging than what ARC-AGI-2 tests. Recent evaluation shows that state-of-theeart reasoning models like Claude 3.7 Sonnet and OpenAT's o3-mini exhibit complete performance collapse beyond 8-9 disks, despite having sufficient. token budget to continue (Shojace et al. 2025
Scientific discovery derives compact governing principles from noisy, real-world obscrvations. Recent break- throughs demonstrate that AI can indeed discover such principles, but through architecturally specialized approaches. AT Feynman succeeds through a physics-constrained symbolic search, whereas the LLM sym- bolic regression relies on pattern-based guessing (Udrescu and Tegmark 2020; Makke and Chawla 2022). These successes require fomaan-speciallinn ors, physics constraints, and symbolic manipulation engines. It remains unclear whether pattern completion alone, however powerful, can accomplish genuine scientific discovery.
architectures with strong inductive biases, such as geometric pri-
7 Implications for Mechanistic Interpretability
Having cstablished three architectural constraints that create the computational split-brain syndrome, we now examine how our framework relates to findings from mechanistic interpretability research. While these sophisticated tools have revealed important insights about model internals, we argue that the architectural constraints we identify suggest an alternative interpretation of what these tools discover: sophisticated pattern coordination rather than algorithmic implementation. This perspective does not diminish the value of interpretability research but rather reframes what we might be learning from it.
7.1 From Activation Patching Sparse Autoencoders
Early mechanistic interpretability relied heavily on activation patching, intervening on specific neurons or attention hcads to understand their causal role in model behavior. This approach revealed that arithmetic in LLMs rely not on clean algorithmic circuits but on what Nikankin et al. (2024) characterize as a "bag of overlapping pattern detectors with no coherent computational structure.
However, activation patching faces several limitations that motivated the development of more sophisticated methods. The polysemantic nature of individual neurons makes it difficult to isolate specific compulations functions through patching alone single neurons often represent multiple concepts simultaneously. Inter- ventions that work on training-like inputs often fail on novel examples, revealing poor out-of-distribution generalization. Furthermore, our analysis suggests that the specific neurons and attention heads involved in any computation are highly path-dependent, varying significantly across training runs even when final
28

---

Published in Transactions on Machine Learning Research (10/2025)
behavior converges, making findings from activation patching potentially idiosyncratic to specific model instances.
Sparse Autoencoders (SAEs) emerged as a proposed solution to the polysemanticity problem (Bricken et al, 2023 Templeton et al. 2024 SAEs are unsupervised learning models that decompose neural network ac- tivations into sparse, interpretable features by learning an overcomplete dictionary of feature directions. By expanding activations into a higher-dimensional space with sparsity constraints, SAEs aim to identify monosemantic features that capture single, interpretable concepts rather than the mixed representa- tions found in individual neurons. These features appear more stable across models, with Wang et al. (2025) demonstrating that similar features emerge across different open-source architectures trained on comparable data.
Recent methods extend interpretability analysis from local interventions to whole-network. computation traces. Linear Computation Graphs (He et al.) 2024) build on SAE features to trace complete computational pathways across layers, while Autribution Graphs (Lindsey al. 2025a) Lrace information flow through raw model activations without requiring feature decomposition. Despite their different approaches one working through SAE features, the other through raw activations both methods reveal similar patterns of distributed, redundant computation rather than clean algorithmic circuits.
7.2 Alternative Interpretations of Interpretability Findings
The interpretability findings discussed above are typically interpreted as evidence computational mecha- nisms within LLMs. Our architectural framework, however, suggests an alternative lens through which to view these same empirical results one that emphasizes pattern coordination over algorithmic implemen- tation. This interpretation, while consistent with the data, differs from how the original authors might characterize their findings.
Consider the Indirect Object Identification (IOI) circuit identified through Linear Computation Graphs He et al. 2024). The circuit traces how models process sentences like "When Mary and John went to the store, John gave Mary a bottle," " identifying features that track entities and their relationships. Yet, our frame- work suggests these features recognize statistical patterns-name appearing early in sentences often receive objects mentioned later rather than implementing genuine reference resolution through variable binding, The original authors might interpret these circuits as genuine computational mechanisms; our framework offers an alternative interpretation where these represent sophisticated statistical pattern recognition.
The arosstarchitectur universality of features (Wang et al. 2025) initially seems to suggest discovery of fundamental computational units. However, our framework suggests this universality could refloct convergent solutions to pattern matching. Whether these represent algorit primitives or statistical patterns remains an open empirical question. Models trained on similar corpora. develop similar pattern-recognition strategies because they encounter similar statistical regularities.
This interpretation aligns with the "bag of heuristics" finding [Nikankin al. 2024 While their analysis examined neuron-level patterns rather than SAF features, our framework predicts though this remains to be empirically tested that similar limitations would apply to SAE features. The heuristics are not imple- mentation accidents that better interpretability tools could resolve into clean algorithms. They reflect the fundamental strategy available to pattern-completion architectures: coordinate multiple pattern detectors to approximate computational outcomes without implementing computational procodures. SAE features may be more stable across training runs than individual neurons they capture statistical regularities that any model trained on similar data must learn. But these stable regularities remain patterns rather than algorithms.
The apparent success of iechanistic interpretability may itself exemplily the computational split-brain syn- drome. We excel at comprehending what features correlate with recognizing that certain features activate on arithmetic, logical relations, or syntactic structures. Yet, we may be mistaking this comprehension for understanding of computational competence. A feature that activates on multiplication and whose ablation disrupts multiplication could be recognizing multiplication patterns rather than implementing multiplication.
29

---

Published in Transactions on Machine Learning Research (10/2025)
The very interpretability of these features that they correspond to human-recognizable concepts concepts-might bias us toward assuming they implement rather than recognize these concepts.
7.3 Critical Questions for Interpretability Research
Our framework suggests two methodological directions that could help distinguish between pattern-matching and algorithmic hypotheses in interpretability research:
Systematic Testing at. Scale. Current interpretability work often demonstrates features expected patterns without systematic out-of-distribution testing, Following the methodology of Nikankin et al. (2024)
and GSM-Symbolic, interpretability claims should be validated across:
Surface form variations ("3 + 5"h vs. "three plus five" vs. "add and 5")
Systematic perturbations (changing numbers, adding irrelevant clauses, reordering)
Distribution shifts that preserve computational requirements
If SAE features truly implement computational operations rather than pattern matching, they should main- lain their function across these variations. Current evidence suggests they do not features that appear lo track arithmetic operations fail when numbers appear in unexpected formats, revealing pattern recognition rather than algorithmic implementation.
Addressing Confirmation Bias. Interpretability rescarch currently suffers from methodological limi- tation: features are lypically interpreted after observing their activation patterns known examples. This post-hoc analysis invites confirmation bias-we see what we expect to sec. A more rigorous approach would:
Interpret features based solely their structure before examining behavior
Make explicit predictions about feature responses to novel inputs
Report both successful and failed predictions transparently
Test features on adversarial examples designed to separate correlation from causation
These methodological improvements would help distinguish between two fundamentally different types of findings:
1. Statistical Mechanism Discovery: Understanding how models coordinate pattern matching to produce outputs what current methods achieve effectively
2. Algorithmic Mechanism Discovery: Identifying computational procedures that generalize be- yond training distributions-wale may be architecturally impossible in current transformers
We emphasize that current interpretability research has made substantial contributions to understanding how LLMs process information. Our critique is not of the methods or findings themselves, but rather suggests a reframing of what these findings might represent given the architectural constraints we identify. By acknowledging that we study sophisticated pattern coordination rather than algorithmic implementation, interpretability rescarch can provide more accurate insights into both the capabilitios and fundamental limitations of current language models. This reframing may ultimately prove more valuable than maintaining the illusion that reverse-engineering computational mechanisms that may not exist.
30

---

Published in Transactions on Machine Learning Research (10/2025)
8 Reflections on the Research Journey
This work emerged from a multi-year investigation that began with a seemingly simple question: how do LLMs perform grade-level math problems that, while conceptually simple, require multi-step algorithmic ex- ecution? The search for generalizable arithmetic circuitry led to a deeper puzzle why should grade-school mathematics ever present an "out-of-disttibuttin"" problem for models trained on vast corpora? In paral- lel, the Reversal Curse and Alice-in- -nn-eeiladd problems revealed that automatic binding to concepts and something humans take for grantod-simply simply does not occur in LLMs. The viral "9.11 9.9" comparison, inilially dismissed as an amusing anecdole, crystallized as fundamental inpul representation problem: tokens carry contextual contamination that prevents clean symbolic binding. Gradually, we rec- ognized these weren't isolated curiosities but systematic failures spanning arithmetic, logic, and relational reasoning- -al stemming from the same architectural constraints. This journey from searching for arithmetic circuits to discovering their impossibility ultimately revealed the computational split-brain syndrome as a unifying explanation for why LLMs can explain what they cannot execute.
9
Conclusion
The apparent intelligence of LLMs belies deeper fragility. Despite their success in pattern-rich tasks, these modols consistently fail to generalize principles, execute symbolic computations, or reason reliably-ever under idealized conditions. Our analysis reveals that these failures are not incidental but structural: LLMs dissociate instruction interpretation from execution, leading to a computational split-brain syndrome that prevents the formation of robust, composable operations.
This diagnosis clarifies why interpretability tools often uncover expressive artifacts rather than reliable al- gorithms, and why model self-explanations may diverge from the actual pathways used in computation. Our finding that instruction and execution pathways are geometrically separated raises fundamental con- cerns: models' explanations of their reasoning may not reflect their actual computational processes, chal- lenging current approaches to explainability. Training order influences which internal patterns are embedded and where-even even when outward bchavior remains stable causing interpretability efforts to surface path- dependent artifacts rather than consistent mechanisms.
These insights reframe current conversations about emergence and scaling. Our findings align with critiques by [ender et al. (2021) and others who argue that the "bigger-is-bctter paradigm has fundamental flaws. By demonstrating that LLM limitations are architectural rather than scale-dependent,, we provide technical evidence that more data or parameters cannot resolve bottlenecks rooted in architectural design. This suggests redirecting resources from pure scaling toward architectural innovations could reduce computational and environmental costs while improving reliability.
These findings have immediate implications for high-stakes applications where reliable reasoning is critical. Current LLMs require careful scaffolding, external verification, hybrid architectures rather than deploy- ment as standalone reasoning systems in domains like medical diagnosis, legal analysis, or safety-critical decision making. The computational split-brain syndrome we identify suggests that apparent fluency in explaining procedures should not be mistaken for reliable execution capability.
If we seek genuinely general and generalizable intelligence, we must look beyond pattern completion engines. Future systems will need metacognitive scaffolding, lifted representations, and architectural support for principled exccution. Our findings offer not only critique of current LLMs, but foundation for building the next generation of intelligent systems: one that can reason, not just react.
Acknowledgments
This work benefited greatly from insightful discussions and valuable feodback from colleagues. Special thanks to Yasser Shaaban, David Paul Wipf, Tong He, and many students for reviewing carly drafts and offering constructive comments. Tong He is also gratefully acknowledged for assistance in refining the experiments.
31

---

Published in Transactions on Machine Learning Research (10/2025)
We thank the TMLR reviewers for their thoughtful feedback, which substantially improved the clarity and rigor of this work.
The author acknowledges the use of Al assistants (Claude, Anthropic; ChatGPT, OpenAl) for brainstorming discussions, related work synthesis, code development and debugging, and mantscript editing. All core theoretical contributions, experimental design, and analytical insights remain the author's own. Finally, the author thanks Kevin Zhang for unknowingly but instrumentally sponsoring his dad's research.
32

---

Published in Transactions on Machine Learning Research (10/2025)
References
Z. Allen-Zhu and Y. Li. Physics of language models: Part 3.1, knowledge storage and extraction. In Proceedings of the 41st International Conference on Machine Learning, ICML '24, July 2024. Full version available at lattp://axe /awxiv..xxvvv... .rgg http:/bb/230..... 14316
ARC Prize Foundation. ARC-AGI-2 leaderboard, 2025. URL /
E. M. Bender, T. Gebru, A. McMillan-Major and S. Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference Fairness, Accounlabilily, and Transparency, pages 610-623. ACM, 2021.
L. Berglund, M. Tong, M. Kaufmann, M. Balesni, A. C. Stickland, T. Korbak, and O. Evans. The reversal curse: LLMs trained on 'a is b' fail to learn "b is a". arXiv preprinl arXiv:2309. 12288, 2023. URL
https://arxivy http:://arrii..or//bb////// www.:/ogggbbbaaaeee 2309 12288
E. Brewer. Towards robust distributed systems. In Proceedings of the nineteenth annual ACM symposium on Principles of distributed computing, page 7. ACM, 2000.
T. Bricken, A. Templeton, B. Chen, C. Olah, et al. Towards monosemanticity: Decompasing language models with dictionary learning. Transformer Circwits Thread, 2023. URL http:://craanafoorrr--rrorttt
html
D. Calanzone, S. Teso, and A. Vergari. Logically consistent language models via neuro-symbolic integration.
In Proceedings of the International Conference on Learning Representations (ICLR), 2025. URL https: Poster Presentation.
//openreview net wttp:oormmidd7PGlupppa
F. Cheng, H. Li, F. Liu, R. van Rooij, K. Zhang, and Z. Lin. Empowering ilms with logical reasoning: A comprehensive survey. arXiv preprint arXi::502. 15652, 2025. URL. httts:://artit http://axxi..orgbb///////sa 15652
K. Cheng, J. Yang, H. Jiang, Z. Wang, B. Huang, R. Li, S. Li, Z. Li, Y. Gao, X. Li, B. Yin, and Y. Sun. Indue- tive or deductive? rethinking the fundamental reasoning abilitics of llms. arXiv preprint arXiv:2408.00 001144 2024.
F. Chollet. The abstraction and reasoning corpus. httpe://aathub.cc//rrhoolllll// 2019. 2024-03-15.
Accessed:
F. Chollet et al. Arc-agi-2: A new challenge for frontier ai reasoning systems. arXiv:2505.1189) 11831, 2025. URL
arXiv preprint
11831
DeepSeek-AI, D. Guo, D. Yang, H. Zhang, J. Song, et al. Deepseek-rl: Incentivizing reasoning capability in 1lms via reinforcement learning. arXiv preprint arXiv:2501. 12948, 2025. URL https::/arxiv..ot org/ abs/ 12948
D. C. Dennett. From Bacteria to Bach and Back: The Evolution of Minds. W. W. Norton & Company, 2017. ISBN 978-0393242072.
F. Dietz and D. Klakow. Ige: Integrating gated calculator into an 1lm to solve arithmetic tasks reliably and efficiently. arXiv preprint arXiw:2011.0088, 2024. URL
H. Dong, J. Mao, T. Lin, C. Wang, L. Li, and D. Zhou. Neural logic machines. In International Conference on Learning Representations (ICLR), 2019. URL
O. Dugan, D. M. Jiménez-Bened Benetó, Loh, Z. Chen, R. Dangovski, and M. Soljačié. Occamllm: Fast and exact language model arithmetic a single step. arXiv preprint ar arXiv:2406.0633 2024.
N. Dziri, X. Lu, M. Sclar, X. L. Li, L. Jiang, B. Y. Lin, P. West, C. Bhagavatula, R. Le Bras, J. D. Hwang, S. Sanyal, Wellock, X. Ren, A. Ettinger, Z. Harchaoui, and Y. Choi. Faith and fate: Limits of transformers on compositionality. In Advances in Neural Information Processing Systems, volume 36, 2023. URL https://arxiv... wtwpe/vv.rr/abbb//00000 18654
33

---

Published in Transactions on Machine Learning Research (10/2025)
J. Gao, A. Lee, P. Mathur, T. Khot, and H. Hajishirzi. AbstRaL: Abstraction--rinnoreee learning enables robust symbolic reasoning in language models. arXiv preprinl arXiv:2506. 1 12345, June 2025.
J. Geng, F. Cai, Y. Wang, H. Koeppl, P. Nakov, and 1. Gurevych. A survey of confidence estimation and calibration in large language models. In Proceedings of NAACL-HLT, pages 6577-6595, 2024.
M. Griot, C. Hemptinne, J. Vanderdonckt, and D. Yuksel. Large language models lack essential metacoguition for reliable medical reasoning. Nature Communications, 15:642, 2025. doi: (0.1088//411467-0224 URL htttp::/www.naturee.comm natura.tuccoommm com/ http://ww........rccc/e///////00
Z. He, X. Ge, Q Tang, T. Sun, Q. Cheng, and X. Qiu. Automatically identifying local and global circuits with linear computation graphs. arXiv preprint arxiv:2405 19868, 2024.
K. Hornik. Approximation capabilities of multilayer feedforward networks. Neural Networks, 4(2):251 257, 1991. doi: 10.1016 11101600993-000 (1893-60000991900000
S. Kadavath, T. Conerly, A. Askell, T. Henighan, D. Drain, E. Perez, N. Schiefer, 7. Hatfield-Dodds. N. DasSarma, E. Tran-Johnson, et al. Language models (mostly) know what they know, arXiv preprint arXiv:2207.0522 2022.
E. R. Kandel, J. D. Koester, S. H. Mack, and S. A. Siegelbaum. Principles of Newral Science. McGraw-HH Hill, 6th edition, 2021.
Z. Li, Y. Cao, X. Xu, J. Jiang, X. Liu, Y. S. Teo, S.-W. Lin, and Y. Liu. Lims for relational reasoning: How far are we? arXiv preprint XrXi:2401.00002 2024a. URL
Z. Li, H. Liu, D. Zhou, and T. Ma. Chain of thought empowers transformers to solve inherently serial problems. arXiv preprint 2rx:::2402. 12875, 2024b.
S.-W. Lin, R. Xu, X. Li, and W. Xu. Rules created by symbolic systems cannot constrain learning system. 2025. Preprint, January 30, 2025.
Available at SSRN:
J. Lindsey, T. Conerly, A. Templeton, J. Marcus, et al. Attribution graphs: Tracing thoughts through language models. Transformer Circuits Thread, 2025a. URL 2025/attribution graphs/ hnt///idexxx.e/ html
pub/
J. Lindsey, W. Gurnee, Ameisen, B. Chen, A. Pearce, N. L. Turner, C. Citro, D. Abrahams, S. Carter, B. Hosmer, J. Marcus, M. Sklar, A. Templeton, T. Bricken, C. McDougall, H. Cunningham, T. Henighan, A. Jermyn, A. Jones, A. Persic, Z. Qi, T. B. Thompson, S. Zimmerman, K. Rivoire, T. Conerly, C. Olah, and J. Batson. On the biology of a large language model. Transformer Circuits, 2025b. https:// transf cirrritircttt.... wwrrccitss.uub//// wabb/225/attribution http:///tttrrbttoooooggggga////a
html
N. Makke and S. Chawla. Interpretable scientific discovery with symbolic regression: A review. arXiv preprint arXiv:2211.008 2022.
S. McLeish, A. Bansal, A. Stein, N. Jain, J. Kirchenbauer, B. R. Bartoldson, B. Kailkhura, A. Bhatele, J. Geiping, A. Schwarzschild, and T. Goldstein. Transformers can do arithmetic with the right embeddings. arXiv preprint ar arxio:2405. 1000051.73999 17399, 2024. URL https::/aarxi..org/// wtwp:///.www.b////.....o/ 17399
T. Mikolov, K. Chen, G. Corrado, and J. Dean. Efficient estimation of word representations in vector space. In Proceedings of the International Conference Learning Represenlations (ICLR), 2013. URL https://arxiv.o httpg//b//3311...... /////1111.337 37881
I. Mirzadch, K. Alizadeh, H. Shahrokhi, O. Tuzel, S. Bengio, and M. Farajtabar. Gsm-symbolic: Understand- ing the limitations of mathematical reasoning in large language models. arXiv preprint ar arXiv:2410.05229 2024.
M. Mitchell. Artificial intelligence learns to reason. Science, 387(6740), 2025. doi: 10.1 10. 5w11
34

---

Published in Transactions on Machine Learning Research (10/2025)
M. Nezhurina, L. Cipolina-Kun, M. Cherti, and J. Jitsev. Alice in wonderland: Simple tasks showing complete reasoning breakdown in state-of-the-art large language models. arXiv preprint arXiv:2406.0280 02061, 2024. URL
wttp://gggg....ooooo// 02061
Y. Nikankin, A. Reusch, A. Mueller, and Y. Belinkov. Arithmetic without algorithms: Language models solve math with a bag of heuristics. arXiv preprinl 2rXiu2240022127. 2024.
OpenAL Introducing openai ol. mntrodtuiing-ppnaa--ll--eevvee
OpenAI Official
Blog, 2024.
htzps://openai.com// con/inexx/
C. Opuss and A. Lawsen. Comment on The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity. arXiv preprint arXiv:2506. 002200 2025. URL
https://arxxa http///xiiiv.. www.aawwwwww..aaaaaa
www.oogg//aaaoaeoooe//..oo..aeaa aaaaabba//255 2506. 0500/0920600000
L. Pan, A. Albalak, X. Wang, and W. Y. Wang. LOGIC-LM: Empowering large language models with symbolic solvers for faithful logical reasoning, In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 3806-3824, 2023. URL
wwg./023.......oooooo foeee.findiigg fiidingnneenninne 248/
B. Peng, S. Narayanan, and C. Papadimitriou. On limitations of the transformer architecture. Xi preprinl arXiv:2402.00 08164, 2024.
A. Plaat, A. Wong, S. Verberne, J. Broekens, N. van Stein, and T. Back. Reasoning with large language models, survey, arXiv preprint 2011:2407.11511,, 2024.
A. Power, Y. Burda, H. Edwards, I. Babuschkin, and V. Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets. Xix preprinl arXiv:2201.00 022177, 2022.
B. Romera-Paredess M. Barekatain, A. Novikov, M. Balog, M. P. Kumar, E. Dupont, F. J. Ruiz, S. Ellenberg, P. Wang, O. Fawzi, al. Mathematical discoveries from program search with large language models. Nature, :4995)95):468 475, 2023.
T. Schick, J. Dwivedi- Yu, R. Dessi, R. Railcanu, M. Lomeli, L. Zettlemoyer, N. Cancedda, and T. Scialom. Toolformer: Language models can teach themselves to use lools. arXiv preprint ar orXiv:2302.04761 2023.
Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint wex:::1101.0066 2017.
N. Shinn, A. Labash, G. Cottrell, and V. Srikumar. Reflexion: Language agents with verbal reinforcement learning and memory. In Advances in Neural Information Processing Systems 36 (NeurIPS), 2023.
P. Shojaee, I. Mirzadeh, K. Alizadeh, M. Horton, S. Bengio, and M. Farajtabar. The illusion of thinking; Understanding the strengths and limitations of reasoning models via the lens of problem complexity. arXiv preprint Xiv:25001. 12948, Jun 2025. URL https://arxiv. http::///xxxvvoog//eaeee wwwg//a/////......////////// 06941
Simons Institute for the Theory of Computing. Transformers as a computational model, September 2024. URL hattps://simns: berkeley www.eeeey.edu/oreeeeeeeeeeeeeeee teteehoppssrraasffoeee--e-ee shop held at the Simons Institute, UC Berkeley, September 23-27, 2024.
modeel Work-
S. Srivastava, M A. PV, S. Menon, A. Sukumar, A. Samod T, A. Philipose, S. Prince, and S. Thomas. Functional benchmarks for robust evaluation of reasoning performance, and the reasoning gap. ar preprinl Xiv:224002. 19450, 2024. URL htLLsttps:://// htttps:///ariii a2gg2
L. Strobl, W. Merrill, G. Weiss, D. Chiang, and D. Angluin. What formal languages can transformers express? a survey. Transactions of the Association for Computational Linguistics, 12:543-561 561, 2024.
A. Templeton, T. Conerly, J. Marcus, Lindscy, T. Bricken, et al. Scaling monosemanticity: Extract- ing interpretable features from claude 3 sonnet. Transformer Circuits Thread, 2024. URL https:
7/transformer--crcautt... wttp://cccccat.o......oooaaaaaaa
35

---

Published in Transactions on Machine Learning Research (10/2025)
C. Tigges, M. Hanna, Q. Yu, and S. Biderman. Llm circuit analyses are consistent across training and scale. arXiv preprint Xrx::22477 10827, 2024.
S.-M. Udrescu and M. Tegmark. Ai feynman: A physics-inspired method for symbolic regression. Science Advances, 6(16),, 2020.
A. Vashishtha, A. Kumar, A. Pandey, A. G. Roddy, K. Ahuja, V. N. Balasubramanian, and A. Sharma. Teaching transformers causal reasoning through axiomatic training. In ICLR 2025 Workshop on Reasoning and Planning for LLMs, 2025.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and 1. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017.
J. Wang, X. Ge, W. Shu, Q. Tang, Y. Zhou, Z. He, and X. Qiu. Towards universality: Studying mechanistic similarity across language model architectures. Proceedings of the International Conference on Learning Representations (ICLR), 2025.
J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, C. Ed, Q. V. Le, and D. Zhou. Chain of-thhouht prompting clicits reasoning in large language models. In Advances in Newral Information Processing Systems, volume 35, pages 14824-24833 2022.
C. Wolff and M. Hulsebos. How well do LLMs reason over tabular data, really? arXiv preprint orXiv:2505.07456 2025.
L. Wu, J. Chen, L. Zhang, Z. Wang, and M. Zhou. Knowledge or reasoning? fine-grained analysis of thouggh in large language models. arXiv preprint Xiv::2250 05432, June 2025.
X. Yang, B. Chen, and Y.-C. Tam. Arithmetic reasoning with Ilm: Prolog generation & permutation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techmologies (Volume 2: Short Papers), pages 699-710, 2024.
S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.00622 2022.
T. Ye, Z. Xu, Y. Li, and Z. Allen-Zhu. Physics of language models: Part 2.1, grade-school math and the hidden reasoning process. In Proceedings of the 13th International Conference on Learning Representations, ICLR 25, Apr. 2025. Full version available at http::///ssm..co/aaaattttttttt//
Y. Zhao, Y. Li, Y. Wang, and G. Neubig, SPOC: Spontaneous self-correction via integrated verifier training, In Proceedings of the 42nd International Conference on Machine Learning (ICML), June 2025.
M. Zimmer, X. Feng, C. Glanois, 7. Jiang, J. Zhang, P. Weng, D. Li, J. Hao, and W. Liu. Differentiable logic machines. Transactions on Machine Learning Research, July 2023. URL https://openreviee: net/ forum?id=mXffKet
N. Zucchet, J. Bornschein, S. Chan, A. Lampinen, R. Pascanu, and S. De. How do language models learn facts? dynamics, curricula and hallucinations. arXiv preprint orXiv:2503. 216766 2025.
A Theoretical Proofs of FFN Limitations
A.1 Piecewise Linearity of Feed-Forward Networks
Thcorem 1 (FFN Piccewise Lincarity). For any choice of weights and biases, the function implemented by a ferd-forward network with ReLU aclivation piecewise linear.
Proof- Consider the FFN computation: FFN(x:) max (0, xW, by W2 + b2
36

---

Published in Transactions on Machine Learning Research (10/2025)
1. Each neuron in the hidden layer computes h2 max(0, x wy by) where wa is the i-th column of W1. The linear function w, + by defines hyperplane in the input space.
2. Each neuron's ReLU activation partitions the imput space based x wy b, creating regions where the neuron is active (x + b; > 0) or inactive (x w; + b, S 0).
3. The hyperplanes from all neurons intersect to partition the input space into polyhedral regions, each with fixed pattern of active/inactive ncurons.
4. Within any region, the activation pattern is constant, the output is Eieactive (x-w; W2 i +2. Since this is a lincar combination of the input z, the function is lincar within each region.
5. Therefore, the overall function is piecewise linear: linear within each region, with boundaries defined by the hyperplanes.
A.2 Impossibility of Exact Multiplication
Thcorem 2 (Multiplication Impossibility). No piccewise linear function can implement eract multiplication [xx, u) ry over an unbounded domain.
Proof. Suppose, for contradiction, that some piecewise linear function g implements exact multiplication over ?
8 The multiplication function f(x, v) = xy has mixed partial derivative dwdy everywhere.
2. Any piecewise linear function has zero mixed partial derivatives within each linear region (since linear functions have constant first derivatives and zero second derivatives).
3. For to equal f everywhere, we would need linear property.
8 g Uzōg
= 1 everywhere, but this contradicts the piecewise
More constructively: consider any bounded region R where g is linear. Within R, we have g(r, y) = ax + by + a for some constants a, b, c. But ax + by + c + ay for all (x, y) E R unless R contains at most finitely many points.
Since multiplication requires unbounded curvature (the function ay becomes arbitrarily steep as |x| or |y| increases), no finite partition into linear pieces can approximate it exactly over an unbounded domain.
This impossibility extends to other non-linear operations requiring precise symbolic manipulation, establish- ing fundamental computational limits for FEN architectures in symbolic reasoning tasks.
Appendix: Role Specialization Under Token Prediction
We provide detailed proof sketch for the role specialization theorem presented in Section 3.2
37

---

Published in Transactions on Machine Learning Research (10/2025)
B.1 Setup and Notation
Consider a translormer with L layers trained on next-token prediction. We denote:
[0 0x, Op): The three parameler sets for embeddings, attention, and FFNs respectively
D: Training distribution over token sequences
( = E ((e log /x The expected cross-entropy loss
xLL) E Rd: Final layer representation
E E RlVixd Embeedingg/unmbbeeee matrix
po(wsle wee) = soltmax(2!44) FT") [wx]: Next-token probability
B.2 Gradient Flow Analysis
For arithmetic tasks like "43 78 3354", the model must minimize:
Llocal = log loo.oo((a000 (33354") 78 =')
This requires (1)) 63331 > (M) ey for all v + 3354 in the vocabulary.
Gradient w.r.t. Final Representation.
The gradient of the loss with respect to the final representation
is:
Oc ET (po Monehot)
OzlL)
where po is the predicted distribution and Youehot is the one-hot encoding of the target token.
Backpropagation Through Components. Since (L)) embeddings + El ( attn
+ E'= AN the
gradient flows to:
1. Embeddings (0x):
ac
0C (z) (L
dembeddings
00g
dembeddings 00g
But embeddings must serve multiple contexts. For token "43", the embedding must minimize loss across:
Mathematical contexts: "43 + 57", "43 2"
Date contexts: "43 BC", "1943"
Version contexts: "version 43.0"
The multi-objective optimization forces:
0% (*43") = arg min
Σ
Le(e)
rEC("43")
This averaging across contexts prevents the isometric properties needed for arithmetic (sce Sec- tion 3.1
Attention (0x): At the a_" position with causal masking, attention computes:
QQT
Al = softmax
Vdg
38

---

Published in Transactions on Machine Learning Research (10/2025)
where Q, K, V are projections of ["43", "x", "78", "=").
Key Constraint: The output is weighted average:
Abattn E conv 02, V3, m4
where e2 are the value vectors. Since 63354 & conv Cx E7g, c=), attention cannot generate the required output direction. It can only create rich encodings of the operation context.
3. FFNs (0r): FFNs must map attention's is encoding to the result. The required function is:
F: haua haua ("multiply 43 by 78") direction(esssa)
By UAT, FFNs with sufficient width approximate any continuous function compact sets. Define:
== = (hatta (a x b) (a, b) E training data)
For any e >> 0, there exist weights such that:
sup [FF(()) - F(h)lle <c
AEK
This approximation only holds on K. For novel arithmotic outside K, no approxi- mation guarantee exists.
Critical Issue:
B.3 Why Memorization Is Inevitable
Architectural Impossibility. FFNs implement FFN(z) = W2 ReeLLRL b1) + b2, which is piecewise lincar. Multiplication f(a, b) = a x not piecewise linear:
8
= 0
dadb
No weight configuration can make a piccowise lincar function compute exact multiplication (soe Theorem 2 in Appendix A
Training Dynamics. Given N training examples ((a b; C.) gradient descent optimizes:
0-
arg min BF
log ((token(c))/haata b:))
Since exact computation is impossible, the optimization converges to memorizing the mapping:
battn b.) + Ec;
This is pattern storage, not algorithmic learning. The UAT capacity that could theoretically approximate multiplication is instead allocated to memorizing these N discrete mappings.
Out-of-Distribation Failure. For novel multiplication (anew, Unow) & training:
hattn (Anow bnew) $ K
FFN has no memorized pattern
Output is interpolation between nearest memorized patterns
Result is essentially random from neighboring vocabulary tokens
39

---

Published in Transactions on Machine Learning Research (10/2025)
B.4 Conclusion
The role specialization is inevitable under gradient descent:
Embeddings must average across diverse contexts, losing arithmetic structure
2. Attention can only compute weighted averages, unable to generate novel tokens
3. FFNs cannot implement multiplication algorithmically, so they memorize patterns
This specialization emerges not from insufficient capacity FFNs have universal approximation capabil- ity but from how the next-token prediction objective allocates that capacity. The result paltern storage rather than algorithmic computation, explaining why LLMs can articulate algorithms perfectly while failing to execute them reliably.
C Metacognitive Assessment Experiments
C.1 Experimental Design
We tested three stato-of-the-art problems using "golden decomposition" prompts that provide explicil step-by-step procedural guidance. Models were constrained to use only internal capabilities without external tools or self-correction.
Sonnet 4, GPT-4o, and Gemini 2.5 Flash- on multiplication
We evaluated two complexity levels: 5-digit numbers (range 10,000-99,999) and 10-digit numbers (range 1, 1.000,000,00009.00 ...99,999999999 with 20 problems per complexity level per model.
C.2
Sample Prompt
The following shows a representative prompt structure used our experiments:
Problem: Calculate 93810 24592
Method: Use place value decomposition to break this into simpler stepa.
Step Decompose 24592 into place values: 24592 2 90 + 500 + 4000 20000
Step 2 Calculate each partial product:
Step 1: 93810 2
Step 2:
93810 90 ?
Step 93810 x 500
Step 4:
93810 x 4000 = ??
Step 93810 20000
Step 3 Add all partial products:
Sum:
?
? ?? =?
Final Answer: ?
This methodology eliminates planning uncertainty by providing the exact algorithmic steps, allowing to isolate execution limitations from decomposition failures.
C.3 Implementation Challenges
GPT-4o exhibited systematic processing limitations: when given the full batch of 20 samples, it repeatedly failed to complete the task. When reduced to batches of 5 samples, GPT-4o often omitted intermediate calculation steps, proceeding directly to final answers without showing the required step-by-step work. This behavior required individual problem presentation to obtain reliable step-by-step responses.
Claude Sonnet and Gemini 2.5 Flash processed full batches reliably and consistently followed the decom- position structure specified.
40

---

Published in Transactions on Machine Learning Research (10/2025)
C.4 Key Findings
The experiments revealed universal compulational split-brain syndrome across all three model families. At 5-digit complexity, performance varied significantly: GPT-4o achieved 5% overall accuracy despite 95-100% step-wise accuracy, Gemini achieved 95% overall accuracy, and Clande achieved perfect performance.
At 10-digit complexity, all models exhibited complete breakdown (0% overall accuracy) but through different failure modes. GPTT40 showed degraded step-wise accuracy (76 100%), while both Gemini and Claude maintained excellent step accuracy (95 100%) but failed completely at final summation.
These results provide direct empirical evidence for the instruction-execction disconnect: models can follow algorithmic procedures perfectly yet systematically fail at computational implementation, demonstrating fundamental limitations in bridging comprehension and competence even under optimal procedural guidance.
41


Reviews:
[{"review_id": "RFzwjj5cJL", "reviewer": "Reviewer_yB2q", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the brittleness of LLMs, and transformer architectures in general, at solving reasoning and mathematical computation tasks, and the phenomenon whereby LLMs fail such tasks but are able to correctly explain the steps one should take to perform them.\n\nThe paper primarily makes three claims about how architectural features of LLMs result in LLMs being incapable of performing reasoning and computation tasks, even when they can still explain the steps needed for performing them. The claims are (paraphrasing):\n- That contextual information in LLM token embeddings makes symbolic computation difficult by inhibiting appropriate abstraction of input variables\n- MLPs cannot perform symbolic computation effectively because, with finite units, they can only approximate and not perfectly match certain functions\n- Predictive learning results in pattern matching that does not learn to properly associate computations and instructions for how to perform them\n\nNotably, the paper argues that all three LLM weaknesses, mentioned in each claim, are inter-related and important for the disconnect between computation and description of computation. To back up the three claims the paper relies heavily on extensive citations, one mathematical proof, and 4 experiments.\n\nThe paper concludes that, among other things, pattern completion algorithms like LLMs will not be able to succeed at learning generalizable rules, as is required by symbolic tasks, without architectural changes.\n\n## Strengths\nThe main strengths of the paper, in my view, are some of the main observations about failure modes of LLMs. For example:\n- The phenomena of LLMs being able to describe the steps needed to perform a task but not being able to perform the task itself is interesting, and a worthy topic of discussion. Relatedly, the demonstration that an representation for describing a task is more similar to representations for describing other tasks than for performing the same task is a cool result\n- It is interesting to consider a potential downside of LLMs not being able perfectly approximate certain function, and that contextual information encoded in embeddings might actually be problematic for abstract tasks\n\nIt is also nice to read a paper where there are very few spelling/grammar errors\n\n## Weaknesses\nThere are three main weaknesses that seem rather severe. I have listed them below, referencing specific problems that I mention in the \"requested changes\" section later in the review:\n- The claims seem incompletely backed up (requested changes 2, 3, 5, 10, 16, 17)\n- The organization of the paper seems confusing at times, and appears to contain redundancy (requested changes 4, 9, 12, 22, 23, 26)\n- Some of the language choices are confusing and could accidentally mislead readers (requested changes 1, 6, 8, 25)\n- Some of the references are used to justify points that I don't believe they truly justify (14)", "claims_and_evidence_yn": "No", "claims_explanation": "I do not believe that the three main claims, or the claim of their interconnected nature, are sufficiently justified. I also believe that some of the results provided are not entirely relevant to the arguments of the paper. See requested changes for full details (e.g. points 2, 3, 5, 10, 16, 17)", "audience_yn": "Yes", "audience_explanation": "Given that the paper is on brittleness of LLMs it is certainly in a relevant enough domain for readers of TMLR.", "requested_changes": "I have added “**C**” at the end of changes that I view as critical and “**S**” after ones that I view more as suggestions.\n\n1. [Abstract/Introduction]: there are two choices of language that I believe might end up misleading people, and that I would like to see changed. First, the use of “split-brain”. As mentioned later in the paper (end of page 1/start of page 2: “artificial and biological intelligence may operate under fundamentally different principles.”), it seems strange to use a biological analogy for a non-biological phenomenon. This seems to be a particularly confusing choice given that certain AI algorithms are actually used to model biology (e.g. certain RNNs), and that other academics have noted problems with anthropomorphizing LLMs. Second, the use of “comprehension” in the abstract and introduction also seems somewhat misleading. Comprehension is a rather cognitively loaded term, and as mentioned later in the paper, it seems LLMs are likely doing pattern completion to solve problems, which to my understanding does not necessarily imply comprehension. **C**\n2. [Section 3 and throughout] It is mentioned in several places in the paper that the 3 claims are interconnected, or reinforce each other, but I don’t think that sufficient evidence is provided to support this. From reading the paper, it seems to me that each claim presents a problem that could potentially be an issue for abstract computation in its own right, and could be addressed independently of the other problems. Could the authors provide more justification for the interconnected nature of the claims? **C**\n3. [Section 3.2] In section 3.2 it is mentioned: (1) “The generation of precise computational results must ultimately occur through the feed-forward networks, which apply position-wise transformations to produce new content.”, and (2) “For arithmetic queries, the required computational results aren’t directly available in the attention-processed representations and must be computed—this is where the FFN serves as the primary computational engine.” To my understanding, self-attention layers can implement rather non-trivial nonlinear computations without needing to stack MLPs between them. If the authors could provide further justification for these claims, either with citations or otherwise, that would be great. **C**\n4. [Why Pattern Storage Wins on page 9; Experimental Validation of Hierarchical Pattern Assembly on page 10] I don’t believe these sections contribute much to the paper. A single residual layer is a universal function approximator because of the MLP. Therefore even just one transformer layer should be able to arbitrarily approximate any function on compact support–which means if approximation will suffice for a given problem then a single layer will, at least in theory, suffice. Multiple layers will perform even better, and I don’t understand why the iterative computation that multiple layers provide is a problem. If one needs a perfect, exact result, then these sections still seem redundant because the “Architectural Impossibility of Exact Computation” section shows that you cannot have this with current deep learning. Please correct me if I’m seeing things incorrectly; otherwise I think these sections are not necessary. **C**\n5. [Figure 3] Given that t-SNE is very affected by hyper-parameter settings I would ideally like to see this plot separately verified with another dimensionality reduction algorithm. **C**\n6. [Throughout paper]: The discussion of “embeddings” in the issue of averaging mentioned in “Claim 1” seems a bit ambiguous to me. These are frequently referred to as “token embeddings”, which I had thought are usually fixed during much of the training of LLMs. However, in section 3.1 it is suggested that the problem with averaging derives from token representations being learned via next-token prediction. Relatedly, I’m a little bit confused as to whether the embeddings mentioned in Figure 1 are token embeddings or are representations taken from intermediate layers. I’d appreciate it if the authors could clarify these things. **C**\n7. Previous work has studied architectures where abstract computation is separated from sensory details, potentially allowing better performance on reasoning tasks and tasks involving binding of abstract variables (On Neural Architecture Inductive Biases for\nRelational Tasks – Kerg et al. 2022; Emergent symbols through binding in external memory – Webb et al. 2021). These seem potentially relevant to the discussion of solutions to the problems discussed in the paper and could be worth citing. **S**\n8. [Line 3, Page 2] When the authors say “this architectural difference” it seems to refer to pattern completion optimized by next-token prediction, but this isn’t an architectural choice, is it? It seems more algorithmic to me **C**\n9. [Page 2] The first set of bullet points seem a little redundant given the second set. Could this be condensed into a single set of bullets? **S**\n10. [claim 3] why is next token prediction the critical ingredient for separate representations? Could it not be that in some layer representing a higher level of abstraction the representations of description and computation might merge, even when trained with next token prediction? It seems that the authors focus on showing that the representations are separate but do not connect next token prediction as being responsible for this. I would thus like to see more justification here **C**\n11. [page 3] first time mentioning “scaffolding strategies” without defining them. Could be useful to give some description at the first mention instead of at the second mention later on the same page **S**\n12. [Section 2] I find section two a little bit confusingly organized. I think the two main reasons for this is that similar topics are returned to more than once–for example “Training Dynamics and Path Dependence” in section 2.2 and “Training-Order Dependencies and Pattern Storage” in section 2.3–and that the headings don’t seem to be organized with a logical flow in mind. To the latter point, I think one way that flow could be improved is by organizing previous literature based on how it applies to each of the three claims (e.g., having a heading for each claim). **S**\n13. [End of intro and section 2.5] Contributions are listed twice, which seems a little redundant. Perhaps have contributions and positioning listed only after related work is discussed? **S**\n14. [“Training and Path Dependence” section, page 4]: in this section the authors list takeaways from the Power et al. 2022 and Tigges et al. 2024 papers that appear different from the content of these papers. In Power et al. they observe Grokking, where the trained networks perfectly fit simple computational tasks, exhibiting full generalization. I could not find mention of the “path dependent artifacts” that the authors of the current paper list as a takeaway. The results of Tigges et al. do suggest that “while computational circuits emerge consistently across scale, their specific implementations vary significantly during training”, however it is not clear the takeaway, listed by the authors of the current manuscript, that “apparent “reasoning circuits” may reflect training-specific pattern coordination rather than universal computational principles”, is supported by the results of Tigges et al. I could imagine that computational principles might remain the same even while circuits implementing them change. Could the authors address these apparent inconsistencies? **C**\n15. [Equations (1) and (2)] the variables in the equations are not defined! Also the equation doesn’t seem to contribute to the paper. Suggest either defining variables and elaborating, or removing it. **C**\n16. [Section 3.1] The authors write: “As Equation 2 predicts, contextually-averaged embeddings resist the domain binding required for stable symbolic circuits.” Could the authors elaborate please? It is not immediately clear to me why this should be. **C**\n17. [Section 3.2] Claim 2 seems to rely on the statement by the authors that “symbolic computation requires exact results across potentially unbounded inputs”, and therefore the approximation abilities of modern deep learning methods are a problem because, without infinite units, they can only provide approximate results. It is interesting to reflect upon this point, but I’m not sure that it’s relevant for the kinds of real world applications for which LLMs are used. I would like to see more justification for why we need exact results with LLMs. **C**\n18. [Section 3.2] It seems obvious to me that ReLU MLPs can only approximate continuous functions, so I’m not sure that a proof is required for this. Also the piecewise linearity proof in the appendix seems perhaps unnecessary for the same reason **S**\n19. [Table 2] Are these results repeated from the Li et al study? If so, I think this should be mentioned in the title or caption of the table. **C**\n20. [“Functional Specialization Architectures” section, Page 20] This section needs citations **C**\n21. [Table 3] This table could use better explanations. E.g., explaining the “sum error” row in the main text, the relevance of the fractions listed after performance. Also, why is it “30.125” and “30.1210” in column 1 and not simply “5” and “10”? **C**\n22. [Section 6, lines 6-11] it seems like this section just repeats the main points of “comprehension vs competence” but as “general vs generalizable”. Perhaps redundant? **S** \n23. [Appendix B.4] If I’m not mistaken, isn’t the content of this section mentioned in the main paper? If so, this seems not necessary to include **S**.\n24. [Section 2.2, paragraph 2] There are places in the paper where author names are repeated before a citation and then within a citation–e.g. Dziri et al. (Dziri et al., 2023). Perhaps there could be a way to avoid repeating the same information twice? **S**\n25. [Section 2.2 paragraph 3 line 1] Is “execution limitation” the “comprehension-execution split”? **S**\n26. [4 lines after Eq. 3] could you elaborate on what you mean by “through weight configuration alone”? **C**", "broader_impact_concerns": "Given that the paper is on LLMs, coupled with the diverse and sometimes quite damaging effects that LLMs are having on society (environmental, misinformation related, surveillance and military purposes, etc), it would be really great to have broader impacts mentioned somewhere. If not in the main paper, then at least in the appendix.", "additional_comments": "While I have organized the review around specific \"requested changes\", I believe that the paper's weaknesses are severe enough to warrant its rejection. If the other reviewers agree, hopefully the requested changes could be used to help make adjustments for a future submission of the paper, if the authors decide to go that route.\n\nI would also like to note that surveying past results seemed to take an almost larger role in the paper than novel results. For this reason, if the authors decide to restructure things it might make sense to consider writing more of a survey (if the reference issues mentioned in point 14, above, are resolved). Still, a couple of the results (see strengths section of the review) seem like they could be used in a research paper if they were elaborated upon.\n\n**Lastly, I will mention that my experience is moreso in deep learning than in the study of LLMs per se. For this reason my knowledge of literature related to the submission is not as good as it should be.**", "extra_scores": {}, "extra_text": {}}, {"review_id": "kavSVVxO9e", "reviewer": "Reviewer_UthV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This submission attempts to advance our understanding of the\nlimitations of LLMs. It is important to note that here the focus is\nonly on \"pretrained transformer-based LLMs trained on next-token\nprediction objectives over natural language corpora, without access to\nexternal tools or explicit reasoning scaffolds.\"\n\nThe authors appear to know the literature well. (I cannot be sure\nabout this, since I do not work in this area, so it could be that\nimportant contributions of which I am unaware have been missed.) In\nany case, there are many summaries and analyses of interesting and\nrelevant work done by others. The quality of the writing is good:\nclear and without grammatical errors/typos etc.\n\nThe results of ARC-AGI-2 show us that there are certain types of task\nthat LLMs are very poor at. The goal of the current paper is not just\nto provide examples of LLM failure, but to explain what it is about\nLLMs that lead to such failures. The authors' explanation is given by\ntheir 3 main claims (p. 2).\n\nA key point of the paper is that the 3 problems identified in the 3 claims are jointly necessary for observed LLM problems.", "claims_and_evidence_yn": "Yes", "claims_explanation": "I am persuaded that the 3 claims are correct. The first two\nessentially state that there are problems using 'continuous' reasoning\nfor essentially 'discrete' tasks (e.g. 'binding' and exact symbolic\noperations). The third points to the LLM's 'flat' representation of\nknowledge: we have no types and therefore (for example) a distinction\nbetween executing and describing algorithms is unrepresentable. Of\ncourse, there is a trade-off here. It is precisely by reducing\neverything to a single continuous optimisation task that leads to much\nof what is so impressive about LLMs.\n\nAlthough the 3 claims are not the sort that can be 'proved' in the\nmathematical sense, the authors take each in turn and provide\nsupporting evidence and reasoned argument to support them. I\nparticularly liked the investigation of geometric separation relevant\nto Claim 3.", "audience_yn": "Yes", "audience_explanation": "The main issue here is significance. The authors state that:\n\"... while previous studies document what [LLM] failures occur across\ndomains, we provide a principled mechanistic explanation for why they\noccur...\" Of course, it is not true that there is *no* existing work\naiming to explain limitations of LLMs. Some of the papers referenced\nhere attempt at least some explanation of their empirical\nfindings. In addition, Section 2.1 explicitly summarises a number of papers providing explanations.\n\nHowever, I am not aware of other work that does as much of\na thorough job as the current paper (but I am not an expert on this\nliterature, so it's quite possible there are such papers I don't know\nabout). Assuming that the current paper has little competition in\nterms of thorough explanation of LLM limitations, there remains the\nissue of the paper's narrow focus. Perhaps we don't care too much\nabout the limitations of \"pretrained transformer-based LLMs trained on\nnext-token prediction objectives over natural language corpora,\nwithout access to external tools or explicit reasoning scaffolds.\"\nsince to get closer to AGI we would not artificially restrict\nourselves to such systems.", "requested_changes": "Assuming that the argument advanced here is indeed original, I am surprised that it (or something similar or a competing but still comprehensive explanation) has not previously been stated, given the intense interest there is in LLMs. If the authors can provide an explanation for this surprising fact that would strengthen the paper.\n\nI would like a stronger argument for the limitations mentioned above. Of course, imposing such limitations make the analysis more straightforward. But, as mentioned earlier, perhaps there is diminishing interest in these particular sorts of LLMs.", "broader_impact_concerns": "I do not have 'concerns' (as in worries) about the ethical implications of the work. I think the ethical implications are positive since the paper points to problems that can occur by being too trusting in LLMs.", "additional_comments": "MINOR POINTS\n\np6: (1) and (2) are not meant to be a formal definition of how \"The\ntraining objective optimizes embeddings through next- token\nprediction:\" but I think \"E\" should be defined. Also what's the\nconnection between (1) and (2).\n\nThe Shojaee et al reference is wrong: wrong arxiv id.", "extra_scores": {}, "extra_text": {}}, {"review_id": "1A8bZacan8", "reviewer": "Reviewer_Jgku", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper core contribution is introducing the split-brain syndrome in LLM/LRMs: the fact that they know an algorithm but they cannot execute it reliably. Specifically, the authors argue that there are architectural limitations on current models that prevent them in really solving this issue by three main claims: token embedding limitations, feed-forward networks limitations, instruction-execution disconnection.\n\nI really enjoyed the paper. It well written, it has a very nice story, it is sound, it has nice visualisations and insights for each of the claims. Limitations are stated clearly at the beginning. Related works are up to date and well discussed. \n\nI just have one minor complain: some things are not very well introduced. Specifically, equations have terms that are not defined and it is not clear what they are (e.g. already eq 1 and 2, only C(w) is explicitly defined, but also for the other equations there are similar issues). It is perhaps not fundamental for the understanding of the paper to have all the details, but it would be nice to have a more formal definition of the terms and an intuitive explanation next to them.\n\nMinor typos: sometimes the wrong character to close quotes is used (e.g. end of page 12, should be `` in latex). Sometimes citet and citep are misplaced (e.g. “(Vashishtha et al., 2025) demonstrates success on causal reasoning” should be \\citet, and “similar to distributional semantics models like Word2Vec Mikolov et al. (2013).” should be \\citep).\n\nFinally, in the limitations of FFNs, I wonder what is the relationship with works on the computational complexity limitations of transformers. For example [1] where they demonstrated that multi-layer transformers cannot solve problems such as 2-SAT, Horn-SAT, etc. Or [2] where they show that a T-chain of thought enables transformers to solve problems corresponding to Boolean circuits of size T. Even though this second might be related more with the self scaffolding part? Maybe the connections between these things can be made more explicit at the beginning of 2.4 where it’s already hinted?\n\n[1] On limitations of the transformer architecture (Peng et al 2024)\n[2] Chain of thought empowers transformers to solve inherently serial problems (Li et al 2024)", "claims_and_evidence_yn": "Yes", "claims_explanation": "Every claim has a proof in the appendix or proper references / experiments.", "audience_yn": "Yes", "audience_explanation": "The topic of the paper is very relevant these days, and it offers an interesting (new) perspective.", "requested_changes": "Fix background and typos (see review).", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)