The Mind's Transformer: Computational Neuroanatomy of LLM-Brain Alignment
Authors: Cheng-Yeh Chen
Raghupathy Sivakumar
Venue: ICLR 2026 Poster (2026)
Keywords: language model
neuroscience
brain alignment
fMRI
Primary area: applications to neuroscience & cognitive science
TL;DR: We introduce computational neuroanatomy to map intermediate states in a transformer block to brain regions, revealing intra-block hierarchy and achieving unprecedented auditory cortex alignment with LLMs.

Abstract: The alignment of Large Language Models (LLMs) and brain activity provides a powerful framework to advance our understanding of cognitive neuroscience and artificial intelligence. In this work, we zoom into one of the fundamental units of LLMs—the transformer block—to provide the first systematic computational neuroanatomy of its internal operations and human brain acitivity during language processing. Analyzing 21 state-of-the-art LLMs across five model families, we extract and evaluate 13 distinct intermediate states per transformer block—from initial layer normalization through attention mechanisms to feed-forward networks (FFNs). Our analysis reveals three key findings: (1) The commonly used hidden states in LLMs are surprisingly suboptimal, with over 90\% of brain voxels in sensory and language regions better explained by previously unexplored intermediate computations; (2) Different computational stages within a single transformer block map to anatomically distinct brain systems, revealing an intra-block hierarchy where early attention states align with sensory cortices while later FFN states correspond to association areas—mirroring the cortical processing hierarchy; (3) Rotary Positional Embeddings (RoPE) specifically enhance alignment along the brain's auditory processing streams. Per-head queries with RoPE best explain 74\% of auditory cortex activity compared to 8\% without RoPE, providing the first neurobiological validation of this architectural component in LLMs. Building on these insights, we propose MindTransformer, a feature selection framework that learns brain-aligned representations from all intermediate states. MindTransformer achieves significant brain alignment performance, with correlation improvements in primary auditory cortex exceeding gains from 456× model scaling. Our computational neuroanatomy approach opens new directions for understanding both biological intelligence through the lens of transformer computations and artificial intelligence through principles of brain organization.

Published as a conference paper at ICLR 2026
THE MIND'S TRANSFORMER: COMPUTATIONAL NEUROANATOMY OF LLM-BRAIN ALIGNMENT
Cheng- Yeh Chen, Raghupathy Sivakumar Georgia Institute of Technology, Atlanta, GA, USA cchen84 70 76 raggttccchh..................... edu, sivaßece, gatech.ee........... edu
ABSTRACT
The alignment of Large Language Models (LLMs) and brain activity provides a powerful framework to advance our understanding of cognitive neuroscience and artificial intelligence. In this work, we zoom into one of the fundamental units of LLMs the transformer block-to provide the first systematic computational neuroanatomy of its internal operations and human brain acitivity during language processing. Analyzing 21 state-of-the-aae LLMs across five model families, we ex- tract and evaluate 13 distinct intermediate states per transformer block- from ini- tial layer normalization through attention mechanisms to feed-forward networks (FFNs). Our analysis reveals three key findings: (1) The commonly used hid- den states in LLMs surprisingly suboptimal, with over 90% of brain voxels in sensory and language regions better explained by previously unexplored interme- diate computations; (2) Different computational stages within a single transformer block map to anatomically distinct brain systems, revealing an intra-block hierar- chy where early attention states align with sensory cortices while later FFN states correspond to association treas-niiroring the cortical processing hierarchy; (3) Rotary Positional Embeddings (RoPE) specifically enhance alignment along the brain's 's auditory processing streams. Per-head queries with RoPE best explain 74% of auditory cortex activity compared 8% without RoPE, providing the first neurobiological validation of this architectural component in LLMs. Build- ing on these insights, we propose MindTraanffomme" a feature selection frame- work that learns brain-aligned representations from all intermediate states. Mind- Transformer achieves significant brain alignment performance, with correlation improvements in primary auditory cortex exceeding gains from 456x model scal- ing. Our computational neuroanatomy approach opens new directions for under- standing both biological intelligence through the lens of transformer computations and artificial intelligence through principles of brain organization.
INTRODUCTION
The remarkable success of Large Language Models (LLMs) has catalyzed fundamental question in cognitive science: do these artificial systems process language through mechanisms similar to the human brain? Recent work has demonstrated striking correlations between LLM representations and neural activity measured through fMRI (Toneva & Wehbe 2019 Schrimpf et al. [021] Caucheteux & King 2022 Goldstein et al.) 2022). with alignment improved through various interventions in- cluding dataset scaling (Antonello et al. 2023) Gokce & Schrimpi, 2025 Ren et al., 2025). model
scaling (Antonello al. [2023]Gokee Gokce & Schrimpi] [025|] Ren et al. [2025] Bonnasse- Gahot & Pal-
lier] (2024), prompting (Sun & Moens, 2023) Ren et al., 2025). fine-tuning (Sun & Moens [023]
Aw et 2024 Oota et al. 2025), and taskonomy (Oota et al. 2022 Aw & Toneva 2023). These
findings suggest that LLMs may serve as computational models of human language processing, with
applications ranging from brain prediction (d' A'Ascooi et al. 2026 to causal manipulation of neural
activity (Tuckute et al. 2024).
However, a critical methodological limitation undermines current understanding: existing ap- proaches treat LLM architectures black boxes, using only single representation per layer while
Source code: https//gthhllcoommmmmayaaaiiiint

---

Published as a conference paper at ICLR 2026
overlooking the rich internal computations within LLMs. This practice assumes that all neurally- relevant information is compressed into a single vector, ignoring the intermediate computations the layer normalization, positional embeddings, multi-head attention projections, and feed-forward transformations- that collectively implement the model's processing. While pioneering studies have examined specific components like attention weights ( @Lamarre et et. 2022) individual atten- tion heads (Kumar et al.) (202)) no systematic analysis exists of how all intermediate computations inside transformer blocks map to brain activity. This gap obscures potential convergence points between the discrete, semantic nature of text processing and the continuous, sensory-driven mecha- nisms of the brain. While acoustic language models have successfully characterized neural activity in low-level auditory regions (Tuckute et al., 2024; Miller et al. [022 Antonello et al. 2023). text-
based LLMSs typically fail to achieve meaningful alignment these areas, suggesting a disconnect between textual and sensory representations (Caucheteux & King 2022; Kauf et al. 2024 Doerig
ct al. 2025 Goldstein et al. 2025). Consequently, the ncurobiological relevance of key architectural components that bridge this gap such as positional embeddings remains unexplored, leaving disconnect between engineering design choices and biological plausibility.
We address these challenges through comprehensive computational neuroanatomy of transformer block architectures, one of the fundamental units for LLMs. Our approach systematically decom- poses each transformer block into 13 distinct intermediate through per-head attention computations to feed-forward network activations and evaluates their individual and collective correspondence with brain activity. Analyzing 21 statc-of-the-art models (270M to 123B parameters) from five major families (Llama, Qwen, Mistral, GPT, Gemma) on naturalistic story listening IMRI data, we make three principal contributions:
pre-attention normalization
Revealing the suboptimality of current practices through systematic analysis. We demonstrate that the commonly used hidden states are remarkably inefficient, with over 90% and 96% of brain voxels in language and sensory regions better explained by previ- ously unexplored intermediate computations. This finding provides new dimension to align neurally-relevant representation in LLMS onto the brain.
Uncovering an intra-block processing hierarchy that mirrors cortical organization. Different computational stages within single transformer block map to anatomically dis- tinct brain systems -ealyy attention states align with sensory cortices while later FFN states correspond to association areas. This reveals fine fine-graing grained computational hierarchy within each block that parallels the brain's 's own anatomical processing hierarchy, extending be- yond the known layer- wise progression in LLMs.
Establishing robust alignment with low-level sensory processing through architectural components. We identify that Rotary Positional Embeddings (RoPE) specifically enhance alignment with the brain's 's auditory processing streams. Inside the multi-head attention, per-head queries with RoPE best explain 74% of auditory cortex voxels versus 8% without, systematically improving aligmment along both ventral and dorsal auditory pathways. This provides the first neurobiological validation of RoPE's functional role and demonstrates that architectural design choices can have direct neural correlates.
Building on these insights, we propose MindTransformer, principled framework that learns brain- aligned representations from all intermediate states. First, it discovers neurally-relevant features through ridge regression on concatenated representations; second, it selects the most informative subset for final model training. MindTraanfformer achieves significant performance in language network and audio cortex, especially with correlation improvements of 0.111 I in primary auditory cortex-gains that exceed those from scaling LLMs by 456x (from 270M to 123B parameters).
2 RELATED WORK
2.1 THE LANDSCAPE OF LLM-BRAIN ALIGNMENT
Research in LLM-brain alignment has established that model representations increasingly corre- spond with neural activity as models and datasets scale (Antonello et al. 2023 Gokce & Schrimpf, 2025| Ren et al. 2025). This scaling allows models to recover fundamental brain properties like hett-hemisphere lateralization (ahho & Pallier 2024). Alignment is further improved by
2

---

Published as a conference paper at ICLR 2026
training models specific objectives, such as cognitively demanding tasks (Oota al. 2022 Aw & Toneva 2023) or instruction corruuctioofoloowinng (Aw et al., 20224] Oota et al. 2025). Recent work has also ex-
plored prompt engineering strategies to enhance alignment (Sun & Moens 2023 Ren et al. 2025)
The strength of this connection has enabled applications ranging from brain prediction (d' Ascoli et a 2026) causal control of neural activity (Tuckute al. 2024). Due much hetter alignment in regions of high-level semantic processing in the brain, these preceding endeavors form the pre- vailing view that LLM-brain alignment converges high-level semantic processing, with low-level sensory regions remaining inaccessible (Caucheteux & King 2022. Goldstein et al. 2025).
Kauf et al. 2024 Doerig et al.,
2025
2.2
FROM MONOLITHIC TO MECHANISTIC ALIGNMENT
While most alignment research uses the final hidden state, growing body of work has begun to probe specific transformer mechanisms, with a strong focus on the attention component. This line of inquiry has shown that raw attention weights (Lamarre et al. (022), their similarity human eye- tracking patterns (Gao et al. 2023), and the specialized computations of individual attention heads (Kumar al. 2024) all predictive of distinct neural activity. Intriguingly, even shallow, untrained attention networks exhibit brain-like properties, highlighting the importance of architectural biases (AlKhamissi et al. 2024).
These pioneering studies validate examining internal mechanisms but leave critical gaps. First, they focus narrowly on attention while ignoring other computational stages- the layer normalization, positional embeddings, and feed-forward networks that comprise the majority of transformer com- putations. Second, they lack systematic analysis across the full range of intermediate states within transformer blocks. Our work provides the first comprehensive computational neuroanatomy analy- sis across all intermediate computations, revealing how each component contributes to brain align- ment and demonstrating that the choice of representation fundamentally determines which brain systems can be modeled.
METHODOLOGY
3.1 BRAIN DATASET AND PREPROCESSING
2022), a dataset specifically We use the publicly available Le Petit Prince fMRI Corpus (Li et designed for studying the neural basis of language during naturalistic story listening. The corpus provides fMRI data for three languages and we use the English subset with 49 English native speak- ers. During fMRI acquisition, each participant listened to an audiobook of the story Le Petil Prince, which is approximately 100 minutes duration and structurally divided into 9 distinct runs.
To create a robust, group-level signal and improve the signal-to-noie ratio (SNR), we average the IMRI time-series across all participant This group-averaged signal was used for all subsequent encoding models, with regions of interest (ROIs) defined using the Harvard-Oxfol structural atlas from et al. [2012] the language localizer from Fedorenko et al. 2010) to enable precise computational neuroanatomy mapping.
3.2 LLM ACTIVATION EXTRACTION AND PREPROCESSING
To comprehensively investigate the computational neuroanatomy of transformer architectures, we analyze 21 state-of-the-art open-weight LLMs spanning five major model families: Llama, Qwen, Mistral, GPT, and Gemma. These models range from 270M to 123B parameters, encompassing di- verse architectural choices including standard multi-head attention (MHA), grouped-query attention (GQA), multi-query attention (MQA), and mixture-of-experts (MoB) architectures (see Appendix for complete architectural specifications).
We process the corresponding stimulus text in the fMRI dataset through all 21 models. The text first tokenized using each model's native tokenizer, then passed through the model to extract intermediate activations. We extract activations from every transformer block across all layers. For
2 All the preprocessing fMRI, including downsampling, averaging, masking, etc. is available in source code http:///ww....occcccceeey/////aa Transsaformnnn...e.........
3

---

Published as a conference paper at ICLR 2026
model with Nayer transformer blocks, this yields Mayar x 13 distinct activation states per token, corresponding to the 13 intermediate states we identify within each block (detailed in Section 3.4).
To align these token-level activations with the temporal resolution of fMRI. we implement word- level aggregation strategy commonly used literature (Antonello et al. 2023 Bonnasse-Gahe & Pallier 2024). Since transformer tokenizers often split words into subword tokens, we average the activation vectors of all tokens belonging to the same word. This produces a sequence word-level activation vectors directly corresponding the word-by-word onset timing the fMRI experiment.
3.3
VOXEL- WISE ENCODING PIPELINE
Our work builds upon the standard voxel-wise linear encoding paradigm, widely adopted and validated methodology for aligning LLM representations with fMRI data (Antonello et al.] [023] Kumar et al., (024 Tuckute al. (024). This pipeline learns mapping from LLM features fMRI signals for each brain voxel, forming the foundation of our computational neuroanatomy approach. Our implementation proceeds as follows
1. HRF Convolution:
The brain's hemodynamic response measured by fMRI slow and delayed relative to neural activity. To account for this neurobiological constraint, we con- volve the time-series of word-level LLM activations (obtained from the preprocessing step above) with canonical Hemodynamie Response Function
the Glover HRF (Glover 1999) to create feature space that is temporally aligned with fMRI signal.
2. Ridge Regression: We use L2-regularized linear regression (Ridge Regression) learn a mapping from the HRF-convolved LLM features to the fMRI signal for each individual brain voxel. The regularization parameter a is optimized via nested cross-validation within the training folds, typically ranging from 10-2 to 10a
3. Robust Cross-Vass Validation: To ensure generalizability of our computational neuroanatomy findings, we employ 9-fold each fold, one run is held out as the test set, and the model is trained on the remaining 8 runs. This guarantees that the test set is always entirely unseen during training The final reported performance is the Pearson correlation between the predicted and actual fMRI signals, averaged across the 9 folds testing.
validation scheme based on the dataset's 9 runs. For
3.4 DISSECTING THE TRANSFORMER BLOCK
Rather than using single representation, we decompose the transformer's internal computation into thirteen distinct states that capture the complete information processing pipeline. This granular ap- proach reveals how different stages of computation correspond to distinct neuroanatomical systems.
We organize these states into three major computational stages that reflect universal computations of transformer blocks for most state-of-the-art LLMs:
1. Block Input consists of the input hidden state from the previous layer and its pre-attention normalized state that stabilizes activations for subsequent processing.
2. Attention Mechanism encompasses seven critical states: per-head queries and keys both before and after applying Rotary Positional Embeddings (RoPE)-allowin us isolate the neural contribution of positional encoding; per-head values containing the content to be attended to; per-head context vectors representing each head's weighted synthesis; and the combined attention output after projection.
3. FFN & Residuals captures the remaining transformation pipeline: the post-attention hid- den state after the first residual connection, its pre-FFre normalized state, the FFN activated state after expansion to higher dimension, and the FFN output after down-projection back to model dimension.
3 All scripts for HRF covolution, ridge regression, and cross-validation available in source code het..h/.//ceeeegyy//nnnniii
"The analysis in Section 4 selects optimal states based on test set performance for exploratory purposes. Conversely, the results in Section eeeffooce strict separation, with feature selection performed solely on the training set prevent data leakage.
4

---

Published as a conference paper at ICLR 2026
Transformer block
Winning ratio each
intermediate state (Over whole-braln vaxels)
Layerboorm Output linerr
Foodlomerr (FPN)
Down projection
State
Ratio
Transformer
FTN Oupui
4.5%
Left lateral view
Upp projectio
PigFiig
4.0%
Layetoom
Btate
Transformer block
Contatee
Right lateral view
Grtspet--h pere--ii-rrryy Alwnroon (DOA)) Mall-Ourr Alienton (PIAA)
Cutpul Projuction
PaiPieed Conkod Vicat
14.31%
Por-ltaad
2.4%%
Altuntion
Transformer block
Pex-Pteet mi HUP
7,83%
Projedict
Projecion
Piojetie
Plrn
14. 6..6%%
Anterior view
Tokenizer & Positional encoding
Pio Apoer
3.00%
Eapetiom
Sidden
2,34%
Posterior view
(a)
(b)
(c)
Figure I: 13 intermediate states extracted from each transformer block inside LLMs. (a) Overall architecture of LLMs. (b) Per-state visualization of position within a transformer block and statistics of winning ratio from each intermediate state, averaged from all the transformer blocks of 21 LLMs in 5 families. (c) Brain plots colored by the best state for each voxel with various views for the 12-th layer of Llama-3.2 8B. The color map provided in (b).
This systematic extraction yields rich, multi-dimensional representation of the transformer's in- ternal computation shown in Figure lb Each state is preserved with its full dimensionality, with tensor shapes ranging from (B, D model) for pre-attention hidden states to (B, Nq. S, Dhead) or [B, Nku, S, Dreat) for per-head attention-related representations and (B, S, Dma for the expanded FFN activation, where B is the batch size, is the sequence length, Diodtel is the model dimension, N₂ and NA. are the number of attention heads for query and key/value, Dread is the per-head dimen- sion, and Dim is the feed-forward inner dimension. We detail the definition of all intermediate states and their tensor dimensions in Appendix B
4 COMPUTATIONAL. NEUROANATOMI ANALYSIS
4.1 BEYOND THE HIDDEN STATE: AN EMERGENT INTRA BLOCK HIERARCHY
Our first major finding in computational neuroanatomy fundamentally challenges the standard prac- tice in LLM- b-barin alignment. We apply the voxel- wise encoding pipeline each intermediate state to create a comprehensive computational neurounatomy map. For each voxel, we identify the best intermediate state that could explain the activity of that voxel. As shown in Figure I the two most commonly used representations- the input hidden state and the per-head context vector (Kumar al.) 2024]-arc suboptimal from a computational neuroanatomy perspective. Together, they best from input hidden state and 14.31 % from per-head context vector in Figure [b) of brain voxels. If we further focus on the audio-sensory cortex (Da Costa et al. 2011 Hamilton et al. 2021), or the language networks (Fedorenko et al. 2010) in the brain (regions interest detailed in Appendix []. we get lower percentage of best explained voxels. Merely 9.91 % and 3.68% of voxels in the language network and audio cortex are best explained by the commonly used states (shown in Appendix D). The vast majority of the brain is better modeled by previously unexplored intermediate states, demonstrating the necessity of our granular computa- tional neuroanatomy approach. Quantitatively, selecting the best intermediate state raises alignment correlations from 0.275 to 0.296 in the whole brain and from 0.433 to 0.450 in the language network, with the auditory cortex showing the largest improvement from 0.407 to 0.475 (Appendix Table
explain the activity in 16.65% (2.34%
Furthermore, our computational neuroanatomy analysis reveals consistent pattern of functional specialization within the transformer block that mirrors the brain't 's own information processing hier- archy. Figure shows that early-stage computations within block, like attention-related states,
5

---

Published as a conference paper at ICLR 2026
Low-level sensory processing
High-level semantic processing
Hescht's Gyrus
Superior Temporal Gyrus (posterior division)
Inferior Frontal Cyrus (pers triengularis)
Angular Gynus
Model Farily
Ulma--3
Wiriing Vone Ratoo
Gemma
0
Diven-3
GPT
Mistral
Modal Size
R
0.278
1230
phe pend 5
ree bosol rape
compined pripoct Latte-alage computition
Er outeed
PRR 200 sero sida
per Jore, 5
per, jost roo
fead co yrote
ain helpe sorm anly Lalt-staage computition
acchviee sisia
pe, sone dins
you 2
contee, vebte
peet all Ausden, porm youp
pre RuAltidddd sooo
St. poppost
sec ieatt 5
contaa, vetter comprell snf shiue
pir Pe. rureen eise de Ruutttitee sise in, miie
ytu, batien
por trail
consired able
me rorm
per. Iusl
Late post yon pesten compusiiion
Early-stage computation
Early-olage computation
Early-Saage computation
Lite-slage computation
Early-stage computation
Figure 2: Functional specialization within a transformer block. Across LLM layers, early-stage states ge. per-head Query with RoPE) consistently dominate in low-level sensory regions (e.g.. Heschl's s gyrus and superior temporal gyrus), while late-stage states (e.g., FFN-related states) dom- inate in high-level association cortex (c.g., inferior frontal gyrus and angular gyrus), revealing an emergent intra-block processing hierarchy.
Welgited Computationl Depih
Mistral
GPT-068
Llama-3
Qwen-3
Gemma-3
FFN octivated state
autpui
A-- 5 Slopeed..2 23
47-0..0
g--1.03 Slopard.23 28
A Blopen8.12 12
R1.0.00 Slaperd do
Pio FFN norm state
77 As
wan hidden state Combined mudpait Per head contect Per head q rope twerd q
Sioe mo 33
A Pa0.7 79
Ruug 74 Slopesk
Slopend.1
MTG
P7
MTG
Figure 3: Weighted Computational Depth versus Cortical Hierarchy across LLM families. The analysis quantifies the topological alignment between the transformer's 's internal processing depth (y-axis) and the brain's cortical hierarchy (x-axis). We distinguish two functional segments: the auditory stream (HG to MTG) and the language network (MTG to AG). High It2 values are observed across most LLM families, characterized by a steep slope in the auditory segment, indicating a rapid progression through early computational states, and a plateau in the language segment, confirming hicrarchical mapping between biological auditory pathways and transformer block computations.
preferentially align with low-level sensory cortices (e.g., Heschl's gyrus and superior temporal gyrus), while late-stage computations, like FFN-related states, align with high-level association cortices (c.g., inferior frontal gyrus and angular gyrus). To quantify this, we define metric of Computational Depth, calculated as the normalized index of the winning intermediate state within the transformer block's s processing sequence. We correlate this with the Cortical Hierarchy of the corresponding brain regions. Detailed mathematical formulation of these metrics is provided in Ap- pendix F As shown in Figure 3 we observe a striking consistency across all LLM families. The auditory processing stream (spanning from Heschl's Gyrus to the Middle Temporal Gyrus) exhibits steep positive slope, demonstrating a strong linear mapping where ascending cortical levels cor- respond to deeper intra-block computational depth. In contrast, the high-level language network (extending from the MTG to the Angular Gyrus) displays a flattened trajectory, indicating a com- putational plateau where alignment stabilizes at the block's 's later stages. This discovery suggests an intra-block computational hierarchy that parallels neuroanatomical organization: early attention- related states process immediate, stimulus-driven information akin to sensory cortices, while later FFN states handle more abstract, integrated information similar to association areas. This finding extends the known layer-wise hierarchy in LLMs MaMs-arrly early layers for syntax and late layers for mantics (Tenney et al. 2019 Rogers et al. 2020 to a more fine- grained computational neuroanatomy.
level of

---

Published as a conference paper at ICLR 2026
Ventral Analysis
Hemispherel
Parcels with top-10 correlation difference
Parcel
Difference
HG 0.1233
(Baayy antedding RuPF))
PT
0.1080
STGant
0.0751
Seany
Copmine
TTGpost
0.0733
lngusiii peresption
POC
0.0510
Dorsal Stream Analysis
Hemisphere)
COC
0.0342
PP 0.0328
Froet redden stane)
0.0312
0.0111
1.0066
Sennary
Cogarii
Dorsal stream audio liggestio porseption
Diffeennc raap
Query embedding RoPE aubstantaaly enhance brain-LLM correlation along streams.
(a)
(b)
(c)
Figure 4: Per-head query with Rotary Positional Embedding (RoPE) substantially enhance L.M-- brain correlation along the ventral and dorsal streams for audio linguistic perception. (a) Correlation map for Query embedding with RoPE and input hidden state. The difference map delineates both the ventral and dorsal streams. (b) Parcels with top-10 10 correlation difference based on structural atlas clearly outline the ventral and dorsal streams, from low-level sensory processing to high-level cognitive processing. (c) Ventral and dorsal streams for audio linguistic perception.
4.2 THE ROLE OF ROPE IN DELINEATING THE BRAIN'S AUDITORY STREAMS
Among all intermediate states examined in our computational neuroanatomy analysis, the per-head query with RoPE provides the most substantial and systematic improvement in brain alignment. We focus on this specific intermediate state inside the attention mechanism and demonstrate its superior correspondence with the neuroanatomy of auditory streams in the brain.
We compare how the per-head query with RoPE improves the brain correlation from the input hidden state through detailed computational neuroanatomy. As illustrated in Figure 4a we observe strong improvement around the Sylvian fissure in the difference map. We then rank the top-10 parcels with highest correlation difference. The ranking table in Figure 4b strikingly delineates the brain's ' canonical dorsal and ventral streams for auditory language processing- a fundamental principle of auditory neuroanatomy- as labeled in Figure 4d The largest improvement is observed in the pri- mary auditory cortex (Heschl's s gyrus, HG), with the effect cascading along both anatomical streams to regions like the planum temporale (PT) and the superior temporal gyrus (STG) in Figure 4b
This computational neuroanatomy result significant for two reasons. First, it provides the first strong evidence of LLM-brain alignment in low-level sensory processing regions, especially in Hes- chl's gyrus. The prevailing consensus has been that LLM-brain alignment is primarily sensitive to high-level semantic information (Caucheteux & King 2022 Kauf et al. 2024 Doerig et al.) 2025 Goldstein et al., (025). Our computational neuroanatomy findings suggest that this was a limitation of the representations being used, not the methodology itself: by examining the correct intermediate computational state through a neuroanatomical lens, we uncover a deep correspondence in how the model and brain align with each other when processing the fundamental auditory signal.
Second, this provides the first neurobiological evidence for the functional role of Rotary Positional Embeddings (RoPE) within a computational neuroanatomy framework. By isolating the effect of RoPE (comparing per-head query with and without it). we find that RoPE is critical for this align- ment with the low-level auditory stream. The winning ratio analysis reveals striking pattern: while per-head query without RoPE wins in only 7.82% of auditory cortex voxels, per-head query with RoPE dominates overwhelmingly at 73.88% (Appendix D nearly tenfold increase. This dramatic shift is specific to auditory regions; in the language network, per-head query actually out- counterpart (19.43% vs. 9.82%, Appendix D) suggesting that RoPE's contribution is precisely tuned to the computational demands of low-level sensory processing, This regional specificity demonstrates that RoPE systematically enhances the ability of attention heads to capture the sequential and positional information critical for processing the structural nature of
performs its

---

Published as a conference paper at ICLR 2026
speech a function that mirrors the neuroanatomical role of the auditory ventral stream. This com- putational neuroanatomy discovery bridges the gap between architectural design choices in artificial systems and their biological counterparts.
5 MINDTRANSFORMER: A COMPUTATIONAL NEUROANATOMY ALIGNMENT FRAMEWORK
Our computational neuroanatomy analysis in the preceding section reveals a key insight: the most neurally-relevant information in a transformer is not localized to a single representation but is distributed across diverse ecosystem of internal computational states that map distinct neu- roanatomical regions. This motivates us to develop MindTTransforeet principled framework that systematically leverages these distributed representations to achieve superior brain alignment.
5.1 THE MINDTRANSFORMER FRAMEWORK
We propose MindTTransfomen in two complementary modes, each addressing different aspects of the LL.M-brain alignment challenge:
Mind Transformer Mode 1: Optimal Single-State Selection. In this mode, we systematically evaluate all thirteen intermediate states through independent ridge regression models identify the single best predictor of brain activity. For each state, we train voxel-wise encoding model in the training set and compute the prediction correlation the testing set. The state achieving the highest correlation across voxels ROIs is selected as the optimal representation. This approach moves beyond the arbitrary selection of hidden states or context vectors, instead letting the brain's response patterns guide the choice of representation. The per-voxel version of this mode is exactly the method implemented Section to obtain the analysis of winning ratio in Figure Th and intra- block hierarchy in Figure
Tranfforrrrrmmr Mode 2: Multi-State Feature Integration. While Mode identifies the sin- gle best state, Mode 2 further integrate the information provided by different intermediate states to deliver even better prediction correlation. Different brain regions may be best explained by the com- bination of different computational components. We first concatenate multiple intermediate state representations to create a comprehensive feature set. We then train a ridge regression model on this high-din titt-iienniinall space, where the learned weights (3) indicate each feature's importance for pre- dicting brain activity. To balance model complexity with interpretability. we select the top-k features with the largest absolute values (where i set to Diwstel and train a final refined model. The fea- ture selection and refined model are completed all in the training set. This two-stage approach offers dual benefits: (1) improved prediction performance through feature combination, and (2) enhanced interpretability by revealing which specific features from which states are most neurally relevant.
5.2 EXPERIMENTAL VALIDATION
We evaluate MindTransformer against two established baselines across auditory language cortex:
Standard Baseline: An encoding model using only the input hidden state, representing
the traditional approach in LLM-bbrin alignment studies (Antonello et al. 2023 Gokce &
Schrimpi 2025 Ren et al. 2025 Konnasse-Gahen & Pallier 2024 Sun & Moens 2023
Aw et al. 2024 Oota et al. 2025
2022 Aw & Toneva 2023).
Context Vector Baseline: An encoding model using the hea-head context vector, recently shown to be effective for brain alignment (Kumar et al. 2024).
Table [ demonstrates the substantial improvements achieved by both MindTraasfformr modes. Mode I, despite using only a single optimally-selected state, shows remarkable gains over both baselines, with particularly dramatic improvements in primary auditory regions. In Heschl's Gyrus, Mode achieves a correlation of 0.454, representing a 27.5% improvement over the standard base- line (0.356) and 25.1% 1% improvement over the context vector baseline (0.363). Mode 2 further enhances performance through multi-state integration, reaching 0.467 in Heschl's Gyrus a 31.0% improvement over the standard baseline.
8

---

Published as a conference paper at ICLR 2026
Table 1: Performance comparison encoding models across auditory and language cortex. Values
represent mean correlation (#std) computed across 21 LLMs with all transformer layers. The im- provement column (Imp.) shows relative gain from Standard Baseline to MindTransformer Mode 2.
Brain Region
Standard Baseline
Context Vector
Proposed
Proposed
Imp.
Baseline
(Mode 1)
(Mode 2)
Heschl's Gyrus
0.356 (#0.049)
0.363 (#0.049)
0.454 (+0.059)
0.467 (+0.056)
+31.0
Planum Temporale
0.341 (#0.048)
0.333 (+0.050)
0.418 (+0.060)
0.436 (+0.055)
+27.8
STG (anterior)
0.367 (#0.043)
0,351 (#0.044)
0.419 (+0.057)
0.435 (#0.051)
+18.7
STG (posterior)
0.423 (#0.057)
0,407 (+0.054)
0,462 (+0.044)
0.477 (+0.043)
+12.6
Auditory Average
0.372
0.363
0.438
0.454
+22.0
MTG (anterior)
0.342 (#0.072)
0,323 (+0.067)
0.357 (+0.070)
0,351 (+0.066)
+2.5
MTG (posterior)
0.356 (=0.101)
0,342 (+0.098)
0.368 (+0. 102)
0.367 (+0.104)
+3.2
MTG (lemp-occipital)
0.405 (#0.086)
0.392 (#0.087)
0.417 (#0.087)
0,415 (#0.086)
+2.5
IFG (pars opercularis)
0.396 (=0.052)
0.377 (#0.053)
0.408 (+0.053)
0.406 (+0.054)
+2.4
IFG (pars triangularis)
0.452 (20.058)
0.430 (20.059)
0.464 (+0.058)
0.465 (+0.058)
+2.9
Angular Gyrus
0.403 (40.062)
0,385 (20.062)
0.419 (20.061)
0.409 (=0.059)
+1.4
Language Average
0.392
0.375
0.406
0.402
+2.6
The contrast between auditory and language regions reveals the targeted effectiveness of our ap- proach. While auditory regions show an average improvement of 22.0%, language network regions show more modest gains averaging 2.6%. This regional specificity validates our computational neuroanatomy hypothesis: the identification of per-head query with RoPE as the optimal state for auditory processing (as shown in Appendix (D translates directly into substantial performance gains in these low-level sensory areas. Notably, in some language regions, Mode I slightly outperforms Mode (e.g., Angular Gyrus: 0.419 vs. 0.409), suggesting that for high-level semantic processing. single well-chosen state may be more effective than multi-state integration.
To contextualize the magnitude these improvements, consider that scaling LLMs from 270M to 123B parameters- 456-fold increase in model size typically yields correlation improvements of approximately 0.02-0.04 in auditory regions (Figure 5). In contrast, MindTraasffrmer Mode achieves improvements of 0.111 in Heschl's Gyrus and 0.095 in Planum Temporale through compu- tational neuroanatomy insights alone, without any increase in model parameters. This demonstrates that understanding the internal computational structure of transformers can yield gains exceeding those from massive scale increases by several folds, particularly in sensory processing regions where traditional approaches have struggled.
5.3 PER-SUBIECT ANALYSIS
To validate that our group-averaged findings are not artifacts of averaging, we analyzed the first five subjects of the Le Petit Prince dataset individually on Llama 3.2 IB. Consistent with group-level results, standard hidden states remain suboptimal, winning in less than 20% of voxels (Appendix Figure 7). MindTransffrmer reliably improves performance the individual level: Mode 2 yields a 21.9% gain in Heschl's 's Gyrus (r = 0.127 vs 0.104) and consistent boosts across auditory regions (Appendix TT). Furthermore, the intra-block hierarchy persists individually: as shown in Ap- pendix Figure D early cortical regions (HG/PT) map to the block's "entry" (RoPF-query) while association areas align with deeper FFN states, confirming that the computational-coria isomor- phism is robust to individual variation.
5.4 ROBUSTNESS ANALYSIS WITH BASELINE ADJUSTMENT AND CONTROLLED REGRESSORS
To verify that our results are driven by the specific information content of transformer states rather than statistical confounds, we implemented two rigorous controls. First, we addressed the concern that states with higher dimensionality (e.g., FFN) yield higher correlations solely due an increased
The result for all the 21 L.LMs demonstrated in Appendix Table

---

Published as a conference paper at ICLR 2026
Low-level sensory processing
High-level semantic processing
Mis Carus
Supper ur Tesrppoel Fatux
trivice Tisrtall Cyrus, pirs manguleri
Anguie Gynu
0.53
Mocel Nurn ly
Ulame-3
Correction (Best
Gemmas-3 3
0.50
Owen-
Migiral
0.45
Raaddard Dasetine
Mi-dMarrsfomm
107
15
10"
10
107
122
10'
107
10
10'
Model Size (Biliion Parameters)
Figure 5: Correlation improvement along model sizes for 5 families of LLMs. The improvement brought by MindTraasffommr in low-level sensory cortex like Heschl's Gyrus is substantially larger than scaling model size by 456 times (from 0.27B to 123B).
number of regressors. By restricting all intermediate states to a fixed dimensionality (D = 2048) via top-k feature selection, we confirmed that the original setup with slightly inflate the performance for FFN as shown in Appendix Figure 8 but the functional dissociation persists: restricted FFN states remain optimal for the language network, while lower-dimensional RoPE-enhanced states continue to dominate the auditory cortex (Appendix Figure and Table IT..
Second, we benchmarked performance against random and GloVe embedding baselines (Bonnasse- Gahot & Pallier 2024). Since fMRI signals possess inherent temporal structure tied to stimulus onsets, even random embeddings can yield non-trivial correlations by tracking these basic statistics. Adjusting for this baseline isolates the specific contribution of the L.MM's contextual processing. As detailed Appendix Tables To low-level sensory regions exhibit weak correlations with random embeddings se 0.026). Consequently, the baseline-adjusted improvement is substantial: Mind- Transformer outperforms the standard baseline by 29.2% (random-adjusted) and 46.0% (GloVe- adjusted) in Heschl's Gyrus. This confirms that the alignment is driven by the unique, context-aware structural dynamics captured by LLM, rather than generic onset tracking or static lexical features.
DISCUSSION AND CONCLUSION
This work introduces computational neuroanatomy systematic framework for understanding LLM-brain alignment. By dissecting the transformer block's 's internal computations and mapping them precise neuroanatomical structures, reveal that the standard practice of using single hidden states overlooks the rich, distributed neural information encoded throughout the transformer's computational pipeline. Our findings demonstrate that different intermediate states correspond to distinct brain systems with remarkable anatomical precision-from low-level sensory processing in Heschl's ' gyrus high-level integration in association cortices.
The discovery that RoPE specifically enhances alignment with the brain's auditory processing streams exemplifies how computational neuroanatomy can bridge artificial and biological intelli- gence, providing neurobiological validation for architectural design choices. Our TrnnTraanformer framework operationalizes these insights, achieving significant performance by intelligently com- bining reuroanatomically--eeeevaa features from across the transformer block.
These results have profound implications for both neuroscience and AI. For neuroscience, compu- tational neuroanatomy offers a new lens for understanding how the brain implements language pro- cessing, with L.LMs serving as explicit computational hypotheses. For AI, our findings suggest that brain-inspired architectural modifications-guuie guided by computational neuroanatomy- lead to more human-lik like and interpretable language models.
Future work should extend this framework to Vision Transformers (ViTs) and multimodal architec- tures. We hypothesize that the intra-block hierarchy- early attention states aligning with sensory inputs and FFNs with semantic objects-men also characterize the visual ventral stream. Mind- Transformer's 's ability to dynamically integrate these distinct computational components offers a promising path for revealing universal principles of intelligence across modalities and creatures.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported part by the Wayne Holman Endowed Chair and the Executive Vice President for Research at Georgia Tech.
REFERENCES
Badr AlKhamissi, Greta Tuckute, Antoine Bosselut, and Martin Schrimpf. Brain-like language pro- cessing via shallow untrained multihead attention network. arXiv preprint arXiv:2406. 15 2024.
Richard Antonello, Aditya Vaidya, and Alexander Huth. Scaling laws for language encoding models in IMRL Advances in Neural Information Processing Systems, 36:21895-21107, 2023.
Khai Loong Aw and Mariya Toneva. Training language models to summarize narratives improves brain alignment. In The Eleventh International Conference on Learning Representations, 2023. URL
net forum (it.rrmiii..aLL445555
Khai Loong Aw, Syrielle Montariol, Badr AlKhamissi, Martin Schrimpf, and Antoine Bosselut. Instruction-tuning Aligns LLMs to the Human Brain. In First Conference Language Modeling, 2024. URL
met/foouume : 6=nXNNOx4wbl
Laurent Bonnasse- and Christophe Pallier. IMRI predictors based on language models of increasing complexity recover brain left lateralization. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024. BtLhhtpss:/// /tttep:://ppenrevviee net forrmm?
1d=XF1 jpckk]
Charlotte Caucheteux and Jean-Rémi King. Brains and algorithms partially converge in natural language processing. Communications biology, 5(1):134, 2022.
Sandra Da Costa, Wietske der Zwaag, Jose P Marques, Richard SJ Frackowiak, Stephanie Clarke, and Melissa Saenz. Human primary auditory cortex follows the shape of Heschl's gyrus. Journal of Neuroscience, 1444)):4067-14475,, 2011.
Stéphane d' d'Asooli, Jérémy Rapin, Yohann Benchetrit, Hubert Banville, and Jean-Remi King. TRIBE: TRImodal brain encoder for whole-brain fMRI response prediction. In The Fourteenth International Conference Learning Representations, 2026. httthtttp:////ppp
net post/fose Eorumoiduddiooo........ biggtttggdg
Adrien Doerig, Tim C Kietzmann, Emily Allen, Yihan Wu, Thomas Naselaris, Kendrick Kay, and Ian Charest. High-level visual representations in the human brain are aligned with large language models. Nature Machine Intelligence, pp. 1-15, 2025.
Evelina Fedorenko, Po-Jang Hsieh, Alfonso Ceeto-Castnnón, Susan Whitfield- Gabbiilli,, and Nancy Kanwisher. New method for fMRI investigations of language: defining ROIs functionally in individual subjects. Journal of neurophysiology. 104(2):1177-1194 2010.
Changjiang Gao, Shujian Huang, Jixing Li, and Jiajun Chen. Roles of Scaling and Instruction Tuning in Language Perception: Model vs. Human Attention. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 13042-13053 13055, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 19.18653/v120223 hww.:///....ninnnnnnnn..... URL / ootpaa/tttooolo.../222..2..33
Findings sendingg--mnnlp... emm...mmpppp.... 8687
Gary H Glover. Deconvolution of impulse response in event-related BOLD fMRII. Neuroimage, (4):416-422) 1999.
Abdulkadir Gokce and Martin Schrimpf. Scaling Laws for Task-Optimized Models of the Primate Visual Ventral Stream. In Forty-second International Conference on Machine Learning. 2025. URL Mtl.hhttts:///penreee net/forummid-ae 61 61MMHY

---

Published as a conference paper at ICLR 2026
Ariel Goldstein, Zaid Zada, Eliav Buchnik, Mariano Schain, Amy Price, Bobbi Aubrey, Samuel A Nastase, Amir Feder, Dotan Emanuel, Alon Cohen, et al. Shared computational principles for language processing in humans and deep language models. Nature neuroscience, 25(3):369-380, 2022.
Ariel Goldstein, Haocheng Wang, Leonard Niekerken, Mariano Schain, Zaid Zada, Bobbi Aubrey, Tom Sheffer, Samuel A Nastase, Harshvardhan Gazula, Aditi Singh, et al. A unified acoustic-to- speech-to-languaee embedding space captures the neural basis of natural language processing in everyday conversations, Nature human behaviour, pp. 1-15, 2025.
Liberty S Hamilton, Yulia Oganian, Jeffery Hall, and Edward Chang. Parallel and distributed encoding of speech across human auditory cortex. Cell, (9:)4622-46339 2021.
Mark Jenkinson, Christian F Beckmann, Timothy EJ Behrens, Mark W Woolrich, and Stephen M Smith. Fsl. Neuroimage, 62(2):782-790, 2012.
Carina Kauf, Greta Tuckute, Roger Levy, Jacob Andreas, and Evelina Fedorenko. Lexical-semantic content, not syntactic structure, is the main contributor to ANN-brain similarity of fMRI responses in the language network. Neurobiology of Language, 5(1):7-42, 2024.
Sreejan Kumar, Theodore R Sumers, Takateru Yamakoshi, Ariel Goldstein, Uri Hasson, Kenneth A Norman, Thomas L Griffiths, Robert D Hawkins, and Samuel A Nastase. Shared functional spe- cialization in transformer-basee language models and the human brain. Nature communications, 15(1):5523, 2024.
Mathis Lamarre, Catherine Chen, and Fatma Deniz. Attention weights accurately predict lan- guage representations in the brain. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 4513-4529, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10. 2006:35/1/2222..mnnnn---mmmm
URL https: httpss:/accaataclooy..ook
Sindings-emn.n 3307
Jixing Li, Shohini Bhattasali, Shulin Zhang, Berta Franzluebbers, Wen-Ming Luh, R Nathan Spreng, Jonathan R Brennan, Yiming Yang, Christophe Pallier, and John Hale. Le Petit Prince multilingual naturalistic fMRT corpus. Scientific data, (1):530, 2022.
Juliette Millet, Charlotte Caucheteux, Pierre Orhan, Yves Boubenec, Alexandre Gramfort, Ewan Dunbar, Christophe Pallier, and Réan-Rémi King. Toward a realistic model of speech processing in the brain with self-supervised learning. In Advances in Neural Information Processing Systems, volume 35, pp. 8765-8779, 2022.
Subba Reddy Oota, Jashn Arora, Veeral Agarwal, Mounika Marreddy, Manish Gupta, and Bapi Surampudi. Neural Language Taskonomy: Which NLP Tasks are the most Predictive of fMRI Brain Activity? In Marine Carpuat, Mathe-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3220-3237, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10. 11218665l/122222naaal main. 235. URL https:://acaaattolooooeeoo www.:/oooooorooooooooooo
.naccl main. 235/
Subba Reddy Oota, Akshett Rai Jindal, Ishani Mondal, Khushbu Pahwa, Satya Sai Srinath Namburi GNVV, Manish Shrivastava, Mancesh Kumar Singh, Bapi Raju Surampudi, and Manish Gupta. Correlating instruction-tuning (in multimodal models) with vision-language processing (in the brain). In The Thirteenth International Conference on Learning Representations, 2025. URL net
https: http:///eeeeeeeeeeeeeeeooerreeee
Yuqi Ren, Renren Jin, Tongxuan Zhang, and Deyi Xiong. Do Large Language Models Mirror Cognitive Language Processing? In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert (eds.), Proceedings of the 31 Interna- tional Conference Computational Linguistics, pp. 2988 00088-001,,400 Abu Dhabi, January 2025. Association for Computational Linguistics. URL ht https://aclant/hloog wwwp:tooo2oooy...22.......
col ing-ma 2017
12

---

Published as a conference paper at ICLR 2026
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A Primer in BERTology: What We Know About How BERT Works. Transactions of the Association for Computational Linguistics, 8: 842 866, 2020. doi: 10.1162/tacl 00349. httpsss//alaaattooorryoooooo tacl -1.411
Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal A Hosseini, Nancy Kan- wisher, Joshua Tenenbaum, and Evelina Fedorenko. The neural architecture language: In- tegrative modeling converges on predictive processing. Proceedings of the National Academy of Sciences, (1845))e2105646 18, 2021.
Jingyuan Sun and Marie-Francinn Moens. Fine-tuned vs. prompt-tuned supervised representations: which better account for brain language representations? In Proceedings of the Thirty- International Joins Conference on Artificial Intelligence, IJCAI '23, 2023, ISBN 978- 1-956792- 2003/
03-4. doi: 10.24963/jjcai 20235//77. URL https: doi http:://dooooogggg 22....24.33..
577
lan Tenney, Dipanjan Das, and Ellie Pavlick. BERT Rediscovers the Classical NLP Pipeline. In Anna Korhonen, David Traum, and Lluis Märquez (eds.). Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pp. 4593-4601, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 1t.1186/v1//1911111 1452. URL https: wac.lanchooog..rrgg//
Mariya Toneva and Leila Wehbe. Interpreting and improving natural-languee processing (in ma- chines) with natural language-processing (in the brain). Advances in neural information process- ing systems, 32, 2019.
Greta Tuckute. Aalok Sathe, Shashank Srikant, Maya Taliaferro, Mingye Wang, Martin Schrimpf, Kendrick Kay, and Evelina Fedorenko. Driving and suppressing the human language network using large language models. Nature Human Behaviour, 8(3):544-561, 2024.
A ARCHITECTURAL PARAMETERS OF VARIOUS LLM FAMILIES
Architectural parameters of various LLM families are provided in Table
Table 2: Architectural Parameters of Various LLM Families.
Family
Model Variant (Model size)
D Mmodel
N
Nkv
Dhead
D
Naaer
Llama 3.2 Instruct (IB)
2048
32
8*
64
8192
16
Llama 3.2 Instruct (3B)
3072
8*
128
8192
Llama
Llama 3.1 Instruct (8B)
4096
32
8*
128
14336
32
Llama 3.3 Instruct (70B)
8192
8*
128
28672
80
Qwen3 (0.6B)
1024
16
8*
128
3072
28
Qwen3 (1.7B)
2048
8*
128
6144
28
Qwen3 (4B)
2560
32
8*
128
9728
36
Qwen
Qwen3 (8B)
4096
8*
128
12288
Qwen3 (14B)
5120
40
8*
128
17408
40
Qwen3 (32B)
5120
8*
128
25600
64
Mistral 7B Instruct v0.2 (7B)
4096
32
8*
128
14336
32
Mistral 7B Instruct v0.3 (7B)
4096
32
8*
128
14336
32
Mistral
Mistral Small Instruct (22B)
6144
48
8*
128
16384
56
Mistral Large Instruct (123B)
12288
8*
128
28672
GPT-oss (20B)
2880
g*
64
2880+
GPT
GPT-oss (120B)
2880
64
8*
64
2880+
36
Gemma 3 Instruct (270M)
640
4
1*
256
2048
18
Gemma 3 Instruct (1B)
1152
1*
256
6912
26
Gemma
Continued page
13

---

Published as a conference paper at ICLR 2026
Table continued from previous page
Family
Model Variant
Dmodel
Ng
Nev
Dhead
Dttn
Mayer
Gemma 3 Instruct (4B)
2560
8
4*
256
10240
34
Gemma 3 Instruct (12B)
3840
16
8*
256
15360
48
Gemma 3 Instruct (27B)
5376
32
16*
128
21504
62
Model uses GQA or MQA, where NR. < Ng.
Value is per expert in Mixture-oo-EEppetss (MoE) model.
B TRANSFORMER STATES AND TENSOR SHAPES
Transformer states and tensor shapes are provided in Table
Table 3: Complete list of intermediate states extracted from transformer blocks with their dimensions and descriptions
Stage
State Name
Tensor Shape
Description
Input Hidden State
(B, S. Dunde)
Output from the previous block
Block Input
Pre-Attention Normalized
(B, Dustal
Output of LayerNorm
Per-Head Query (Q)
(B, No. S. Docal)
Q projection output
Per-Head Key (K)
R. Neu S. Draa)
projection output
Per-Head with RoPE
B. Ng, S. Doea))
projection output with RoPE upplied
Attention Mechanism
Per-Head K with RoPE
(B, Nov, S, Dresa)
K projection output with RoPE applied
Per-Head Value (V)
(B, Now: S. Dheaa)
projection output
Per-Head Context Vector
No. S, Dreas)
Context vector after attention mechanism
Combined Attention Output
(B,S. Duodel)
Output of the final attention projection
Post-Atiention Hidden State
Dunta)
Sum from first residual connection
Pre-FTUN Nornulized State
[B, S. Duudal
Output of the second LayerNorm
Residuals
FFN Activated State
[B, S, Dmi)
Output of FFN up up-prjecttio
FFN Ouiput
(B, S, Dunta)
Output of FFN down-pnijcction
C REGIONS OF INTEREST DEFINITION AND CHARACTERISTICS
Regions of interest definition and characteristics are provided in Table 4
Table 4: Anatomical and functional characteristics of regions of interest (ROIs) used in our analysis
ROI
Vosel Count
Auatomical Coverage
Constituent Parcels
Functional Significance
Whole-Brain Voxels
25,870
Complete contical and subcortical coverage within the IMRI noquisition field of view
All 48 cortical regions from the Harvand-Oxford atlas plus suh- certical stracturcs
Comprehensive neural pracess- ing. serving huseline all sensory, motor, and higher-order association cortices
Fedorenko Language
1,740
Distributed, stroglyy lef-llteraaliee nomlo-temporal pariotal regions de- lined by langungessclocive Lune-
Orbital inferior froetal (IFGorb)
High-level language compre- bension, including semantic synlactic parsing, dis- course integration, abstract
Network
Inferior froetal gyrus (IFG)
Middle frontal
(MFG)
Anterior temporal (An(Temp)
Posterior temporal (Pos/Temp)
linguistic reasoning
Angular gyrus (AngGyt)
Anditory-Sensory
325
superior temporal regions surrounding the Sylvian fissure, in- cluding primary and secondary audi- tory corlex
Gynu (HG)
Low-level acoustic feature traction, spestrotemporal analy- pitch processing, and early stages speceh perception
Cortex
Planum Tempomle (PT)
Superior Temporal Gynus, anterior division (ST (ipst))
Superior Temporal Gynus, posterior division (STGamc)
The three ROIs represent a hierarchical organization of language processing in the brain. The whole- brain ROI (25,870 voxels) provides an unbiased view of all cortical and subcortical processing. The Fedorenko language network (1,740 voxels) (Fedorenko et al.) 2010) represents domain-specific, high-level language regions identified through functional localizers, showing selective responses to linguistic versus non-ling non-linguistic stimuli. The auditory cortex (325 voxels) captures early, stimulus-driven processing of acoustic and phonological features critical for speech perception. This hicrarchical organization allows us to trace how different transformer components align with the progression from sensory semantic processing in the human brain.
14

---

Published as a conference paper at ICLR 2026
Table 5: Performance comparison across major brain networks. ( standard deviation across models).
Values represent mean correlation
Brain Region
Standard Baseline
Context Vector Baseline
Optimal State
Whole Brain
0.275 (1:0.011)
0.280 (1000114
0.296 (10.011)
Language Network
0.433 (+:0.016)
0.429 (+.00199)
0.450
Audio Cortex
0.407 (#:0.010)
0.416 (=00021)
0.475 (1+00015)
D STATE-WISE WINNING RATIOS ACROSS BRAIN REGIONS
To provide comprehensive view of how different intermediate states dominate various brain regions, we analyzed the winning ratios (percentage of voxels where each state provides the best en- coding performance) for three distinct regions: whole-brain, auditory-sensory cortex, and Fedorenko language network, in Figure
Winning ratio each intormediate (Over whole brain voxels)
Winning ratlo each intormediate (Over Language Network)
Winning ratio of each intermediate
(Over audio-sensory cortex)
State
Ratio
State
Ratio
Ratr (60
29
Ourput
0.00%
FFN Cuput
0.04%
PPN Aikatte Baie
14.80%
FPN Adtiaied Sate
19 13.00%
ARNAchaddd State
Pre-T
4.79%
PT
7.80%
Nomiciied Stale
Stzte
State
Psal-Ubardon dolen
State
0.76%
Ourjut
10.53%
Ourput
3.40%
Combones Afamntnn
Per-Haad Contoo Vecior
14.31%
Par-Head Conteat
Sector
8.20%
Verttr
Planthaa Value
2.44%
Par-Theed Vadue
Per-lead Value
0.00%
8.42%
Poi-tead RoPE
Pan-lead with RoPt
7.83%
Re-lltned with Rupt
9 82%
Pentlean with Rart
88%
Pen-ttee Kay
14.56%
Pe--itedd Kay (X)
16.04%
For Head Key [X
2.30%
13.03%
Nerna lood Stute
Pio Aluntion
3.00%
Nomalles and Blute
Pre-Aac fion
5.34%
Frevidtation hommulaed Slute
1.55%
inp Mooee Stack
2.34%
Inguit Hödden Stane
Input H4 oiless Ston
(a) Whole-brain voxels
(b) Language network
(c) Auditory-sensory cortex
Figure Winning ratio distributions of intermediate states across different brain regions. The dominance of different states varies dramatically by region, with FFN states being more prominent in language networks and per-head query with RoPE dominating in auditory cortex.
Tables
and provide detailed statisties of winning ratio across all five model families.
Table 6: Per-LLMMffamily winning ratio for whole-brain voxel (%)
State
Mistral
GPT-oss
Llama-3
Qwen-3
Gemma-3
Input Hidden State
1.22
2.54
1.40
0.66
2.61
Pre-Attn Norm State
2.36
8.17
3.46
2.40
2.03
Per-Head Q
13.92
10.55
12.10
14.18
12.69
Per-Head K
13.18
8.01
14.03
16.14
16.82
Per-Head Q w/ RoPE
6.16
4.48
6.87
8.31
10.68
Per-Head K w/ RoPE
3.85
4.12
5.13
9.88
6.27
Per-Head V
2.19
5.49
2.18
2.23
1.88
Per-Head Context
17.80
22.98
12.85
12.82
11.01
Combined Attn Output
13.09
9,82
13.83
11.51
6.19
Post-Attn Hidden State
1.38
5,94
2.93
1.30
0,00
Pre-FFN Norm State
2.92
13.78
3.79
3.37
5.20
FEN Activated State
16.98
0.00
15.60
13.31
20.36
FEN Output
4.96
4.12
5.81
3.88
4.26
15

---

Published as a conference paper at ICLR 2026
Table 7: Per-LLM-family winning ratio for Fedorenko language network (%)
State
Mistral
GPT-oss
Llama-3
Qwen-3
Gemma-3
Input Hidden State
1.77
1.35
1.70
0.37
3.16
Pre-Attn Norm State
4.90
14.89
5.93
3.48
3.64
Per-Head Q
24.91
12.13
17.56
24.04
13.91
Per-Head K
15.47
2.39
15.63
18.63
21.70
Per-Head Q w/ RoPE
6.18
5.14
7.83
13.97
11.22
Per-Head K w/ RoPE
1.75
1.58
7.43
7.08
2.87
Per-Head V
0.14
1.26
0.09
0.41
0.44
Per-Head Context
12.30
5.87
Combined Attn Output
4.34
1.24
3.82
3.76
2.99
Post-Atta Hidden State
1.55
9.86
4.81
0.95
0,00
Pre-FEN Norm State
5.46
32.39
8.64
4.25
2.72
FFN Activated State
20.07
0.00
19.07
16.59
30.68
FFN Output
1.15
0.92
1.32
0.39
0.79
Table 8: Per-LLMMfamily winning ratio for auditory-sensory cortex (%)
State
Mistral
GPT-oss
Llama-3
Qwen-3
Gemma-3
Input Hidden State
0.62
0.15
0.31
0.05
0.86
Pre-Attn Norm State
2.00
3.85
1,60
Per-Head Q
17.38
7.69
7.62
5.28
3.45
Per-Head K
3.38
0.00
0.85
2.67
3,08
Per-Head Q w/ RoPE
66.77
42.15
72.85
84.31
80.55
Per-Head K w/ RoPE
4.38
0.00
10.54
4.26
4.98
Per-Head V
0.00
0.00
0.00
0.10
0.00
Per-Head Context
1.23
29.85
0.08
0.31
0.37
Combined Attn Output
0.00
0,00
0.00
0.10
0.31
Post-Attn Hidden State
0.92
4.31
0.69
0.15
0.00
Pre-FFN Norm State
1.15
0.31
0.18
FEN Activated State
2.15
0.00
3.15
2.10
4.49
FFN Output
0.00
0.15
0.00
0.00
0.12
The stark differences in winning ratios across brain regions reveal distinct computational prefer- ences. In the language network (Table 7 shows a more distributed pattern with FFN Activated State and Per-Head Q/K states sharing dominance, reflecting the complex, multi-faceted nature of language processing. In contrast, the auditory-sensory cortex (Table B), Per-Head Q with RoPE dominates overwhelmingly. winning in 66.77-84.314 of voxels across model families, highlight- ing its critical role in low-level sensory processing. These regional specializations provide strong evidence for our computational neuroanatomy framework, demonstrating that different transformer components align with functionally distinct brain systems.
E QUANTIFICATION OF COMPUTATIONAL DEPTH
To quantitatively compare the processing hierarchy of the biological brain with the internal compu- tational flow the transformer block, we introduce method to map discrete transformer states onto continuous hierarchical axis. We utilize consistent terminology where Computational Depth refers to the normalized position of specific intermediate state, and Weighted Computational Depth refers to the aggregate metric for brain region.
State Selection and Normalization. While we extract 13 intermediate states for general analysis, for the specific purpose of quantifying the intra-block hierarchy, we define a focused subset of states S that represents the core computational trajectory. We identify the Per-Ilead Query as the functional entry point for the hierarchy, as it is the first layer consistently exhibiting alignment with early sensory cortices. We exclude the initial input and pre-attention normalization states as they precede this functional entry point. Additionally, we exclude Key and Value states to avoid redundancy, they occupy the same topological layer Queries but exhibit weaker alignment.
16

---

Published as a conference paper at ICLR 2026
The resulting ordered set S consists of the following 8 states:
1. Per-head Query
2. Per-head Query w/ RoPE
3. Per-head Context Vector
4. Combined Attention Output
5. Post-Attention Hidden State
6. Pre-FFN Normalized State
7. FFN Activated State
8. FFN Output
We assign a normalized Computational Depth score d(s;) E [0, 1] each state s; S. The depth is calculated as:
8(s,) S -1
(1)
where S 8. This establishes a linear coordinate system where the Per-head Query corresponds to and the FFN Output corresponds to d = 1.
Weighted Computational Depth. To map brain regions onto this axis, we compute the Weighted Computational Depth DR for given Region of Interest (ROI) R. For each voxel within the region, we first identify the optimal state s*(v) from the set S that yields the highest encoding correlation. We then compute the average depth across all voxels in the region:
DR
Σ (()(w)) DEVR
VR
This metric represents the "center of gravity" of alignment for brain region within the transformer block's 's computational order. In Figure we correlate this metric with the cortical hierarchy of brain regions along the audio stream and the language network.
Par-subject winning ratio (Over whole-brain voxels)
Per-subject winning ratio (Over language network)
Per-subject winning ratio (Over audio-sensory cortex)
State
State
FFN Output
13.2%
Culput
FFN Output
FFN Autivated Stute
6.1%
Activated
Activate Btura
1.2h
Normalized Stare
Pre-FEN
2.%
Normekaad State
Pre FFN
2.0%
Stade
Hidsen
1.%%
Post-Ata Hildon
Post Attartion Hidden
State
Combaned
Dutput
Combined Attention
Oudput
5.3%
Combined Altention
Outpul
5.%
Por Huad Contect
Vestor
7.%4
Vector
Doment
5.0%
Hor-
Per-Hep He--Hedd Viluue
1.5%
Per-Heed with
57%
Pa--Head
Per-Head with RuPE
Per-Head RaPE
Per-fferd Key (K]
Key
Head Key
Par-Head Quary (O)
0.9%
Query
Query
Nommalized State
Pro Altention
2.0%
4.0%
PrevAtbetion
Inpur Hidden Statte
4.27
Input
Inpul Hikdien
(a) Whole-braie voxels
(b) Language network
(c) Auditory sensoory cortex
Figure 7: Per-subject winning ratio distributions of intermediate states across different brain regions over Llama 3.2 IB. The dominance of different states varies dramatically by region, with FFN states being more prominent in language networks and per-head query with RoPE dominating auditory cortex.
17

---

Published as a conference paper at ICLR 2026
FFN Dimensionality Comparison (Absolute)
Configuration
FFN (8192 features)
FFN (2048 features)
Pearson Correlation
0.15
0.10
0.05
0.00
Hesch'ss Gyrus
Planum Temporale
STG (anterior)
STG (poaterior)
MTG (enterior)
MTG (postertar)
MTG (temp comp-acciiital)
IFG (pars operculeris)
IFG tears triangularis
Angular Gyrus
(a) FFN Activated State Comparison
Mode
Performance: FFN 8192 vs FFN 2048 (Absolute)
0.25
Configuration
Mode 1 (FFN 8192)
0.20
Mode (FFN 2048)
Pearson Correiation
0.15
0.10
0.05
0.00
Hescht's Gyrus
Planum Temporale
STG (anterior)
STG (postorior)
MTG (enterior)
MTG (postarior)
MTG (lemp- (lemp-occipp accipitl)))
IFG (pars operculeris)
IFG (pars riangularts)
Angular Gyrus
(b) MindTransformer Mode Comparison
Figure 8: Impact of dimensionality control on alignment performance (Llama-3.2 IB). (a) Restrict- ing the FFN activated state from its native dimensionality (8192) to controlled size (2048) via top-k feature selection results a minor relative performance degradation no more than 5.19% indicating that the high feature count is not the primary driver of alignment. (b) For MindTrransformer Mode 1, the degradation is negligible (<1.19%), demonstrating the framework's 's robustness: even when FFN performance dips slightly, the optimal selection from the remaining 12 states maintains high alignment accuracy.
Table 9: Per-subject performance comparison across major brain networks. Values represent mean correlation (# for standard deviation across 5 subjects).
Brain Region
Standard Baseline
Context Vector Baseline
Optimal State
Whole Brain
0.098 (100019)
0.103 (10.018)
0.119 (10.019)
Language Network
0.187 (+:0.334)
0.184 (+:0.030)
0.202 (+0.0344)
Audio Cortex
0.162 (+.0.199)
0.163 (+.0.019)
0.189 (++00022
18

---

Published as a conference paper at ICLR 2026
FFN output
Weighted Computational Depth
FFN activated state
Pre-FFN norm state
Post-Attn hidden state
Combined attn output
Per-head context
R2=0.96
Slope=-0.01
Per-head Q w/ RoPE
R2=0.99 Slope=0.69
Per-head Q
HG
PT
STG
MTG
IFG
AG
Cortical Hierarchy
Figure 9: Per-subject analysis of Weighted Computational Depth versus Cortical Depth (Llama-3.2 IB). We reproduce the hicrarchy analysis for the first five subjects individually. Consis- tent with the group-level observations (Figure B). each subject exhibits robust topological align- ment: early auditory regions (HG, PT) map the "entry" layers of the transformer block (specifi- cally per-head query with RoPE), while higher-order association areas (IFG, Angular Gyrus) align with deeper FFN states. The characteristic steep slope the auditory stream followed by plateau in the language network is preserved across individuals, confirming that this computational-cortical isomorphism is not an artifact of group averaging.
19

---

Published as a conference paper at ICLR 2026
Table 10: Per-subject baseline correlation comparison across auditory and language regions. Values
represent mean correlation (#standard deviation across the mean correlation of 5 subjects).
Brain Region
Random Baseline
Glove Baseline
Auditory Cortex
Heschl's Gyrus
0.026 (+0.025)
0.054 (=0.027)
Planum Temporale
0.043 (#0.026)
0.068 (=0.027)
STG (anterior)
0.051 (#0.027)
0.092 (#0.025)
STG (posterior)
0.059 (#0.044)
0.119 (=0.041)
Auditory Average
0.045
0.083
Language Network
MTG (anterior)
0.032 (#0.022)
0.067 (=0.032)
MTG (posterior)
0.039 (#0.027)
0.079 (#0.039)
MTG
0,053 (#0.037)
0.117 (=0.034)
IFG (pars opercularis)
0.046 (#0.027)
0.086 (=0.037)
IFG (pars triangularis)
0.061 (=0.031)
0.123 (#0.042)
Angular Gyrus
0.058 (#0.038)
0.101 (=0.041)
Language Average
0.048
0.096
Table 11: Per-subject raw correlation performance comparison of encoding models across auditory and language regions. Values represent mean correlation (tstandard deviation across the mean cor- relation of subjects). The improvement column (Imp.) shows the relative gain from the Standard Baseline to the proposed method (Mode 2 for auditory regions, Mode for language regions).
Brain Region
Standard Baseline
Context Vector
Proposed (Mode 1)
Proposed (Mode 2)
Imp. (%)
Baseline
Auditory Cortex
Heschl's Gyrus
0.104 (+0.012)
0.096 (+0.013)
0.125 (+0.009)
0.127 (+0.015)
+21.9
Planum Temporale
0.131 (#0.024)
0.124 (#0.024)
0.144 (+0.021)
0.151 (+0.024)
+15.1
STG (anterior)
0.169 (#0.021)
0.160 (#0.020)
0.173 (+0.020)
0.181 (#0.022)
+6.6
STG (posterior)
0.218 (#0.039)
0.209 (#0.036)
0.225 (#0.036)
0.228 (+0.031)
+4.9
Auditory Average
0.156
0.147
0.167
0.172
+12.1
Language Network
MTG (anterior)
0.126 (#0.044)
0.121 (+0.042)
0.130 (+0.044)
0.128 (+0.042)
+3.0
MTG (posterior)
0.143 (#0.044)
0.138 (+0.042)
0.145 (+0.044)
0.143 (+0.043)
+1.5
MTG ((emp-occipital)
0.184 (#0.047)
0.176 (#0.044)
0.188 (+0.047)
0.182 (+0.047)
+1.8
IFG (pars opercularis)
0.158 (20.039)
0.150 (#0.036)
0.162 (+0.041)
0.159 (+0.043)
+1.9
IFG (pars triangularis)
0.206 (20.044)
0.190 (40.039)
0.207 (+0.044)
0.207 (+0.043)
+0.3
Angular Gyrus
0.183 (#0.053)
0.176 (#0.048)
0.188 (+0.054)
0.199 (#0.044)
+2.8
Language Average
0.167
0.159
0.170
0.170
+1.9
20

---

Published as a conference paper at ICLR 2026
performance comparison of encoding models across auditory and language regions. Values represent mean correlation (#standard deviation across the mean correlation of 5 subjects). The improvement column (Imp.) shows the relative gain from the Standard Baseline to the proposed method (Mode for auditory regions, Mode for language regions).
Table 12: Per-subject randon-embedding-adiieiie
Brain Region
Standard Baseline
Context Vector Baseline
Proposed (Mode 1)
Proposed (Mode 2)
Imp. (%)
Auditory Cortex
Heschl's Gyrus
0.078 (#0.016)
0.070 (+0.017)
0.099 (+0.016)
0.101 (+0.019)
+29.2
Planum Temporale
0.088 (#0.022)
0.081 (+0.018)
0.101 (+0.017)
0.108 (+0.017)
+22.4
STG (anterior)
0.118 (#0.019)
0.109 (+0.020)
0.121 (+0.018)
0.132 (+0.023)
+12.0
STG (posterior)
0.159 (#0.029)
0.150 (+0.027)
0.166 (+0.021)
0.169 (+0.028)
+6.7
Auditory Average
0.111
0.103
0.122
0.128
+17.6
Language Network
MTG (anterior)
0.095 (#0.023)
0.089 (#0.021)
0.098 (+0.023)
0.096 (+0.021)
+4.0
MTG (posterior)
0.104 (=0.023)
0.099 (20.020)
0.106 (10.021)
0.103 (20.019)
+2.1
MTG (temp-occipital)
0.131 (#0.027)
0.123 (40.024)
0.135 (+0.025)
0.129 (40.028)
+2.6
IFG (pars opercularis)
0.112 (#0.045)
0.103 (#0.041)
0.115 (20.045)
0.113 (#0.047)
+2.7
IFG (pars triangularis)
0.145 (=0.020)
0.129 (#0.017)
0.146 (#0.019)
0.146 (+0.017)
+0.4
Angular Gyrus
0.125 (#0.031)
0.118 (+0.027)
0.130 (+0.030)
0.128 (+0.032)
+4.1
Language Averuge
0.119
0.110
0.122
0.119
+2.6
Table 13: Per-subject aooVe-embedding-djjieeee performance comparison of encoding models across auditory and language regions. Values represent mean correlation (#standard deviation across the mean correlation of 5 subjects). The improvement column (Imp.) shows the relative gain from the Standard Baseline to the proposed method (Mode for auditory regions, Mode for language regions).
Brain Region
Standard Baseline
Context Vector
Proposed (Mode 1)
Proposed (Mode 2)
Imp- (%)
Baseline
Auditory Cortex
Heschl's Gyrus
0.049 (#0.018)
0.041 (+0.020)
0.070 (40.019)
0.072 (+0.024)
+46.0
Planum Temporale
0.063 (#0.012)
0.056 (#0.009)
0.076 (#0.013)
0.082 (+0.014)
+31.4
STG (anterior)
0.077 (20.010)
0.068 (10.012)
0,080 (10.012)
0.085 (10.016)
+9.8
STG (posterior)
0.099 (20.007)
0.090 (40.008)
0.106 (+0.013)
0.109 (40.012)
+10.7
Auditory Average
0.072
0.064
0.083
0.087
+24.5
Language Network
MTG (anterior)
0.059 (#0.015)
0.054 (#0.017)
0.063 (+0.018)
0.061 (#0.017)
+6.3
MTG (posterior)
0.065 (=0.011)
0.059 (+0.009)
0.067 (+0.013)
0.064 (#0.014)
+3.4
MTG (temp-occipital)
0.067 (#0.022)
0.059 (+0.018)
0.071 (+0.020)
0.065 (+0.021)
+5.0
IFG (pars opercularis)
0.072 (#0.012)
0.064 (#0.008)
0.075 (+0.013)
0.073 (#0.016)
+4,2
IFG (pars triangularis)
0.083 (#0.017)
0.067 (#0.017)
0.083 (+0.018)
0.083 (+0.019)
+0.7
Angular Gyrus
0.082 (#0.016)
0.075 (+0.010)
0.087 (+0.018)
0.085 (+0.016)
+6.2
Language Average
0.071
0.063
0.074
0.072
+4.3
21

---

Published as a conference paper at ICLR 2026
Correlation Region (LPP)
H4
NTC
6141
0114
2391
630
10/8
138
6371
10034
10116
6000
10347
109
0312
1010
0335
1114
16100
6632
33280
20340
33300
Josl 1010
6339
0.00
3434
6364
835
9339
015
Ar Densskio
Auy eddiets
Avg De
Jag Garddda (1
Kag Ca
Figure 10: Average performance of intermediate states across the cortical hierarchy. Each subplot displays the mean Pearson correlation of the 13 intermediate states (averaged across all layers of Llama 3.2 IB) for a specific ROL. The results reveal a clear computational crossover: (1) early auditory regions (HG, PT), attention-related states speecifccall Per-head Query with PoPE-achieve the highest alignment, exhibiting a large relative improvement over the standard input hidden state. (2) As information progresses to the language network (IFG, AG), the FFN Activated State becomes the dominant predictor, though the relative performance delta here is more modest. This quantitative double dissociation validates the intra-block hicrarchy proposed in our computational neuroanatomy framework.
Table 14: Per-subject encoding performance averaged over 21 LLMs across auditory cortex regions using three metrics: raw correlation, random- ceriooddimbdddinngg adjusted, and GloVe adjusted. Values represent mean correlation (#standard deviation across 5 subjects). The improvement column (Imp.) shows the relative gain from the Standard Baseline to MindTraasfformer (Mode 2).
Brain Region
Standard
MindTransformer (Mode 2)
Imp. (%)
Baseline
Metric 1: Raw Correlation
Heschl's Gyrus
0.110 (+0.016)
0.135 (#0.020)
+22.7
Planum Temporale
0.135 (+0.025)
0.155 (+0.028)
+14.8
STG (anterior)
0.175 (+0.024)
0.185 (#0.028)
+5.7
STG (posterior)
0.225 (+0.040)
0.234 (#0.040)
+4.0
Metric 2: Gaaddo--Embeddding Adjusted
Heschl's Gyrus
0.084 (10.015)
0.110 (+0.018)
+30.9
Planum Temporale
0.092 (+0.023)
0.112 (#0.018)
+21.7
STG (anterior)
0.123 (+0.019)
0.136 (#0.024)
+10.5
STG (posterior)
0.166 (+0.027)
0.175 (#0.028)
+5.4
Metric 3:
Adjusted
Heschl's Gyrus
0.055 (+0.014)
0.081 (#0.022)
447.2
Planum Temporale
0.067 (+0.009)
0.087 (#0.008)
+29.8
STG (anterior)
0.082 (+0.013)
0.089 (#0.013)
+8.5
STG (posterior)
0.106 (#0.013)
0.115 (#0.014)
+8.4
22

---

Published as a conference paper at ICLR 2026
Table 15: Bootstrap significance analysis (FDR q 0.05) comparing the Standard Baseline and MindTransformer Mode 2. Values represent the percentage of significant voxels in the region, with the average correlation (r) of those significant voxels in parentheses. Best performing coverage is bolded.
Brain Region
Standard Baseline
Proposed (Mode 2)
Auditory Cortex
Heschl's Gyrus (includes HI and H2)
78.6% 00.11)
89.0% (r 0.166)
Planum Temporale
88.0% (r 00.162)
93.0% (r = 0..655
STG (posterior)
97.4% (r 0.227)
98.4% (r 0.236)
STG (anterior)
94.0% (r = 0.191)
95.7% (r 0.187)
Language Network
MTG (temporooccipital)
90.6% (r = 0.211)
94.9% (r = 0.199)
MTG (postcrior)
78.9% (r = 0.192)
82.4% (r = 0.170)
MTG (anterior)
82.5% (r 0.158)
86.2% (r 0.152)
IFG (pars opercularis)
90.6% (r 0.183)
89.9%
0.180)
IFG (pars triangularis)
96.8% (r 0.219)
98.5% (r 0.212)
Angular Gyrus
94.0% (r 0.197)
97.3% (r 0.196)
23


Reviews:
[{"review_id": "q9nfwjEYox", "reviewer": "Reviewer_NyvA", "summary": "The authors investigated whether internal computations inside transformer blocks, not just the usual layer hidden states, better align with human brain activity during naturalistic language comprehension, and whether specific architectural choices (notably RoPE) have distinct neurobiological correspondences. It introduces a “computational neuroanatomy” mapping from 13 intra-block states to brain voxels and proposes a feature-selection framework (MindTransformer) to exploit these states for brain prediction.\n\nfMRI from Le Petit Prince (English; 49 native speakers; ~100 minutes). Signals are group-averaged; ROIs include Harvard–Oxford parcels and a language localizer set. Twenty-one open-weight LLMs across 5 families are analyzed. For each transformer block, the authors extract 13 states. \n\nThe authors found that >90% of voxels in language/sensory regions are better predicted by previously unused intermediate states (vs. standard hidden/context). Early attention states align with low-level sensory cortex; later FFN states with association areas, revealing a fine-grained hierarchy within blocks (not just across layers).", "questions": "1. How do the “winning state” maps and MindTransformer gains look when models are trained/tested per subject (or with mixed-effects statistics) rather than on the group-average? Any meaningful variance across individuals? \n2. Do results persist with subject- or voxel-wise HRFs, or FIR bases capturing variable latencies, especially for early auditory regions where timing is critical? \n3. For the RoPE result, I wonder what would happen if you repeated the analysis with otherwise-identical models differing only in positional encoding (e.g., learned absolute vs RoPE) to isolate architectural effects from family-wise confounds.\n4. In the multi-state model, which specific features/states dominate the selected top-k set in different ROIs? A stability analysis (e.g., across folds and models) would clarify interpretability and guard against overfitting. \n5. The language-network average improvement is ~2–3%. Do certain layers/states (e.g., FFN activation vs key/query) dominate there, and can targeted combination rules outperform the generic concatenation + top-k approach?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The intra-block, 13-state decomposition is principled and broadly applicable; the “winning ratio” analysis shows strong evidence that representation choice dictates which brain systems one can model. \n2. The sharp auditory-stream improvement for per-head Q+RoPE (and the contrast without RoPE) is novel and interpretable, linking an computational component to neurobiology.\n3. MindTransformer is simple (ridge + feature selection) yet yields sizable, ROI-specific improvements that outstrip very large model-scaling gains—important for resource-efficient brain modeling.", "weaknesses": "1. The paper averaged BOLD time series over multiple participants, which boosts SNR but obscures individual variability and may inflate voxel-wise correlations; it’s unclear how robust the intra-block specializations are at the single-subject level. Ability to conduct analysis on individual subject is crucial for future application.\n\n2. The RoPE claim is framed as “neurobiological validation,” but evidence is correlational from encoding models; causal or ablation-style tests on models/representations (beyond contrasting with/without RoPE states) would strengthen the claim.\n\n3. Scope beyond language/auditory. The strongest effects are in low-level auditory cortex; improvements in classical high-level language ROIs are modest, and it’s not fully explored why FFN/attention integration doesn’t translate to larger gains there.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lw6N3HY0cF", "reviewer": "Reviewer_aPhw", "summary": "This paper explores how internal computations of transformer-based LLMs align with brain activity during naturalistic language processing. Rather than using the usual hidden states, the authors decompose each transformer block into 13 intermediate computational stages and test their correlation with fMRI responses from the Le Petit Prince dataset. They show that most brain voxels are better explained by these internal states than by the standard representations, that early attention computations align with low-level sensory areas while later FFN states align with association cortex, and that Rotary Positional Embeddings (RoPE) strongly improve alignment in auditory regions. Building on this, they propose MindTransformer, a framework that automatically selects or integrates the most brain-aligned states to improve prediction. The work is carefully executed and clearly presented. The methodology is solid and the figures are convincing; results seem plausible and the code availability supports reproducibility. The main limitation is the interpretation of correlation as evidence of shared computation remains debatable. Overall, this is a strong and well-written paper with interesting insights connecting transformer architecture and brain organization. While not revolutionary, it represents a meaningful methodological and interpretive step forward.", "questions": "Do you expect MindTransformer to generalise to vision ou multimodal brain datasets ?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This work goes beyond previous studies by going deeper into the architecture of Transformers, allowing better results.", "weaknesses": "The inherent method of comparing brain activity with transformer activity by correlation might be debatable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xXVxarf0pP", "reviewer": "Reviewer_6Yhf", "summary": "The paper investigates the alignment of different states in LLMs' (n=21) blocks to brain activity on the *Le Petit Prince fMRI Corpus*.\nThe main claims are that\n(1) the typical hidden states used by the field are suboptimal,\n(2) the stages within a block reveal a sensory-to-association hierarchy,\nand that (3) RoPE improves alignment with auditory streams.\nThe authors combine these findings into a brain model termed MindTransformer which uses a learned feature retrieval and for which they claim SOTA brain alignment.", "questions": "See Weaknesses above (Minor not that important).\n\nI am happy to increase my score if the generalization is shown, ideally on new datasets and without selecting states again.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 4, "strengths": "1. The study is systematic in evaluating various models, testing 21 LLMs that range in scale from 270M to 123B parameters and stem from various families including LLaMA, Qwen, Mistral, GPT, and Gemma.\n\n2. Improvements in brain alignment with RoPE seem solid and are novel as far as I am aware.\n\n3. The MindTransformer framework nicely ties together the selection of feature states for potentially improved brain predictivity.\n\n\nFurther strengths:\n* Figures are clear, text is straightforward to follow\n* The claim for a block-intermediate state hierarchy corresponding to the cortical hierarchy is interesting and, to me, unexpected. But I have concerns about its validity (see below)\n* I also highly appreciate papers that improve the state-of-the-art models on brain function, but I think this needs to be more rigorous (also below) \n\nSource code available which is great for reproduction!", "weaknesses": "### Claiming SOTA via training on test\nMy biggest concern is that the MindTransformer framework seems to select intermediate states on the test set itself. There is no mention of a separate training and validation split, such that the predictive representations are chosen via the test set. This would also yield high alignment for a random input feature set. There needs to be separate validation *after locking down the selected states*, which I think you could do via additional datasets (below).\n\n### Single dataset\nEvaluation on a single dataset is too limited for claiming SOTA. There are several publicly available language brain recordings, e.g. the set of Brain-Score Language benchmarks. What is the alignment of the (frozen) model on those?\n\n### Hierarchical claim not well supported\nFor Claim 2, I'm not actually sure the hierarchy is actually there. It's hard to tell from Figure 2, but for e.g. GPT-oss it seems that STG has a higher contribution of later block stages than the higher-level IFG and angular gyrus. Please quantify the actual correspondence between within-block hierarchy and cortical hierarchy, e.g. by correlating one with the other per model.\n\n\nMinor:\n* L043 related work should include Toneva et al. 2019, Schrimpf et al. 2021, Caucheteux et al. 2022, and Goldstein et al. 2022.\n* L318 \"Among all intermediate states examined in our computational neuroanatomy analysis, the per-head query with RoPE provides the most substantial and systematic improvement in brain alignment\" -- isn't the corresponding 13.03% winning ratio lower than e.g. the FFN activation at 14.86%?\n* Figure 3 / section 4.2: missing stats on RoPE improvements. Effect size seems pretty clear though.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yYRAQPc4vj", "reviewer": "Reviewer_c5Ks", "summary": "The authors dive deep into the transformer block, pulling out 13 intermediate states across 21 LLMs to map them onto fMRI brain activity during story listening. They claim hidden states are surprisingly insufficient (>90% of voxels do better with intermediates), and aligning intermediate states to brain activity reveals an intra-block hierarchy. They have also demonstrated the importance of RoPE. They wrap it up with MindTransformer, a two-stage selector that beats 456× scaling in A1.", "questions": "1.In the encoding model, statistical control is missing. Brain activity of each voxel is predicted by building a regression model, however, not all the predictions are significantly meaningful. It is expected to perform statistical control to determine the voxels whose neural signal can be successfully predicted by the encoding model. Is the correlation between predicted activity and ground-truth statistically above chance level? It also would be helpful to perform noise ceiling validation.\n\n2.In the experiment, fMRI data were averaged over subjects. This operation may help to decrease noise. However, it is unknown that the findings reported in this study are reproducible in different subject. The authors are encouraged to provide additional results for individual subject.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "It’s the first time someone’s looked at *all* intermediate states at this scale, and the intra-block hierarchy is neat. The RoPE-auditory link is interesting if it holds up. The scale is impressive—21 models, 13 intermediate states.", "weaknesses": "1.Some critical statistical rigor is missing. \n2.Averaging brain activities across subject may introduce information loss.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Lg8FQnSOcE", "reviewer": "Reviewer_RBfX", "summary": "This paper investigates how well each component of a transformer block (e.g., \"per-head query\", \"per-head query with RoPE\", \"FFN Activated State\", \"FFN Output\") provides representations to predict brain activity during natural language processing. The work systematically compares these components across a large variety of model sizes and model families. It finally shows that combining them provides better brain scores than simply using the commonly used hidden states.", "questions": "In brief (see Weaknesses section for more detail):\n- What are the results when comparing the different components using an equal number of features?\n- What are the raw values of the brain scores, and how do they compare across model sizes, families, and brain regions?\n- What are the correlations (and differences between conditions) yielded by proper baselines?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 1, "strengths": "- The work investigates a large number of models from five model families, with sizes ranging from 270M to 123B parameters. The two largest models, above 100B parameters, are relatively rare in this type of literature, and not easily computationally accessible for many researchers, which could be of interest to the community.\n\n- Another strength is the systematic approach the authors propose in studying all the different components of each transformer block.", "weaknesses": "There are several significant limitations that lead me to reject the paper.\n\n- The work compares different representations that do not have the same number of features, which leads to unfair comparisons. Indeed, all else being equal, the brain predictivity increases with the number of potential regressors. Here, all the components in a transformer block do not have the same number of features, and this can vary considerably: in particular, the hidden size of the MLP block (called \"ffn activation state\" here) is usually much larger than the hidden size (e.g., 3x for Qwen3 0.6B, 4x for LLama 3.2 1B, or even 5x for Qwen3 32B). Unsurprisingly, it is indeed this particular component that wins most of the times (see Fig 1 and Fig 5b). One solution to provide fair comparisons might be to equate the dimensions of all the vectors, either by applying a PCA or by randomly dropping some features. For the same reason, the fact that combining all the different features (which the authors call the \"Mind's Transformer\") leads to better results is not very surprising nor insightful at this stage. Moreover, previous work has investigated proper ways to combine various features: see Dupré la Tour et al. (2022).\n\n- Another important limitation of the work is the absence of proper baselines. Previous work (see e.g. Schrimpf et al., 2021; Pasquiou et al., 2022; Bonnasse-Gahot & Pallier, 2024) has shown that random baselines or untrained models can achieve pretty high brain scores. Here, it is necessary to compare the results with proper baselines such as random vectors, random embeddings, or untrained networks. For instance, in the auditory cortex, simply having random embeddings that can be used to track some acoustic property such as speech rate can be enough to actually yield a brain score that is as good as a pretrained LLM, given that the LLMs used here are fed with text rather than speech.\nThis is notably particularly important for the authors' main claim of improvements in brain predictivity in the primary auditory cortex (see also below). Random baselines should put these results in perspective, notably when comparing the different brain regions.\n\n- The work does not in general show all full values, but presents a winning ratio instead. This is quite underspecified, as one component could \"win\" but by a very small margin. This is important to assess whether it is really worthy to consider these different components. This issue, combined with the two previous ones, makes it impossible to extract insights from the current results. It would be better to simply (or also) present the raw values (provided the comparisons are fair and use proper baselines): brain scores as a function of size, for the different models, different families, different components, and possibly different brain regions.\n\n- One of the main claims of the authors is that their \"framework\" achieves improvements in brain predictivity in the primary auditory cortex that \"substantially exceed gains from 456$\\times$ model scaling\". This claim is problematic for several reasons. \nFirst, although the human participants hear the story, the LLMs are fed with a textual version. The fact that there is not a high brain score in the auditory regions is therefore not surprising. Note that this does not mean that this is an \"unresolved challenge in the field\" as the authors claim. Acoustic language models are indeed able to well predict brain activity in the auditory cortex (see Tuckute et al., 2024; Millet et al., 2022; Antonello et al., 2023). Moreover, comparing against model scaling in these areas does not make much sense, as low-level features are already well-predicted with small models, larger models providing only marginal gains in these sensory regions (see Antonello et al., 2023, for speech; see also similar findings by Raugel et al., 2025, for vision). Finally, the improvements might be based on using the onsets of the words, which helps modeling the acoustic envelope or speech rate, which might lead to an increase in brain score in the auditory cortex. In this view, it is then not surprising that the higher gain is obtained when considering the component just after RoPE, which allows the brain fit to take advantage of the position of the words. Once again, this underscores the need to compare with a proper baseline. The outcome might not be as important as the authors claim, and may be attributable to unsurprising low-level reasons.\n\nMinor comments\n\n- The abstract states that the fundamental unit of a large language model is the transformer block. However, not all LLMs are based on Transformers: see for instance models based on the Mamba architecture.\n\n- Fig. 4: I assume the size of the marker is related to the size of the LLM, but this information is already provided by the x-axis, so it does not bring anything new and obscures the reading of the figure.\n\n- Given the intention of the authors to systematically investigate all components of the transformer block, it might be interesting to also examine the representation before and after the gating for models such as Gemma with gated activation functions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)