Brain-like Functional Organization within Large Language Models

## Abstract

Abstract The human brain has long inspired the pursuit of artificial intelligence (AI). Recently, neuroimaging studies provide compelling evidence of alignment between the computational representation of artificial neural networks (ANNs) and the neural responses of the human brain to external stimuli, suggesting that ANNs may employ brain-like information processing strategies. While such alignment has been observed across sensory modalities—visual, auditory, and linguistic—much of the focus has been on the behaviors of artificial neurons (ANs) at the population level, leaving the functional organization of individual ANs that facilitates such brain-like processes largely unexplored. In this study, we bridge this gap by directly coupling sub-groups of artificial neurons with functional brain networks (FBNs), the foundational organizational structure of the human brain. Specifically, we extract representative patterns from temporal responses of ANs in large language models (LLMs), and use them as fixed regressors to construct voxel-wise encoding models to predict brain activity recorded by functional magnetic resonance imaging (fMRI). This framework effectively links the AN sub-groups to FBNs, enabling the delineation of brain-like functional organization within LLMs. Our findings reveal that LLMs (BERT and Llama 1–3) exhibit brain-like functional architecture, with sub-groups of artificial neurons mirroring the organizational patterns of well-established FBNs. Notably, the brain-like functional organization of LLMs evolves with the increased sophistication and capability, achieving an improved balance between the diversity of computational behaviors and the consistency of functional specializations. This research represents the first exploration of brain-like functional organization within LLMs, offering novel insights to inform the development of artificial general intelligence (AGI) with human brain principles.

## 1 Introduction

The human brain, with its unparalleled capacities in perception, cognition, reasoning, and creativity, stands as the pinnacle of biological intelligence and complexity (Sporns et al., ; Bassett & Gazzaniga, ). Understanding the mechanisms behind these cognitive abilities has been one of the most formidable challenges in neuroscience for decades (Brodmann, ; Hubel & Wiesel, ; Belliveau et al., ; Bear et al., ). Despite significant advances, the intricate processes through which the brain organizes and interprets information—transforming raw sensory inputs into meaningful representations that guide behavior and decision-making—remain elusive. Unraveling these complexities would not only deepen our understanding of human cognition but also lay the foundation for creating machines that truly emulate the intelligence of the human brain (Nilsson, ; Lu et al., ; Zhao et al., ).

In recent years, artificial neural networks (ANNs), initially inspired by the architecture of the human brain, have emerged as one of the most promising technologies in quest to build machines capable of cognitive functions. Modern ANNs have groundbreaking abilities in fields like visual recognition (He et al., ; Dosovitskiy, ; Oquab et al., ), language processing (Vaswani, ; Devlin et al., ; Brown et al., ), and even reasoning tasks—domains that were once considered exclusive to human intelligence. These networks, with their remarkable capacity to learn complex patterns from vast amounts of data, have ignited hope that machines may one day replicate or even surpass human cognitive abilities. Yet, as we edge closer to this possibility, one fundamental question lingers: Can artificial systems be powered by the same organizational principles that underpin human intelligence? This question becomes even more pertinent in light of recent advances in neuroimaging, which have revealed striking alignment between the computational representations of ANNs and the neural responses of the human brain. Through functional magnetic resonance imaging (fMRI), researchers have shown that ANN representations of external stimuli—whether visual (Zhao et al., ; Yamins & DiCarlo, ; Kriegeskorte, ), auditory (Zhou et al., ; Li et al., ; Millet et al., ; Tuckute et al., ), or linguistic (Liu et al., ; Caucheteux & King, ; Schrimpf et al., ; Oota et al., )—exhibit patterns that closely mirror patterns observed in the brain. This fascinating correspondence has led to a tantalizing hypothesis: that ANNs, in their computational architectures, might develop information processing strategies akin to those employed by the brain itself.

However, while these findings are compelling, most studies have focused on population-level comparisons between ANNs and brain activity. This offers a broad view of alignment but lacks the granularity required to uncover the functional organization within ANNs. In neuroscience, it is well established that the brain is composed of specialized regions, each responsible for distinct cognitive functions, which together form functional brain networks (FBNs) (Bassett & Bullmore, ; Power et al., ; Park & Friston, ). These networks are dedicated to processing specific types of information, such as visual input or linguistic content, and their interactions are essential for the brain’s overall functionality (Smith et al., ). Similarly, it is plausible that within ANNs, a similar form of organization may exist, where sub-groups of artificial neurons play specialized roles in processing information.

Building on this inspiration, our study seeks to explore the functional organization of ANNs by directly coupling sub-groups of artificial neurons with FBNs. We focus on large language models (LLMs), such as BERT (Devlin et al., ) and the Llama family (Touvron et al., ; ; Dubey et al., ), which are composed of vast numbers of artificial neurons capable of processing complex linguistic information. To achieve this, we extract representative temporal patterns from the activity of artificial neurons in LLMs and use them as fixed regressors in voxel-wise encoding models. These models enable us to predict brain activity recorded via fMRI, thus linking the sub-groups of artificial neurons in LLMs with their corresponding functional brain networks in the human brain. Our findings demonstrate that sub-groups of artificial neurons in LLMs align closely with the functional interactions within well-established brain networks. By analyzing the evolution of these relationships across four LLMs (BERT and Llama 1-3), we observe that the brain-like functional organization in these models becomes increasingly pronounced as their capabilities grow, achieving an improved balance between the diversity of computational behaviors and the consistency of functional specializations. This research is the first to characterize the brain-like functional organization within LLMs, providing keys insights that could shape the future development of brain-inspired AI systems and engineer brain-like intelligence.

## 2 Related Work

### 2.1 Neural Encoding of Computational Language Models

Neural encoding studies have demonstrated that computational language models based on deep neural networks exhibit considerable representational alignment to neural activity in the human brain  (Abdou, ; Schrimpf et al., ; Oota et al., ; Antonello et al., ). Most prior research has adopted linear encoding models to map between the computational representation of language models and neural responses elicited by the same set of stimuli. Although these studies have yielded promising results, they primarily focus on modeling the behaviors of ANs at the population level. Specifically, they utilize layer-level embeddings—comprising a collection of individual AN’s responses—to predict neural activity, thereby leaving the functional organization of individual ANs largely unexplored.

### 2.2 Interpreting Behaviors of Individual ANs

Researchers have developed various strategies to interpret the behaviors of individual ANs in computational language models  (Zhao et al., ). These strategies include feature attribution, probing, neuron activation analysis, attention visualization, adversarial example, and inverse recognition, among others (Wu et al., ; Zhang et al., ; Yeh et al., ; Wang et al., ). Recently, researchers have employed more advanced models such as GPT-4 to automate the interpretation of large scale individual ANs in less capable models such as GPT-2  (Bills et al., ). Singh et al. () similarly use LLMs to generate candidate explanations for text modules, such as a neuron in LLM, based on the n-grams that elicit the most activation from the neuron. Synthetic data is then generated based on these explanations, and the neuron’s activation to the data is assessed to identify the top candidate explanations. While these strategies provide valuable insights into our understanding of language models, the functional organization of individual ANs has rarely been explored. Furthermore, the behaviors of individual ANs have yet to be linked to neural response, leaving the question of whether the organization of ANs mirrors the functional structure and organization found in the brain inadequately addressed.

## 3 Methods

### 3.1 Overview

The study overview is illustrated in Figure 1. We begin by defining artificial neurons (ANs) in LLMs and quantifying their temporal responses to external stimuli $\mathbf{X}\in\mathbb{R}^{t\times n}$ (Figure 1a). Subsequently, we employ a sparse representation  (Mairal et al., ) scheme to learn a set of representative temporal response patterns, referred to as a dictionary $\mathbf{D_{\textit{AN}}}\in\mathbb{R}^{t\times k}$ (Figure 1b). Afterwards, we use the dictionary $\mathbf{D_{\textit{AN}}}$ as regressors to build voxel-wise encoding models to predict fMRI brain activity. The encoding coefficients associated with each atom reveal how that atom couples with functional activity of the entire brain (Figure 1c). By integrating this coupling relationship with the association between ANs and $\mathbf{D_{\textit{AN}}}$ established during learning of representative temporal responses, we infer brain-like functional organization in LLMs.

Figure: Figure 1: The study overview. We learn representative patterns $\mathbf{D_{\textit{AN}}}$ (b) from the temporal responses of ANs in LLMs (a) and use $\mathbf{D_{\textit{AN}}}$ as regressors to reconstruct fMRI brain activity recorded by fMRI (c). Atoms in $\mathbf{D_{\textit{AN}}}$ selectively activate specific brain areas/networks.
Refer to caption: extracted/5967503/Figures/overview.png

### 3.2 Artificial Neurons in LLMs and Their Temporal Responses

In this study, we focus on four LLMs: the pre-trained BERT model (Devlin et al., ), which serves as a foundational transformer-based language model, and three progressively advanced models from the evolutionary Llama family, Llama 1-3 (Touvron et al., ; ; Dubey et al., ). BERT, with its bidirectional encoder, is a widely recognized baseline model to capture rich, contextualized word representations. In contrast, the Llama models, employing a decoder-based architecture, represent a more advanced, contemporary approach, exhibiting superior performance across diverse tasks. Examining the evolution of these LLMs may offer insights into the development of functional organization within these models.

Building on the established definitions of Artificial Neurons (ANs) in large language models (LLMs)  (Bills et al., ; Samek et al., ), we define each neuron in the second fully connected layer of the feed-forward network within each transformer block as an individual AN (Figure 1a). In BERT, this applies to the encoder blocks, while in the Llama models, it applies to the decoder blocks. With this definition, the number of ANs corresponds to the dimensionality of the output embedding in each transformer block. For instance, BERT consists of 12 layers, yielding 9,216 ANs (12 layers × 768 dimensions per layer), whereas the Llama models, with 32 layers, define 131,072 ANs (32 layers × 4096 dimensions per layer).

Given a text input, the temporal responses of each AN are formally defined as it activations in response to the sequence of input tokens. The temporal responses of all ANs at layer $l$ can be readily obtained through the layer’s output $\mathbf{X}_{l}\in\mathbb{R}^{t\times n_{l}}$, where $t$ is the the number of tokens in the input sequence, and $n$ is the number of ANs at layer $l$, corresponding to the dimensionality of the output.

Additionally, it is critical to synchronize the temporal responses of artificial neurons (ANs) with the fMRI timeline. To achieve this, we align the text tokens with the corresponding fMRI volumes using the time-stamped word-level transcripts from the Narratives fMRI dataset  (Nastase et al., ). However an fMRI volume generally spans multiple text tokens, we follow common practice in brain encoding studies by averaging the ANs’ responses over these tokens within each fMRI time interval. This produces a temporal response curve that matches the length of the fMRI sequence. Finally, we convolve the temporal response curve of each AN with a canonical hemodynamic response function (HRF) implemented in SPM(^1^11https://www.fil.ion.ucl.ac.uk/spm/), to account for the hemodynamic delay inherent in fMRI recordings.

### 3.3 Representative Temporal Response Patterns of ANs

Identifying representative temporal response patterns of ANs is crucial for simplifying the analysis given the vast number of ANs in models like BERT and Llama. With thousands of ANs in each model (e.g., 9,216 in BERT and 131,072 in Llama), analyzing individual responses is not only impractical but also risks obscuring key trends due to factors such as noise and self-correlation among the ANs. In this study, we employ a sparse representation scheme  (Mairal et al., ) to learn a set of representative patterns from the temporal responses of the entire group of ANs.

Given the set of temporal responses $\mathbf{X}\in\mathbb{R}^{t\times n}$, where $n$ is the total number of ANs and $t$ is the length of the temporal responses, the objective is to find a sparse representation $\mathbf{A_{\textit{AN}}}\in\mathbb{R}^{k\times n}$ over a dictionary $\mathbf{D_{\textit{AN}}}\in\mathbb{R}^{t\times k}$, minimizing the reconstruction error while imposing a sparsity constraint on $\mathbf{A_{\textit{AN}}}$  (Mairal et al., ):

$$ $\min_{\mathbf{A_{\textit{AN}}}}\left\|\mathbf{X}-\mathbf{D_{\textit{AN}}} \mathbf{A_{\textit{AN}}}\right\|_{2}+\lambda_{AN}\left\|\mathbf{A_{\textit{AN} }}\right\|_{1}$ $$

where $\lambda_{AN}$ is a regularization parameter that controls the trade-off between reconstruction accuracy and sparsity of $\mathbf{A_{\textit{AN}}}$. In this context, $\mathbf{D_{\textit{AN}}}$ represents a set of basis vectors or atoms, which is the representative temporal patterns that capture the essential dynamics of the ANs’ temporal responses. The sparsity constraint ensures that each temporal response is characterized by only a few key patterns. By learning this dictionary, we can express the entire set of temporal responses $\mathbf{X}$ as a combination of these representative patterns, weighted by the sparse coefficients in $\mathbf{A_{\textit{AN}}}$.

### 3.4 Voxel-wise Encoding of fMRI Brain Activity

We construct voxel-wise encoding models to establish the relationship between ANs and brain activities. This approach allows us to determine how the temporal responses of ANs can predict or account for the neural signals captured by fMRI. The encoding models are based on a similar scheme to the one used for learning representative temporal response pattern. The key difference is that we fix the dictionary $\mathbf{D_{\textit{AN}}}$ to learn a sparse representation $\mathbf{A_{\textit{f}}}\in\mathbb{R}^{k\times N}$ for reconstructing the fMRI brain activity $\textbf{S}\in\mathbb{R}^{t\times N}$, where $N$ is the number of voxles:

$$ $\min_{\mathbf{A_{\textit{f}}}}\left\|\textbf{S}-\mathbf{D_{\textit{AN}}} \mathbf{A_{\textit{f}}}\right\|_{2}+\lambda_{f}\left\|\mathbf{A_{\textit{f}}} \right\|_{1}$ $$

Each row in $\mathbf{A_{\textit{f}}}$ indicates the importance of the corresponding atom of $\mathbf{D_{\textit{AN}}}$ in reconstructing fMRI brain activities at each voxel (Figure 1c). It is noted that the voxel-wise encoding models are constructed for each subject independently. A one-sample t-test over the entire population of subjects is conducted for each voxel to examine whether the encoding coefficient of a given atom $\mathbf{D_{\textit{AN}}}$ is above chance level (FDR corrected). Rather than simply showing voxel-level activations, this statistical map provides a spatial depiction of the brain regions linked to each representative pattern, offering a more interpretable perspective. For simplification, we refer to this as a brain map.

### 3.5 Relationship Inference between ANs and Brain Networks

The representative temporal response patterns $\mathbf{D_{\textit{AN}}}$ serve as a bridge between ANs and brain activity. Specifically, the $i^{th}$ AN can be associated with the $j^{th}$ atom of $\mathbf{D_{\textit{AN}}}$ where $\mathbf{A_{\textit{AN}}(\cdot,\textit{i})}$ is maximized. In this way, each atom in $\mathbf{D_{\textit{AN}}}$ corresponds to a subset of ANs. Simultaneously, the voxel-wise encoding models link each atom in $\mathbf{D_{\textit{AN}}}$ to specific brain regions, forming a brain map that typically spans multiple brain networks, such as auditory, language and visual networks. We utilize a network correspondence tool  (Kong et al., ) to automatically identify the brain networks involved in these brain maps by referring to the 17 FBNs reported previously  (Yeo et al., ). This approach allows us to infer the relationship between subsets of ANs and brain networks, revealing how these subsets and corresponding temporal patterns align with the brain’s functional architecture.

### 3.6 Implementation Details

We use the “Narratives” fMRI dataset  (Nastase et al., ) in this study. The fMRI data were acquired when human subjects listened to 27 spoken stories and released with various pre-processed versions. We use the AFNI-nosmooth version of one fMRI session, the “Shapes”, due to the high spatial resolution ($\rm 2\times 2\times 2mm^{3}$), adequate number of subjects (59 subjects) and the integrity of the narrative stimuli. The fMRI volumes before the onset and after the end of the story are discarded. The time courses of each voxel is normalized to have unit norm. For the LLMs, we use the pre-trained BERT (^2^22https://huggingface.co/docs/transformers/model-doc/bert) and Llama family (^3^33https://huggingface.co/meta-llama/) (Llama1-7B, Llama2-7B and Llama3-8B). In the sparse representation of ANs’ temporal responses, the dictionary size (k) is set as 64, and the sparsity constraint parameter $\lambda_{AN}$ is set as 0.15 for all the LLMs. The $\lambda_{\textit{f}}$=0.08 is used in the sparse reconstruction of fMRI activity.

## 4 Results

### 4.1 Sparse Representation of ANs and Brain Activity

The temporal responses of ANs can be effectively represented by the dictionary $\mathbf{D_{\textit{AN}}}$, as evidenced by the high $\mathrm{R^{2}}$ values shown in Figure 2(a). Among the Llama family, the $\mathrm{R^{2}}s$ values are comparable, with measurements of 0.5021±0.1119, 0.5005±0.1114 and 0.5032±0.1139, respectively. The BERT model demonstrates relatively higher $\mathrm{R^{2}}$ values (0.6005±0.1127) compared to the Llama family. This discrepancy may be attributed to the significantly smaller number of ANs in BERT compared to the Llama models, which results in lower diversity of temporal response patterns of the ANs and consequently improves the performance of dictionary learning.

The distribution of the $\mathrm{R^{2}}s$ vaules for the sparse reconstruction of fMRI brain activity across the four LLMs shown in Figure 2(b) indicates that the Llama family correlates more closely with brain activity compared to BERT, which is in accordance with their performance in natural language tasks. Meanwhile, the spatial distributions of the $\mathrm{R^{2}}s$ across the four LLMs (Figure 2c-f) show considerable alignment with one another. The brain regions with superior encoding performance encompass the superior and middle temporal lobes, lateral and medial occipital lobes, angular gyrus, anterior and posterior cingulate cortices, temporo-parietal junction, and wide spread areas in the frontal lobe. This spatial distribution pattern, as well as the range of $\mathrm{R^{2}}$, closely resembles the brain scores from previous studies on neural encoding of natural language processing models  (Antonello et al., ; Schrimpf et al., ; Caucheteux & King, ), validating the reliability of the voxel-wise encoding models in this study.

Figure: Figure 2: The $\mathrm{R^{2}}$ values in the sparse representation of temporal responses of ANs (a) and in the sparse reconstruction of fMRI activity (b) using $\mathbf{D_{\textit{AN}}}$. (c-f): The spatial distribution of the $\mathrm{R^{2}}s$ in BERT and Llama family visualized on the cortical surface, respectively.
Refer to caption: extracted/5967503/Figures/R2.png

Figure: Figure 3: (a) Two exemplar brain maps corresponding to the atom #9 and atom #17 in Llama3. (b) Automatic brain network labelling of atom #9, illustrating the activation of the language, salienceA and salienceB networks, and the deactivation of the lateral visual (LatVis) cortex.
Refer to caption: extracted/5967503/Figures/exemplarMaps.png

### 4.2 Brain Map Analysis

The brain maps reveal intricate functional interactions and competitions among well-established FBNs. Figure 3 shows two exemplar brain maps corresponding to atom 9 and atom 17 in Llama3 (Figure 3a), along with the automatic brain network labelling of atom 9 which exhibits the concurrent activation of the language, salienceA and salienceB networks, and the deactivation of the lateral visual (LatVis) cortex (Figure 3b). A comprehensive visualization for all the 64 brain maps across the four LLMs is provided in  A.1.

We observed notable variability in the involvement of FBNs in brain maps across different FBNs (Figure 4a). A subset of FBNs, including the LatVis cortex, language network, default mode network (DMN), working memory (WM) network, primary auditory cortex, salience network, fronto-parietal network (FPN) and dorsal attention network (DAN), are more frequently engaged in brain maps, with both positive (activation) and negative (deactivation) involvement. On the contrary, the meidal visual (MedVis) cortex, parietal memory (ParMemory) cortex, sensorymotor network (SMN), LimbicA and LimbicB are less frequently involved. Notably, the patterns of FBN engagement in brain maps are consistent across the four models.

In line with previous findings on neural language processing, our results highlight the engagement of functionally specialized brain regions/networks including the primary auditory cortex, visual cortex, language and FPN  (Friederici, ; Caucheteux & King, ; Schrimpf et al., ). More importantly, our results further underscore the importance of domain-general brain regions/networks in this process, particularly the DMN, WM network and DAN. These findings are consistent with previous neuroimaging studies using dynamic naturalistic stimuli (e.g., auditory stories and movies), which suggest that the DMN plays a key role in integrating incoming extrinsic information, temporarily stored in the WM, with prior intrinsic information over relatively long timescales to form context dependent models  (Yeshurun et al., ).

The brain maps also exhibit relatively complex functional interactions among FBNs. Specifically, most brain maps involve the concurrent activation or deactivation of multiple FBNs, as illustrated by the distribution of the brain maps associated with different number of FNBs (Figure 4b). In this context, our experimental results highlight the cooperative interaction of FBNs in neural language processing  (Horwitz & Braun, ; Schoffelen et al., ; Fedorenko et al., ).

Figure: Figure 4: (a) The number of brain maps associated with different FBNs. (b) The distribution of the brain maps associated with different number of FBNs.
Refer to caption: extracted/5967503/Figures/FBNDistribution.png

### 4.3 Evolution of Brain-like Functional Organization within LLMs

We present a detailed analysis of the FBN components involved in brain maps across the four models in Figure 5, with the aim of exploring the evolution of brain-like organization patterns within these models. The color-coding in Figure 5 represents the Dice coefficient obtained from FBN labelling  (Kong et al., ), quantifying the spatial overlap between brain maps and FBNs. Positive and negative values denote activation and deactivation of FBNs, respectively. The y-axis is the FBN index, reordered in descending order according to the frequency of FBN involvement in brain maps (both activation and deactivation), with the actual reordered indices provided at the bottom of each sub-figure. The x-axis is the brain map index, organized according to the presence order of FBNs (activation first, followed by deactivation). One noteworthy observation is that a greater number of brain maps with identical FBN labels appears in more advanced LLMs, as highlighted by the braces in Figure 5. In addition, the overall distribution of FBN involvement is noticeably sparser in Llama3 compared to other models. This observation suggests that more advanced LLMs may promote more compact brain-like functional organizations. One possible explanation for this observation is that more advanced LLMs tend to learn more compact representational policies and integrate these policies more efficiently to achieve improved performance on language tasks.

Figure: Figure 5: The detailed FBN components involved in brain maps. The color-coding represents the Dice coefficients obtained from FBN labeling, which measures the spatial overlap between brain maps and FBNs. A negative Dice value here indicates deactivation of FBNs in a given brain map. Braces are used to highlight brain maps that have identical FBN labels.
Refer to caption: extracted/5967503/Figures/reorder.png

It is hypothesized that brain maps with identical FBN labels share similar functional interactions among the associated FBNs. To test this hypothesis, we focused on a subset of brain maps displaying functional interactions between the LatVis (activation) and the language network (deactivation), a pattern consistently observed across the four LLMs (Figure 5, braces with stars). For each LLMs (Figure 6a-d), we show one exemplar brain map (Figure 6, first column) from this subset (subset size: 5/4/6/3 for BERT/Llama1/Llama2/Llama3, respectively), and evaluate the temporal consistency of the subset by calculating the inter-atom Pearson correlation coefficients (Figure 6, second column) of their temporal responses (columns in $\mathbf{D_{\textit{AN}}}$). We also illustrate the distribution of number of ANs on LLM layers (Figure 6, third column).

Our results show that the variability of temporal correlation coefficients decrease sequentially in BERT, Llama1, Llama2 and Llama3, as evidenced by the standard deviations (0.2141, 0.1765, 0.1603 and 0.0233 for the four models, respectively). The high variability in BERT, Llama1 and Llama2 indicates that the subset of atoms in these models exhibit distinct functional processing patterns, despite involving identical FBNs. Meanwhile, the subset of atoms in Llama3 shows the highest temporal consistency (0.236, Figure 6e) compared to other models. Notably, the moderate value of temporal consistency in Llama3 implies a coexistence of both shared and distinctive functional processing patterns among those atoms. These findings provide novel evidence for the principle of functional organization in LLMs: the ANs in more advanced models are organized to achieve an enhanced balance between the diversity of computational behaviors and the consistency of functional specializations.

To further investigate the properties of those atoms within LLMs, we identified the ANs that anchor to a each specific atom and evaluated the consistency of their distribution pattern on LLM layers by calculating the average Pearson correlation coefficient over all possible atom pairs. Our results (Figure 6f) show that the AN distribution patterns are more consistent in Llama3 compared to BERT, Llama1 and Llama2, suggesting a more hierarchical organization of ANs within Llama3. Intriguingly, we observed a greater concentration of ANs in the deeper layers of Llama3. Given that this subset of atoms reveals the activation of LatVis and deactivation of the language network, this finding resonates with neuroscience evidence suggesting that visual imagery is represented at a higher level of the language hierarchy (Speed et al., ; Zwaan, ; Bergen et al., ), highlighting the potential for Llama3 to capture complex linguistic and cognitive processes.

Figure: Figure 6: The subset of atoms that reveals functional interaction between LatVis (activation) and the language network (deactivation) across the four LLMs. (a-d) are for BERT, Llama1, Llama2 and Llama3, respectively, showing the brain maps (column 1), Pearson correlation coefficients between the temporal responses of atom-pairs (column 2), and the distribution pattern of ANs on LLM layers (column 3, the x-axis is layer index). (e) The temporal consistency of atoms. (f) The consistency of the distribution patterns of ANs on LLM layers.
Refer to caption: extracted/5967503/Figures/evolution.png

Figure: Figure 7: (a-b) Two atoms in Llama3 with opposite brain activity patterns and distributions of ANs on layers. (c-d) Two atoms with opposite brain activity patterns but similar distributions of ANs on layers. “+” and “-” represent activation and deactivation, respectively.
Refer to caption: extracted/5967503/Figures/moreAtomsLLaMA3.png

## 5 Conclusion and Discussion

In this study, we explored the brain-like functional organization within LLMs. We built a neural encoding model that uses the representative patterns learned from the temporal responses of AN populations defined in LLMs as fixed regressors to predict functional brain activity. These representative patterns serves as a bridge between AN sub-groups to functional brain networks, enabling us to disentangle how individual ANs within LLMs are functionally organized to support their unprecedented capabilities in language tasks. The proposed framework addresses a key limitations in previous research that examined the behaviors of artificial neurons at a population level, which has hindered a clear understanding of the functional organization within LLMs. Our experimental results demonstrate that the brain-like functional organization within LLMs evolves with their capabilities, where more advanced LLMs achieve an improved balance between diverse computational behaviors and consistent functional specializations.

The present study acknowledges several limitations. First, we fixed the number of atoms (dictionary size) in the dictionary which describes the representative patterns of temporal responses of ANs, despite the fact that the number of ANs varies across different LLMs. Identifying model-specific dictionary size may facilitate a more accurate depiction of the brain-like functional organization within LLMs. Second, our assessment of the coupling relationships between AN sub-groups and functional brain networks was conducted for only a limited number of atoms. However, these coupling relationships described by the remaining atoms could carry valuable clues to investigate neural language processing in the human brain. For example, Figure 7(a-b) illustrate two atoms in Llama3 exhibiting opposite brain activity patterns in the language network, FPN, and LatVis. In accordance, the corresponding distributions of ANs on layers display inverse patterns. On the contrary, Figure 7(c-d) show two atoms demonstrating opposite brain activity patterns in LatVis, MedVis and ParMemory, while the corresponding distributions of ANs on layers remain similar. Thus, future research could aim to further elucidate the brain-like functional organization within LLMs and the neural mechanism underlying language processing by linking brain activity patterns with ANs’ computational behaviors, specifically their selective responses to external stimuli. Third, our experiments were limited to one fMRI session, validating and evaluating this framework on a larger scale fMRI cohort is essential for future studies. Finally, applying the proposed framework to the foundation models in other modalities could provide additional evidence regarding the brain-like functional organization in modern artificial general intelligence models.

## Appendix A Appendix

### A.1 The Brain Maps and distribution of ANs on Layers

Figure: Figure 8: The visualization of brain maps for all the 64 atoms in BERT.
Refer to caption: extracted/5967503/Figures/shapesBERTBrainNet.png

Figure: Figure 9: The distribution of ANs on layers for all the 64 atoms in BERT.
Refer to caption: extracted/5967503/Figures/shapesBERTANdistribution.png

Figure: Figure 10: The visualization of brain maps for all the 64 atoms in Llama1.
Refer to caption: extracted/5967503/Figures/shapesLLaMA1BrainNet.png

Figure: Figure 11: The distribution of ANs on layers for all the 64 atoms in Llama1.
Refer to caption: extracted/5967503/Figures/shapesLLaMA1ANdistribution.png

Figure: Figure 12: The visualization of brain maps for all the 64 atoms in Llama2.
Refer to caption: extracted/5967503/Figures/shapesLLaMA2BrainNet.png

Figure: Figure 13: The distribution of ANs on layers for all the 64 atoms in Llama2.
Refer to caption: extracted/5967503/Figures/shapesLLaMA2ANdistribution.png

Figure: Figure 14: The visualization of brain maps for all the 64 atoms in Llama3.
Refer to caption: extracted/5967503/Figures/shapesLLaMA3BrainNet.png

Figure: Figure 15: The distribution of ANs on layers for all the 64 atoms in Llama3.
Refer to caption: extracted/5967503/Figures/shapesLLaMA3ANdistribution.png

## Peer Reviews

confidence: 4
contribution: 1
initial_rating: 1
presentation: 3
questions: 1. Why is there such a strong activation/deactivation of the lateral visual cortex for fMRI data with no visual component?
2. How was data aggregated across participants?
soundness: 1
strengths: 1. The method is clearly presented.
2. The paper compares scores with prior research to ensure consistency.
summary: This paper utilizes a sparse representation method to extract representative response patterns from multiple LLMs. These are then used to create voxel-wise encoding models of fMRI data. The performance of these models is used to link functional brain networks to the response patterns in the LLMs.
weaknesses: 1. It is unclear why the whole brain was used in this analysis especially when the dataset itself has no inter-subject correlation for most of the brain (Fig 4 in Nastase et al. 2021). This is including the lateral visual cortex which is heavily discussed within the paper (Fig 3. among others).
2. I am unsure how this encoding model design can give insight into how LLM's are functionally organized. This is a regression approach in which the atoms can both linearly encode both some aspects of the brain and the LLM. Already the r^2 values for predicting brain response is 0.1 at max (fig. 2). Along with the values for the LLM being .5-.6 it is unclear what exactly is being captured.
3. It is unclear how the data is aggregated across fMRI subjects or if it is at all. Furthermore the only measure of consistency is the number of atoms that are active/deactive for each FBN (Fig 4). No inter-subject comparison is shown. Fig 6 e and f also show fairly low temporal consistency of atoms.
4. "offering novel insights to inform the development of artificial general intelligence (AGI) with human brain principles." There is no mention for how this method can lead to different approaches in building LLM's. 




Small corrections:
Line 258 wrong spelling: "vaules"

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: I’m confused by figure 6. The second column seems to compare temporal consistency amongst atoms, but the diagonal (measuring the consistency of the same atom) appears to be 0. Should these values not be the highest, and—if so—why are they colored green? I also am unsure what the third column to meant to convey. They plots seem to be histograms of ANs across layers: should the values all be small (due to the sparsity constraint), and how does this relate to the functional organization of the LLM and its correspondence to the brain?

Figure 4 shows the brain maps associated with different FBNs. What is the criteria for a brain map to be classified as ‘associated’?
soundness: 3
strengths: Strengths:

Nicely written intro with a clear motivation and big question

Overall the sparse encoding model linking LLMs to the brains is an interesting and topical approach with relevance to on-going work in both machine learning and cognitive neuroscience. 

Additionally, the focus on sub-population encoding models provides a novel approach in comparing LLMs to the brain.
summary: This paper aims to compare the functional organization of large language models (LLMs) with the brain. It measures this correspondence using sub-groups of artificial neurons (ANs) in LLMs, established functional brain networks (FBNs) in the literature, and Narratives—a publicly available fMRI dataset. 

In each of 4 popular LLMs, the authors extract the responses of all ANs to the tokenized input of the Narratives stimuli. These activations are then sparsely decomposed into representative ‘atoms,’ which are subsequently used in voxel-wise encoding model. This method yields predicitvity scores of fMRI responses in individual subjects. 

Finally, the paper investigates the activity and organization of the representative atoms in each of the LLMs by correlating temporal responses and measures the spatial distribution of atoms across LLM layers. 

Key to this paper’s claim is that most models comparing neural networks to the brain do so at a population level. The ‘atoms’ representing LLM activity are thus a way to represent individual and sub-groups of ANs and their correspondence to the brain. 

Should the weaknesses described below be adequately addressed, I would weakly recommend accepting the paper.
weaknesses: Weaknesses:
The critical method used in this paper is the sparse decomposition of LLM responses into ‘atoms’ that represent sub-groups of ANs.  The parameters of this representation are stated clearly in lines 243-245, but there is no motivation provided for choosing these values. Why 64 atoms, what are the consequences of other numbers, and what goes into the choice of sparsity constraints? These questions should be acknowledged. 

The paper claims to be the first study to characterize functional brain-like organization in LLMs, yet similar work (eg. Schrimpf et al, 2021) is acknowledged in the paper, apparently contradicting this claim. Instead, perhaps the claim should focus on characterization using sparse, sub-population level methods. 

Automatic brain labeling (Kong el al, 2024) is a central method to this paper’s findings, but how this value is calculated is not mentioned. This method should be motivated and explained in more detail. 

The brain predictivity scores are modest (figure 2B) and do not seem comparable to cited similar studies despite the authors’ claims (lines 265-268).

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. The abstract and introduction should be reframed to prevent the work from over promising both in terms of novelty, and over promising in terms of the granularity of the analysis. See weaknesses Major 1 for more context. 
2. Suggestions/take aways based on experiments should be reframed to clarify that these are just one plausible interpretation based on the results, and either alternatives should be described as well or limitations should be clearly highlighted which impact the interpretation of the results. One example is provided in weakness Major 6. 
3. How do the insights from this approach differ from insights from prior approaches which align LLM layer representations to brain activity as measured by fMRI? 
4. How were the hyperparemeters referenced in the paper such as dictionary size (k), sparsity constraint (lambda_AN), and lambda_f chosen? 
5. How were voxel-wise encoding models were trained, see weakness major 3b for some examples of the missing information? 
6. What are the missing key experimental details highlighted in weaknesses major 4b? 
7. Additional details should be provided to explain the low R^2 for the encoding models in Fig 2b, whether that is a reference to explicit ranges of other encoding models using the same dataset and session, or applying the proposed approach to another dataset or session to verify that the low predictive ability was due to the data as opposed to the approach.
soundness: 2
strengths: 1. This work focuses on a important research question, are there similarities in the organization of LLMs and the brain. This question is of interest to both the ML and neuroscience community, as insights into this question will likely help inform increased understanding of the brain as well as provide insights into how to build better ML models. 
2. This work proposes a novel approach to study the alignment between LLMs and brain activity as measured by fMRI. The new approach takes all artificial neurons (ANs) in a LLM and extracts representative patterns from the ANs. The novelty is that this approach enables the LLM representation to span across the LLMs layers, whereas prior approaches focused on layer specific representations.
summary: This work focuses on a important research question, are there similarities in the organization of LLMs and the brain. This question is of interest to both the ML and neuroscience community, as insights into this question will likely help inform increased understanding of the brain as well as provide insights into how to build better ML models. 

To address this question, this work proposes a novel approach to study the alignment between LLMs and brain activity as measured by fMRI. The new approach takes all artificial neurons (ANs) in a LLM and extracts representative patterns from the ANs. The novelty is that this approach enables the LLM representation to span across the LLMs layers, whereas prior approaches focused on layer specific representations.   

This approach is applied to one session from the Narratives fMRI dataset, and 4 models (BERT, Llama 1-3), to begin to yield insights into the scientific quesetion. Some analysis of the output of applying this approach is then presented.
weaknesses: Major: 
1. Imprecise framing leads to the paper over promising what the paper presents: 
(a) The paper proposes as a gap in the field that prior work was at a "population-level" and "individual ANs have yet to be linked to neural response". Despite this framing this paper doesn't fill this gap. Specifically, this work doesn't link individual ANs to neural response. Rather this work takes all ANs and extracts representative patterns from these ANs (dictionary) and then links these representative patterns to neural response (brain activity as measured by fMRI). While this is still a novel approach, the framing suggests that this paper links individual ANs to brain activity and the subsequent work presented does not accomplish this. 
(b) Additionally, the abstract claims that "This research represent the first exploration of brain-like functional organization within LLMs". It is true that this is the first time this specific approach has been proposed, however it is easy to argue that the multiple prior works on alignment between LLMs and "neural responses" were also focused on understanding brain-like functional organization within LLMs. 

2. While the approach is novel it is unclear from the work presented if it is simply a new approach, or if the scientific insights are complementary to prior approaches which align LLM layer representations to brain activity as measured by fMRI. As the goal of both types of approaches is to study the similarities between LLMs and the brain it would be valuable to highlight if this novel approach leads to new scientific insights. 

3. Missing Key Technical Details: 
(a) Multiple hyperparameters are mentioned, however there are no details shared about how these hyperparameters were selected. This includes the dictionary size (k), sparsity constraint (lambda_AN), and lambda_f. Especially as there is a clear decrease in the performance of dictionary learning for the larger models (Fig. 2 a) it is important to understand why the specific dictionary size, as opposed to a larger dictionary size was chosen. 
(b) Details on how the voxel-wise encoding models were trained are missing, for example was cross-validation used, how large were the folds, etc. These are crucial details to be able to replicate the work and judge the validity of the learned models. 

4. Missing Key Experimental Details: 
(a) Details on how the FBNs that were chosen are missing. There are many networks defined in many different ways throughout the computational neuroscience literature, therefore this choice can have a big impact on scientific insights from the approach presented as well as provide insights into how generalizable this new approach is. 
(b) Limited details about the Narratives dataset are presented. For example it is unclear how many TRs exist, the length of the session used, whether the subjects were instructed to stare at something during the experiment, etc. 

5. Low R^2 for encoding models: Despite the paper stating that the R^2 range in Figure 2b "closely resembles" scores from prior studies, these R^2 values are actually low relative to what is common from encoding models predicting fMRI activity for naturalistic stimuli. This suggests that the representative patterns from these ANs is not as good of a predictor from fMRI activity as other representations. If low R^2's are common for this specific dataset and selected fMRI session selected, it would be beneficial to apply the proposed approach to another dataset or session to verify that the low predictive ability was due to the data as opposed to the approach. It would also be helpful to reference (if they exist) the explicit ranges of other encoding models using the same dataset and session. 
 
6. Overly strong take aways based off limited results: There are multiple suggestions/claims that are made without considering alternative explanations or explaining potential bias based on the limited results presented. For example, in section 4.3 there is a focus on the average correlation coefficient over all possible atom pairs 6f, the results lead to a take away about Llama3. However given the limited number of results the take away is based on, an alternative explanation is that there are fewer (only 3) atoms for the Llama3 example as opposed to 4-6 atoms for the other models and given this it's possible that the higher correlation is simply due to having fewer atoms to measure correlation between leading to less variability. 

Minor: 
1. There are some typos throughout the paper, including (a) page 5 section 4.1 "vaules" is likely intended to be "values", (b) page 6 section 4.2 "involvement of FBNs...across different FBNs" it is likely that the second FBN was meant to refer to something else, (c) page 7 section 4.2 mentions "studies" but only cites one study Yeshurun et al. 2021. 

2. At times the paper is unclear especially in how the figures are presented and described in the text. For example in Figure 2 the audience is left to interpret what probability (the y-axis) is in reference to. Also in figure 2 the corresponding part of the figure is sometimes before the description and sometimes after the description.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 1
questions: How does this paper differ from Kumar et al., (2024)?

How do the authors reconcile the primary finding that visual/language networks are represented in the LLMs with the claim of the paper: brain-like functional organization in LLMs? (The brain is not limited to just visual and language networks.)

The authors write that this approach distinguishes itself from a population-level approach by identifying sub-groups. What is the difference between a population-level and sub-group level? 

More minor questions:

BERT seems to be an ill-chosen model for this comparison (human v LLM), given that it is bidirectional. In the human dataset, words are presented sequentially (unidirectionally). Yet with BERT, words are provided/accessed in parallel; past words have access (or attended) to embeddings from future words. Conceptually, it’s not clear to me why one would include this model for analysis/comparison with human data, since it’s an unrealistic scenario.

On line 370, they mention the observation that FBN involvement is sparser in Llama3, and is suggested that more advanced LLM models may promote more compact brain-like functional organizations. Does this analysis account/control for LLM size?

What number of atoms were used when learning their dictionary? Do the results depend on the choice of k?
soundness: 3
strengths: The technical components and the methods in the paper are sound, i.e., dictionary learning and voxel-wise encoding models. Their findings are also corroborated by related papers in the field, which also show strong correspondence between brain activity and LLMs (Schrimpf et al., 2021; Tuckute et al., 2023, Caucheteux et al., 2022). The fact that the learned dictionary response patterns can capture brain responses at all is interesting, given that the AN temporal response patterns are derived independent of brain activity.
summary: In this paper, the authors relate the units of LLMs to different functional brain networks during narrative comprehension using functional magnetic resonance imaging (fMRI). First, the authors identify the temporal response pattern of artificial neurons in the LLM, and then use these response profiles as regressors to then map onto fMRI brain activity. They perform this on several common LLMs, including BERT and the Llama family of models. The authors’ main finding is that LLM activity maps onto meaningful clusters of brain activity (i.e., functional brain network patterns), suggesting that LLMs are organized similarly to the human brain.
weaknesses: My greatest concern is that the conceptual motivation for this manuscript appears confused and limited. The manuscript is largely motivated by understanding the functional organization of LLMs through the lens of the human brain. Understanding LLMs through the lens of the human brain assumes that we have a strong understanding of how the human brain works. But one can argue that our understanding of LLMs is actually stronger than our understanding of the human brain: We have access to all parameters, the objective function, and optimization procedures. Thus, it’s unclear what understanding is gained by understanding LLMs through the ‘functional organization of human brain networks’. 

Suppose we give the manuscript the benefit of the doubt; perhaps there is something to be learned by evaluating whether LLMs exhibit the same functional organization as human brain networks. From the reported results, it appears that only language and visual networks appear to be represented. These findings seem limited, and have already been previously described (e.g., Kumar et al., 2024). Moreover, not all functional brain networks appear to be equally represented within the LLM, which is not entirely consistent with the title of the paper.

In terms of precedent, it’s also slightly unclear how the main findings of this paper conceptually differ from Kumar et al. (2024), which doesn’t appear to be cited.

Kumar, Sreejan, Theodore R. Sumers, Takateru Yamakoshi, Ariel Goldstein, Uri Hasson, Kenneth A. Norman, Thomas L. Griffiths, Robert D. Hawkins, and Samuel A. Nastase. “Shared Functional Specialization in Transformer-Based Language Models and the Human Brain.” Nature Communications 15, no. 1 (June 29, 2024): 5523. https://doi.org/10.1038/s41467-024-49173-5.